Skip to content

Add initial integration test code #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
foxish merged 22 commits into from
Dec 20, 2017
Merged

Add initial integration test code #3

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e82d4c5
Made test code compile
kimoonkim Dec 15, 2017
7c8612c
Clean up pom.xml
kimoonkim Dec 15, 2017
5a950eb
Unpack distro in place
kimoonkim Dec 16, 2017
b14430e
Clean up redundant resources
kimoonkim Dec 16, 2017
0443af8
Avoid buggy truezip
kimoonkim Dec 16, 2017
567ad3e
Cleaned up docker image builder
kimoonkim Dec 16, 2017
74c158b
Builds some docker images
kimoonkim Dec 16, 2017
bde1cf9
Drop http jar support for now
kimoonkim Dec 16, 2017
8624545
Clean up
kimoonkim Dec 16, 2017
5585a0a
Use spark-submit
kimoonkim Dec 16, 2017
f793e38
Tests pass
kimoonkim Dec 16, 2017
069ae50
Add hacks for dockerfiles and entrypoint.sh
kimoonkim Dec 17, 2017
204c51f
Address review comments
kimoonkim Dec 18, 2017
6a93a9e
Clean up pom.xml
kimoonkim Dec 18, 2017
b3e4ee0
Add instructions in README.md
kimoonkim Dec 18, 2017
989a371
Switch to container.image property keys
kimoonkim Dec 18, 2017
46d1f5f
Define version properies in pom.xml
kimoonkim Dec 18, 2017
5f88921
Fix a bug in pom.xml
kimoonkim Dec 19, 2017
a68bd5f
Clean up and fix README.md
kimoonkim Dec 19, 2017
0f8fad9
Fix README.md
kimoonkim Dec 19, 2017
7bc4aa2
Remove unnecessary close
kimoonkim Dec 19, 2017
9ec071a
Clean up
kimoonkim Dec 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Define version properies in pom.xml
  • Loading branch information
@kimoonkim
kimoonkim committed Dec 18, 2017
commit 46d1f5f49a8111d96dce15aaefafbbca6fe3e695
31 changes: 20 additions & 11 deletions integration-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,20 @@
<version>0.1-SNAPSHOT</version>
<properties>
<commons-lang3.version>3.5</commons-lang3.version>
<commons-logging.version>1.1.1</commons-logging.version>
<docker-client.version>5.0.2</docker-client.version>
<download-maven-plugin.version>1.3.0</download-maven-plugin.version>
<exec-maven-plugin.version>1.4.0</exec-maven-plugin.version>
<guava.version>18.0</guava.version>
<jsr305.version>1.3.9</jsr305.version>
<kubernetes.client.version>3.0.0</kubernetes.client.version>
<kubernetes-client.version>3.0.0</kubernetes-client.version>
<log4j.version>1.2.17</log4j.version>
<scala.version>2.11.8</scala.version>
<scala.binary.version>2.11</scala.binary.version>
<scala-maven-plugin.version>3.2.2</scala-maven-plugin.version>
<scalatest.version>2.2.6</scalatest.version>
<scalatest-maven-plugin.version>1.0</scalatest-maven-plugin.version>
<slf4j-log4j12.version>1.7.24</slf4j-log4j12.version>
<sbt.project.name>kubernetes-integration-tests</sbt.project.name>
<spark-distro-tgz>YOUR-SPARK-DISTRO-TARBALL-HERE</spark-distro-tgz>
<spark-dockerfiles-dir>YOUR-DOCKERFILES-DIR-HERE</spark-dockerfiles-dir>
Expand All @@ -40,7 +49,7 @@
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
<version>1.1.1</version>
<version>${commons-logging.version}</version>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
Expand All @@ -52,18 +61,18 @@
<artifactId>guava</artifactId>
<scope>test</scope>
<!-- For compatibility with Docker client. Should be fine since this is just for tests.-->
<version>18.0</version>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>com.spotify</groupId>
<artifactId>docker-client</artifactId>
<version>5.0.2</version>
<version>${docker-client.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
<version>${kubernetes.client.version}</version>
<version>${kubernetes-client.version}</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
Expand All @@ -83,13 +92,13 @@
<dependency>
<groupId>org.scalatest</groupId>
<artifactId>scalatest_${scala.binary.version}</artifactId>
<version>2.2.6</version>
<version>${scalatest.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.24</version>
<version>${slf4j-log4j12.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
Expand All @@ -99,7 +108,7 @@
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<version>${scala-maven-plugin.version}</version>
<executions>
<execution>
<goals>
Expand All @@ -112,7 +121,7 @@
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.4.0</version>
<version>${exec-maven-plugin.version}</version>
<executions>
<execution>
<id>unpack-spark-distro</id>
Expand Down Expand Up @@ -166,7 +175,7 @@
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
<version>1.3.0</version>
<version>${download-maven-plugin.version}</version>
<executions>
<execution>
<id>download-minikube-linux</id>
Copy link
Contributor

@mccheah mccheah Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be downloading here, we should be using the built-in Minikube binary.

Copy link
Member Author

@kimoonkim kimoonkim Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that's what we want, but I suggest we address this in a future PR. I think we'd like to start testing the upstream sooner than later.

Copy link
Contributor

@mccheah mccheah Dec 18, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the Riselab nodes where we are running this will have Minikube installed already, as it assumes the setup where the Minikube instance is being re-used.

Copy link
Contributor

@mccheah mccheah Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So it should be just as easy to use the pre-installed minikube as it would be to download a new one every time. That is, the total time from now to when the tests start running against upstream should be equivalent regardless of which branch we use - so we should use the more robust mode from the start.

Copy link
Member Author

@kimoonkim kimoonkim Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copying my reply from below:

I am not familiar with #521 yet. So I don't know how much extra time it would need to address the code and potential review comments that the new code would invite. And I personally don't have a lot of extra time to spend on this beyond what I spent already.

I also don't know much about the riselab setup. I was hoping to delay that until @ssuchter comes back. We can use Pepperdata jenkins in the meantime.

So I really hope that we can keep this PR simple and address gaps in future PRs.

Expand Down Expand Up @@ -199,7 +208,7 @@
the test phase. -->
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<version>${scalatest-maven-plugin.version}</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
Expand Down