Skip to content

Add initial integration test code #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
foxish merged 22 commits into from
Dec 20, 2017
Merged

Add initial integration test code #3

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e82d4c5
Made test code compile
kimoonkim Dec 15, 2017
7c8612c
Clean up pom.xml
kimoonkim Dec 15, 2017
5a950eb
Unpack distro in place
kimoonkim Dec 16, 2017
b14430e
Clean up redundant resources
kimoonkim Dec 16, 2017
0443af8
Avoid buggy truezip
kimoonkim Dec 16, 2017
567ad3e
Cleaned up docker image builder
kimoonkim Dec 16, 2017
74c158b
Builds some docker images
kimoonkim Dec 16, 2017
bde1cf9
Drop http jar support for now
kimoonkim Dec 16, 2017
8624545
Clean up
kimoonkim Dec 16, 2017
5585a0a
Use spark-submit
kimoonkim Dec 16, 2017
f793e38
Tests pass
kimoonkim Dec 16, 2017
069ae50
Add hacks for dockerfiles and entrypoint.sh
kimoonkim Dec 17, 2017
204c51f
Address review comments
kimoonkim Dec 18, 2017
6a93a9e
Clean up pom.xml
kimoonkim Dec 18, 2017
b3e4ee0
Add instructions in README.md
kimoonkim Dec 18, 2017
989a371
Switch to container.image property keys
kimoonkim Dec 18, 2017
46d1f5f
Define version properies in pom.xml
kimoonkim Dec 18, 2017
5f88921
Fix a bug in pom.xml
kimoonkim Dec 19, 2017
a68bd5f
Clean up and fix README.md
kimoonkim Dec 19, 2017
0f8fad9
Fix README.md
kimoonkim Dec 19, 2017
7bc4aa2
Remove unnecessary close
kimoonkim Dec 19, 2017
9ec071a
Clean up
kimoonkim Dec 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clean up pom.xml
  • Loading branch information
@kimoonkim
kimoonkim committed Dec 18, 2017
commit 6a93a9edfd31d7df0fd84284ceeddfcffbd72e32
24 changes: 22 additions & 2 deletions integration-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
<sbt.project.name>kubernetes-integration-tests</sbt.project.name>
<spark-distro-tgz>YOUR-SPARK-DISTRO-TARBALL-HERE</spark-distro-tgz>
<spark-dockerfiles-dir>YOUR-DOCKERFILES-DIR-HERE</spark-dockerfiles-dir>
<test.exclude.tags></test.exclude.tags>
</properties>
<packaging>jar</packaging>
<name>Spark Project Kubernetes Integration Tests</name>
Expand Down Expand Up @@ -85,6 +86,12 @@
<version>2.2.6</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<version>1.7.24</version>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down Expand Up @@ -118,7 +125,7 @@
<executable>/bin/sh</executable>
<arguments>
<argument>-c</argument>
<argument>mkdir spark-distro-tmp; cd spark-distro-tmp; tar xfz ${spark-distro-tgz}; mv * ../spark-distro; cd ..; rm -rf spark-distro-tmp</argument>
<argument>rm -rf spark-distro; mkdir spark-distro-tmp; cd spark-distro-tmp; tar xfz ${spark-distro-tgz}; mv * ../spark-distro; cd ..; rm -rf spark-distro-tmp</argument>
</arguments>
</configuration>
</execution>
Expand All @@ -139,7 +146,7 @@
</configuration>
</execution>
<execution>
<!-- TODO: Remove this hack once the upstream is fixed -->
<!-- TODO: Remove this hack once upstream is fixed by SPARK-22777 -->
<id>set-exec-bit-on-docker-entrypoint-sh</id>
Copy link
Contributor

@mccheah mccheah Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just set it in the Dockerfile using: RUN chmod +x /opt/entrypoint.sh.

Copy link
Member Author

@kimoonkim kimoonkim Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's better to be done by the upstream code. It's hard for this integration code to surgically do in-place edit of Dockerfile.

Copy link
Contributor

@mccheah mccheah Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Member

@foxish foxish Dec 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR has merged now.

<phase>pre-integration-test</phase>
<goals>
Expand Down Expand Up @@ -192,6 +199,19 @@
the test phase. -->
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>1.0</version>
<configuration>
<reportsDirectory>${project.build.directory}/surefire-reports</reportsDirectory>
<junitxml>.</junitxml>
<filereports>SparkTestSuite.txt</filereports>
<argLine>-ea -Xmx3g -XX:ReservedCodeCacheSize=512m ${extraScalaTestArgs}</argLine>
<stderr/>
<systemProperties>
<log4j.configuration>file:src/test/resources/log4j.properties</log4j.configuration>
<java.awt.headless>true</java.awt.headless>
</systemProperties>
<tagsToExclude>${test.exclude.tags}</tagsToExclude>
</configuration>
<executions>
<execution>
<id>test</id>
Expand Down