Skip to content

Add initial integration test code #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
foxish merged 22 commits into from
Dec 20, 2017
Merged

Add initial integration test code #3

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e82d4c5
Made test code compile
kimoonkim Dec 15, 2017
7c8612c
Clean up pom.xml
kimoonkim Dec 15, 2017
5a950eb
Unpack distro in place
kimoonkim Dec 16, 2017
b14430e
Clean up redundant resources
kimoonkim Dec 16, 2017
0443af8
Avoid buggy truezip
kimoonkim Dec 16, 2017
567ad3e
Cleaned up docker image builder
kimoonkim Dec 16, 2017
74c158b
Builds some docker images
kimoonkim Dec 16, 2017
bde1cf9
Drop http jar support for now
kimoonkim Dec 16, 2017
8624545
Clean up
kimoonkim Dec 16, 2017
5585a0a
Use spark-submit
kimoonkim Dec 16, 2017
f793e38
Tests pass
kimoonkim Dec 16, 2017
069ae50
Add hacks for dockerfiles and entrypoint.sh
kimoonkim Dec 17, 2017
204c51f
Address review comments
kimoonkim Dec 18, 2017
6a93a9e
Clean up pom.xml
kimoonkim Dec 18, 2017
b3e4ee0
Add instructions in README.md
kimoonkim Dec 18, 2017
989a371
Switch to container.image property keys
kimoonkim Dec 18, 2017
46d1f5f
Define version properies in pom.xml
kimoonkim Dec 18, 2017
5f88921
Fix a bug in pom.xml
kimoonkim Dec 19, 2017
a68bd5f
Clean up and fix README.md
kimoonkim Dec 19, 2017
0f8fad9
Fix README.md
kimoonkim Dec 19, 2017
7bc4aa2
Remove unnecessary close
kimoonkim Dec 19, 2017
9ec071a
Clean up
kimoonkim Dec 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Tests pass
  • Loading branch information
@kimoonkim
kimoonkim committed Dec 16, 2017
commit f793e387bae5a083a7cded7a366b5d0ed3386f17
22 changes: 0 additions & 22 deletions integration-test/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,28 +123,6 @@
</execution>
</executions>
</plugin>
<plugin>
<artifactId>maven-resources-plugin</artifactId>
<version>3.0.2</version>
<executions>
<execution>
<id>copy-integration-test-http-server-dockerfile</id>
<phase>pre-integration-test</phase>
<goals>
<goal>copy-resources</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/spark-distro/dockerfiles</outputDirectory>
<resources>
<resource>
<directory>src/main/docker</directory>
<filtering>true</filtering>
</resource>
</resources>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>com.googlecode.maven-download-plugin</groupId>
<artifactId>download-maven-plugin</artifactId>
Expand Down
21 changes: 0 additions & 21 deletions integration-test/src/main/docker/integration-test-asset-server/Dockerfile
View file
Open in desktop

This file was deleted.

9 changes: 3 additions & 6 deletions ...ion-test/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,18 +16,15 @@
*/
package org.apache.spark.deploy.k8s.integrationtest

import java.io.{File, FileOutputStream}
import java.io.File
import java.nio.file.Paths
import java.util.{Properties, UUID}
import java.util.UUID
import java.util.regex.Pattern

import com.google.common.base.Charsets
import com.google.common.io.{Files, PatternFilenameFilter}
import io.fabric8.kubernetes.client.internal.readiness.Readiness
import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll, FunSuite}
import org.scalatest.concurrent.{Eventually, PatienceConfiguration}
import org.scalatest.time.{Minutes, Seconds, Span}
import scala.collection.JavaConverters._

import org.apache.spark.deploy.k8s.integrationtest.backend.IntegrationTestBackendFactory
import org.apache.spark.deploy.k8s.integrationtest.constants.MINIKUBE_TEST_BACKEND
Expand Down Expand Up @@ -118,7 +115,7 @@ private[spark] object KubernetesSuite {
"examples", "jars")
.toFile
.listFiles(new PatternFilenameFilter(Pattern.compile("^spark-examples_.*\\.jar$")))(0)
val CONTAINER_LOCAL_SPARK_DISTRO_EXAMPLES_JAR: String = s"local:///opt/spark/examples/" +
val CONTAINER_LOCAL_SPARK_DISTRO_EXAMPLES_JAR: String = s"local:///opt/spark/examples/jars/" +
s"${SPARK_DISTRO_EXAMPLES_JAR_FILE.getName}"
val SPARK_PI_MAIN_CLASS: String = "org.apache.spark.examples.SparkPi"

Expand Down
2 changes: 1 addition & 1 deletion ...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import java.nio.file.Paths
import java.util.UUID

import io.fabric8.kubernetes.client.DefaultKubernetesClient
import org.scalatest.concurrent.{Eventually, PatienceConfiguration}
import org.scalatest.concurrent.Eventually
import scala.collection.mutable
Copy link
Member

@liyinan926 liyinan926 Dec 18, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Scala packages should be in a group after java packages.

import scala.collection.JavaConverters._

Expand Down
62 changes: 0 additions & 62 deletions ...rc/test/scala/org/apache/spark/deploy/k8s/integrationtest/StaticAssetServerLauncher.scala
View file
Open in desktop

This file was deleted.

2 changes: 1 addition & 1 deletion integration-test/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
*/
package org.apache.spark.deploy.k8s.integrationtest

import java.io.{Closeable, InputStream, IOException, OutputStream}
import java.io.Closeable

object Utils {

Expand Down
3 changes: 1 addition & 2 deletions ...st/scala/org/apache/spark/deploy/k8s/integrationtest/backend/IntegrationTestBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ package org.apache.spark.deploy.k8s.integrationtest.backend
import io.fabric8.kubernetes.client.DefaultKubernetesClient

import org.apache.spark.deploy.k8s.integrationtest.backend.GCE.GCETestBackend
import org.apache.spark.deploy.k8s.integrationtest.backend.minikube.{Minikube, MinikubeTestBackend}
import org.apache.spark.deploy.k8s.integrationtest.docker.SparkDockerImageBuilder
import org.apache.spark.deploy.k8s.integrationtest.backend.minikube.MinikubeTestBackend

private[spark] trait IntegrationTestBackend {
def name(): String
Expand Down