Skip to content

Add initial integration test code #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
foxish merged 22 commits into from
Dec 20, 2017
Merged

Add initial integration test code #3

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
e82d4c5
Made test code compile
kimoonkim Dec 15, 2017
7c8612c
Clean up pom.xml
kimoonkim Dec 15, 2017
5a950eb
Unpack distro in place
kimoonkim Dec 16, 2017
b14430e
Clean up redundant resources
kimoonkim Dec 16, 2017
0443af8
Avoid buggy truezip
kimoonkim Dec 16, 2017
567ad3e
Cleaned up docker image builder
kimoonkim Dec 16, 2017
74c158b
Builds some docker images
kimoonkim Dec 16, 2017
bde1cf9
Drop http jar support for now
kimoonkim Dec 16, 2017
8624545
Clean up
kimoonkim Dec 16, 2017
5585a0a
Use spark-submit
kimoonkim Dec 16, 2017
f793e38
Tests pass
kimoonkim Dec 16, 2017
069ae50
Add hacks for dockerfiles and entrypoint.sh
kimoonkim Dec 17, 2017
204c51f
Address review comments
kimoonkim Dec 18, 2017
6a93a9e
Clean up pom.xml
kimoonkim Dec 18, 2017
b3e4ee0
Add instructions in README.md
kimoonkim Dec 18, 2017
989a371
Switch to container.image property keys
kimoonkim Dec 18, 2017
46d1f5f
Define version properies in pom.xml
kimoonkim Dec 18, 2017
5f88921
Fix a bug in pom.xml
kimoonkim Dec 19, 2017
a68bd5f
Clean up and fix README.md
kimoonkim Dec 19, 2017
0f8fad9
Fix README.md
kimoonkim Dec 19, 2017
7bc4aa2
Remove unnecessary close
kimoonkim Dec 19, 2017
9ec071a
Clean up
kimoonkim Dec 19, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Cleaned up docker image builder
  • Loading branch information
@kimoonkim
kimoonkim committed Dec 16, 2017
commit 567ad3eab6eeba500b1420b597fbe5cd564bb15f
17 changes: 0 additions & 17 deletions ...st/scala/org/apache/spark/deploy/k8s/integrationtest/docker/SparkDockerImageBuilder.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,9 @@
*/
package org.apache.spark.deploy.k8s.integrationtest.docker

import java.io.File
import java.net.URI
import java.nio.file.Paths

import scala.collection.JavaConverters._

import com.spotify.docker.client.{DefaultDockerClient, DockerCertificates, LoggingBuildHandler}
import org.apache.http.client.utils.URIBuilder
import org.scalatest.concurrent.{Eventually, PatienceConfiguration}
Expand All @@ -36,14 +33,7 @@ private[spark] class SparkDockerImageBuilder
// Dockerfile paths must be relative to the build path.
private val BASE_DOCKER_FILE = "dockerfiles/spark-base/Dockerfile"
private val DRIVER_DOCKER_FILE = "dockerfiles/driver/Dockerfile"
private val DRIVERPY_DOCKER_FILE = "dockerfiles/driver-py/Dockerfile"
private val DRIVERR_DOCKER_FILE = "dockerfiles/driver-r/Dockerfile"
private val EXECUTOR_DOCKER_FILE = "dockerfiles/executor/Dockerfile"
private val EXECUTORPY_DOCKER_FILE = "dockerfiles/executor-py/Dockerfile"
private val EXECUTORR_DOCKER_FILE = "dockerfiles/executor-r/Dockerfile"
private val SHUFFLE_SERVICE_DOCKER_FILE = "dockerfiles/shuffle-service/Dockerfile"
private val INIT_CONTAINER_DOCKER_FILE = "dockerfiles/init-container/Dockerfile"
private val STAGING_SERVER_DOCKER_FILE = "dockerfiles/resource-staging-server/Dockerfile"
private val STATIC_ASSET_SERVER_DOCKER_FILE =
"dockerfiles/integration-test-asset-server/Dockerfile"
private val TIMEOUT = PatienceConfiguration.Timeout(Span(2, Minutes))
Expand Down Expand Up @@ -73,14 +63,7 @@ private[spark] class SparkDockerImageBuilder
Eventually.eventually(TIMEOUT, INTERVAL) { dockerClient.ping() }
buildImage("spark-base", BASE_DOCKER_FILE)
buildImage("spark-driver", DRIVER_DOCKER_FILE)
buildImage("spark-driver-py", DRIVERPY_DOCKER_FILE)
buildImage("spark-driver-r", DRIVERR_DOCKER_FILE)
buildImage("spark-executor", EXECUTOR_DOCKER_FILE)
buildImage("spark-executor-py", EXECUTORPY_DOCKER_FILE)
buildImage("spark-executor-r", EXECUTORR_DOCKER_FILE)
buildImage("spark-shuffle", SHUFFLE_SERVICE_DOCKER_FILE)
buildImage("spark-resource-staging-server", STAGING_SERVER_DOCKER_FILE)
buildImage("spark-init", INIT_CONTAINER_DOCKER_FILE)
buildImage("spark-integration-test-asset-server", STATIC_ASSET_SERVER_DOCKER_FILE)
}

Expand Down