Skip to content

[SPARK-26083][k8s] Add Copy pyspark into corresponding dir cmd in pyspark Dockerfile #23037

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
AzureQ wants to merge 6 commits into from
Closed

[SPARK-26083][k8s] Add Copy pyspark into corresponding dir cmd in pyspark Dockerfile #23037

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c2f782b
Copy python/pyspark to ${SPARK_HOME}/python/pyspark to make bin/pyspa…
AzureQ Nov 14, 2018
4bf6bc6
Added test for client mode pyspark shell into PythonTestsSuite
AzureQ Nov 19, 2018
c626bd1
Correct pyspark shell test's driver image
AzureQ Nov 20, 2018
149bed4
Remove erroring test case for pyspark shell
AzureQ Nov 29, 2018
ff8d0ba
Merge remote-tracking branch 'upstream/master'
AzureQ Dec 1, 2018
5f6aaf8
Updated docker-image-tool.sh with copying pyspark
AzureQ Dec 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Correct pyspark shell test's driver image
  • Loading branch information
@AzureQ
AzureQ committed Nov 20, 2018
commit c626bd1cc3ed754502373a1df4fb8943cf56fde6
6 changes: 3 additions & 3 deletions ...n-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/PythonTestsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ private[spark] trait PythonTestsSuite { k8sSuite: KubernetesSuite =>
pyFiles = Some(PYSPARK_CONTAINER_TESTS))
}

test("Run bin/pyspark in client mode", k8sTestTag) {
test("Run PySpark shell", k8sTestTag) {
val labels = Map("spark-app-selector" -> driverPodName)
val driverPort = 7077
val blockManagerPort = 10000
Expand Down Expand Up @@ -135,8 +135,8 @@ private[spark] trait PythonTestsSuite { k8sSuite: KubernetesSuite =>
.withNewSpec()
.withServiceAccountName(kubernetesTestComponents.serviceAccountName)
.addNewContainer()
.withName("pyspark-example")
.withImage(image)
.withName("pyspark-shell")
.withImage(pyImage)
.withImagePullPolicy("IfNotPresent")
.withCommand("/opt/spark/bin/pyspark")
.addToArgs("--master", s"k8s://https://kubernetes.default.svc")
Expand Down