[SPARK-26742][K8s][branch-2.4] Update k8s client version to 4.1.2 #23993

skonto · 2019-03-06T18:02:09Z

What changes were proposed in this pull request?

Updates client version and fixes some related issues.

How was this patch tested?

Tested with the latest minikube version and k8s 1.13.
KubernetesSuite:

Run SparkPi with no resources
Run SparkPi with a very long application name.
Use SparkLauncher.NO_RESOURCE
Run SparkPi with a master URL without a scheme.
Run SparkPi with an argument.
Run SparkPi with custom labels, annotations, and environment variables.
Run extraJVMOptions check on driver
Run SparkRemoteFileTest using a remote data file
Run SparkPi with env and mount secrets.
Run PySpark on simple pi.py example
Run PySpark with Python2 to test a pyfiles example
Run PySpark with Python3 to test a pyfiles example
Run PySpark with memory customization
Run in client mode.
Run completed in 4 minutes, 20 seconds.
Total number of tests run: 14
Suites: completed 2, aborted 0
Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM 2.4.2-SNAPSHOT ............ SUCCESS [ 2.980 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 2.880 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 1.954 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 3.369 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 1.791 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 1.845 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 3.725 s]
[INFO] Spark Project Core ................................. SUCCESS [ 23.572 s]
[INFO] Spark Project Kubernetes Integration Tests 2.4.2-SNAPSHOT SUCCESS [04:25 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:08 min
[INFO] Finished at: 2019-03-06T18:03:55Z
[INFO] ------------------------------------------------------------------------

SparkQA · 2019-03-06T18:10:36Z

Test build #103105 has finished for PR 23993 at commit 5e6e770.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-06T18:17:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8567/

SparkQA · 2019-03-06T18:27:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8567/

skonto · 2019-03-06T18:29:59Z

@shaneknapp I have this for 2.4.1 for testing as well. Tests fail due to the minikube issue.

skonto · 2019-03-06T18:32:20Z

jenkins test this please

SparkQA · 2019-03-06T18:39:27Z

Test build #103107 has finished for PR 23993 at commit 976a2eb.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-06T18:47:38Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8569/

skonto · 2019-03-06T18:52:02Z

I removed hadoop-2.6 deps but this still fails (although I tried to align the PR with #23814).
One question is whethere to add logging-interceptor-3.12.0.jar to the deps.

skonto · 2019-03-06T18:54:16Z

...s/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/backend/minikube/Minikube.scala

we need this in order to support latest minikube versions.

skonto · 2019-03-06T18:55:42Z

...ernetes/core/src/main/scala/org/apache/spark/deploy/k8s/submit/LoggingPodStatusWatcher.scala

this part of the codebase has changed on master, but I am not porting back all the changes, just fixing this locally.

SparkQA · 2019-03-06T19:00:07Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8569/

Jeffwan · 2019-03-06T19:05:00Z

@skonto Thanks! Good to know it's due to minikube issue. I was stucked on this for a while. Glad you figure it out

dongjoon-hyun

Hi, @skonto . Please use the script to change dependency manifests.

$ dev/test-dependencies.sh --replace-manifest

Currently, the following is missing in this PR.

-logging-interceptor-3.8.1.jar
+logging-interceptor-3.12.0.jar

SparkQA · 2019-03-06T19:37:57Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8570/

SparkQA · 2019-03-06T19:49:26Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8570/

skonto · 2019-03-06T20:17:38Z

tests pass as reported here: https://issues.apache.org/jira/browse/SPARK-26742

SparkQA · 2019-03-07T00:10:16Z

Test build #103108 has finished for PR 23993 at commit 5de30f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

@shaneknapp

I dismissed my review since this will be handled by @shaneknapp simultaneously.

shaneknapp · 2019-03-12T00:16:57Z

@gatorsmile -- am i missing something, or did this not get merged?

shaneknapp · 2019-03-13T18:19:22Z

ping @gatorsmile

gatorsmile · 2019-03-13T19:20:07Z

This is not merged. I accidentally closed it.

SparkQA · 2019-03-13T19:20:33Z

Test build #103452 has started for PR 23993 at commit 5de30f5.

SparkQA · 2019-03-13T19:36:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8872/

SparkQA · 2019-03-13T19:47:00Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8872/

shaneknapp · 2019-03-13T19:56:47Z

k8s test failure is expected

shaneknapp · 2019-03-13T22:08:55Z

test this please

shaneknapp · 2019-03-13T22:17:22Z

test this please

SparkQA · 2019-03-13T22:31:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/

SparkQA · 2019-03-13T22:42:02Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/8875/

shaneknapp · 2019-03-13T22:50:18Z

we're going to wait to see if the regular non-k8s PRB job passes before merging.

SparkQA · 2019-03-14T02:53:52Z

Test build #103456 has finished for PR 23993 at commit 5de30f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-03-14T02:58:33Z

Test build #103458 has finished for PR 23993 at commit 5de30f5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-03-14T16:29:24Z

Merging to 2.4.

## What changes were proposed in this pull request? Updates client version and fixes some related issues. ## How was this patch tested? Tested with the latest minikube version and k8s 1.13. KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. Run completed in 4 minutes, 20 seconds. Total number of tests run: 14 Suites: completed 2, aborted 0 Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0 All tests passed. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM 2.4.2-SNAPSHOT ............ SUCCESS [ 2.980 s] [INFO] Spark Project Tags ................................. SUCCESS [ 2.880 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 1.954 s] [INFO] Spark Project Networking ........................... SUCCESS [ 3.369 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 1.791 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 1.845 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 3.725 s] [INFO] Spark Project Core ................................. SUCCESS [ 23.572 s] [INFO] Spark Project Kubernetes Integration Tests 2.4.2-SNAPSHOT SUCCESS [04:25 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:08 min [INFO] Finished at: 2019-03-06T18:03:55Z [INFO] ------------------------------------------------------------------------ Closes #23993 from skonto/fix-k8s-version. Authored-by: Stavros Kontopoulos <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>

yuhuali1989 · 2019-06-12T03:50:26Z

Hi，
I report an issue related with kubernetes-client 4.1.2 which introduce non daemon thread which block jvm exit.
Any Idea to fix it ? Thanks.

More details please check https://issues.apache.org/jira/browse/SPARK-27812

By the way, I believe update kubernetes-client is a necessary choice.
Without this version, I have to fix another issue with aws eks.

Jeffwan · 2019-06-12T19:10:15Z

@yuhuali1989 Could I know more details about issue with aws eks?
I think maybe we can patch SDK client and then upgrade client in next 2.x release?

## What changes were proposed in this pull request? Updates client version and fixes some related issues. ## How was this patch tested? Tested with the latest minikube version and k8s 1.13. KubernetesSuite: - Run SparkPi with no resources - Run SparkPi with a very long application name. - Use SparkLauncher.NO_RESOURCE - Run SparkPi with a master URL without a scheme. - Run SparkPi with an argument. - Run SparkPi with custom labels, annotations, and environment variables. - Run extraJVMOptions check on driver - Run SparkRemoteFileTest using a remote data file - Run SparkPi with env and mount secrets. - Run PySpark on simple pi.py example - Run PySpark with Python2 to test a pyfiles example - Run PySpark with Python3 to test a pyfiles example - Run PySpark with memory customization - Run in client mode. Run completed in 4 minutes, 20 seconds. Total number of tests run: 14 Suites: completed 2, aborted 0 Tests: succeeded 14, failed 0, canceled 0, ignored 0, pending 0 All tests passed. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM 2.4.2-SNAPSHOT ............ SUCCESS [ 2.980 s] [INFO] Spark Project Tags ................................. SUCCESS [ 2.880 s] [INFO] Spark Project Local DB ............................. SUCCESS [ 1.954 s] [INFO] Spark Project Networking ........................... SUCCESS [ 3.369 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 1.791 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 1.845 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 3.725 s] [INFO] Spark Project Core ................................. SUCCESS [ 23.572 s] [INFO] Spark Project Kubernetes Integration Tests 2.4.2-SNAPSHOT SUCCESS [04:25 min] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:08 min [INFO] Finished at: 2019-03-06T18:03:55Z [INFO] ------------------------------------------------------------------------ Closes apache#23993 from skonto/fix-k8s-version. Authored-by: Stavros Kontopoulos <[email protected]> Signed-off-by: Marcelo Vanzin <[email protected]>

skonto force-pushed the fix-k8s-version branch from 5e6e770 to 976a2eb Compare March 6, 2019 18:29

skonto commented Mar 6, 2019

View reviewed changes

dongjoon-hyun previously requested changes Mar 6, 2019

View reviewed changes

update k8s client to 4.1.2

5de30f5

skonto force-pushed the fix-k8s-version branch from 976a2eb to 5de30f5 Compare March 6, 2019 19:20

shaneknapp mentioned this pull request Mar 6, 2019

[SPARK-26742][K8S] Update Kubernetes-Client version to 4.1.2 #23814

Closed

dongjoon-hyun self-requested a review March 6, 2019 22:38

dongjoon-hyun removed their request for review March 7, 2019 00:27

Jeffwan mentioned this pull request Mar 7, 2019

[SPARK-26742][K8S] Update Kubernetes-Client version to 4.1.2 #24002

Closed

gatorsmile closed this Mar 9, 2019

gatorsmile reopened this Mar 13, 2019

vanzin changed the title ~~[SPARK-26742][K8s]Update k8s client version to 4.1.2 for Spark 2.4.1~~ [SPARK-26742][K8s][branch-2.4] Update k8s client version to 4.1.2 Mar 14, 2019

vanzin closed this Mar 14, 2019

[SPARK-26742][K8s][branch-2.4] Update k8s client version to 4.1.2 #23993

[SPARK-26742][K8s][branch-2.4] Update k8s client version to 4.1.2 #23993

Uh oh!

Conversation

skonto commented Mar 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

skonto commented Mar 6, 2019

Uh oh!

skonto commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

skonto commented Mar 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skonto Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

skonto Mar 6, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

Jeffwan commented Mar 6, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 6, 2019

Uh oh!

skonto commented Mar 6, 2019

Uh oh!

SparkQA commented Mar 7, 2019

Uh oh!

shaneknapp commented Mar 12, 2019

Uh oh!

shaneknapp commented Mar 13, 2019

Uh oh!

gatorsmile commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 13, 2019

Uh oh!

shaneknapp commented Mar 13, 2019

Uh oh!

shaneknapp commented Mar 13, 2019

Uh oh!

shaneknapp commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 13, 2019

Uh oh!

shaneknapp commented Mar 13, 2019

Uh oh!

SparkQA commented Mar 14, 2019

Uh oh!

SparkQA commented Mar 14, 2019

Uh oh!

vanzin commented Mar 14, 2019

Uh oh!

yuhuali1989 commented Jun 12, 2019

Uh oh!

Jeffwan commented Jun 12, 2019

skonto commented Mar 6, 2019 •

edited

Loading

skonto commented Mar 6, 2019 •

edited

Loading