[SPARK-43540][K8S][CORE] Add working directory into classpath on the driver in K8S cluster mode #41201

turboFei · 2023-05-17T14:42:49Z

What changes were proposed in this pull request?

Adding working directory into classpath on the driver in K8S cluster mode.

Why are the changes needed?

After #37417, the spark.files, spark.jars are placed in the working directory.
But seems that the spark context classloader can not access them because they are not in the classpath by default.
This pr adds the current working directory into classpath, so that the spark.files, spark.jars placed in the working directory can be accessible by the classloader.
For example, the hive-site.xml uploaded by spark.files.

Does this PR introduce any user-facing change?

yes, users do not need to add the working directory into spark classpath manually.

How was this patch tested?

UT.

core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala

dongjoon-hyun · 2023-05-17T15:31:32Z

cc @pralabhkumar and @holdenk from #37417

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

holdenk · 2023-05-17T17:54:27Z

+1 looks reasonable module the existing suggestions (clean up the logging + tighten the test). Thanks for making this PR :)

pralabhkumar · 2023-05-18T02:58:12Z

LGTM .

advancedxy · 2023-05-18T08:01:15Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

It might affect other modes. A better place would be the entrypoint.sh for resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh to include work-dir into the classpath.

here we have checked the k8s cluster mode

if (isKubernetesClusterModeDriver) { ... }

This doesn't add current working dir to executor's class path right?

Just checked with yarn's behavior, yarn add CWD to both driver and executor. And it puts CWD before localized SPARK_CONF and HADOOP_CONF.

See

spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala

Line 1442 in 014685c

addClasspathEntry(Environment.PWD.$$(), env)

To get the similar behavior, I believe it would be easier to leverage the entrypoint.sh here when running on K8S.

will check it

how about now? @advancedxy
I checked the code, for driver, if just leverage the entrypoint.sh, it is difficult to keep the behavior as mentioned above.

So I just leverage the entrypoint.sh for executor.

turboFei · 2023-05-23T03:54:24Z

gentle ping @dongjoon-hyun @holdenk @pralabhkumar @pan3793 for the latest change, thanks a lot.

pan3793

Please change the title, not only Driver.

dongjoon-hyun · 2023-05-30T22:23:03Z

resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh

Sorry, but why this is prepended? For this specific part, I'm strong negative because this could have a side-effect like spark.executor.userClassPathFirst=true. As we know, we don't recommend userClassPathFirst at all.

thanks for the correction，will check and address it.

Sorry, but why this is prepended? For this specific part, I'm strong negative because this could have a side-effect like spark.executor.userClassPathFirst=true. As we know, we don't recommend userClassPathFirst at all.

Hi @dongjoon-hyun, just to be clear, are you against for putting the PWD into executor's class path or for the PWD being the first in the class path?

In my opinion, to align with spark on yarn's behavior, PWD should be put in both driver and executor's class path. But I'm ok for it to be last or anywhere in the class path.

By the way, this pr put PWD(.) in the first of class path for driver, if you have concern about PWD being first, the driver may have the same issue here.

Ya. The second one. I'm only worrying about the prepending. :)

then, @turboFei would you mind to do some search to make sure which place the PWD should be put in the executor’s class path? sorry for the inconvenience.

sorry for the late reply, I checked the code.

For driver:

I think for driver, now the working directory is just placed before user jars, likes

--jars

localPrimaryResource

It should be fine.

and for executor classpath, I think that we can place the working directory at the tail of the classpath.

dongjoon-hyun · 2023-05-31T06:18:49Z

cc @advancedxy once more because of his #41201 (comment) . Now, the last commit reverted entrypoint.sh change .

…driver in K8S cluster mode

dongjoon-hyun

+1, LGTM. Thank you, @turboFei and all!

dongjoon-hyun · 2023-06-07T22:38:34Z

The sql - other tests failure is a known flaky one which is irrelevant to this one. Let me merge this PR.

turboFei · 2023-06-08T02:38:48Z

thanks all!

…driver in K8S cluster mode ### What changes were proposed in this pull request? Adding working directory into classpath on the driver in K8S cluster mode. ### Why are the changes needed? After apache#37417, the spark.files, spark.jars are placed in the working directory. But seems that the spark context classloader can not access them because they are not in the classpath by default. This pr adds the current working directory into classpath, so that the spark.files, spark.jars placed in the working directory can be accessible by the classloader. For example, the `hive-site.xml` uploaded by `spark.files`. ### Does this PR introduce _any_ user-facing change? yes, users do not need to add the working directory into spark classpath manually. ### How was this patch tested? UT. Closes apache#41201 from turboFei/work_dir_classpath. Authored-by: fwang12 <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Signed-off-by: Yikun Jiang <[email protected]>

github-actions bot added the CORE label May 17, 2023

dongjoon-hyun reviewed May 17, 2023

View reviewed changes

core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala Outdated Show resolved Hide resolved

pan3793 reviewed May 17, 2023

View reviewed changes

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala Outdated Show resolved Hide resolved

pan3793 approved these changes May 18, 2023

View reviewed changes

turboFei force-pushed the work_dir_classpath branch 2 times, most recently from 485a96d to 25305b4 Compare May 18, 2023 07:40

advancedxy reviewed May 18, 2023

View reviewed changes

github-actions bot added the KUBERNETES label May 22, 2023

advancedxy approved these changes May 26, 2023

View reviewed changes

pan3793 approved these changes May 26, 2023

View reviewed changes

This comment was marked as duplicate.

Sign in to view

turboFei changed the title ~~[SPARK-43540][K8S][CORE] Add working directory into classpath on the driver in K8S cluster mode~~ [SPARK-43540][K8S][CORE] Add working directory into classpath on the driver/executor in K8S cluster mode May 26, 2023

dongjoon-hyun reviewed May 30, 2023

View reviewed changes

github-actions bot removed the KUBERNETES label May 31, 2023

turboFei changed the title ~~[SPARK-43540][K8S][CORE] Add working directory into classpath on the driver/executor in K8S cluster mode~~ [SPARK-43540][K8S][CORE] Add working directory into classpath on the driverin K8S cluster mode May 31, 2023

turboFei changed the title ~~[SPARK-43540][K8S][CORE] Add working directory into classpath on the driverin K8S cluster mode~~ [SPARK-43540][K8S][CORE] Add working directory into classpath on the driver in K8S cluster mode May 31, 2023

turboFei added 7 commits June 5, 2023 12:41

[SPARK-43540][K8S][CORE] Add working directory into classpath on the …

58b808c

…driver in K8S cluster mode

minmize ut

413881b

logging format

5e9cfd3

getCanonicalFile for source and dest - retest please

33c287a

add into executor classpath

95f85d2

revert

97b61ea

add working directory at the tail of the classpath

11b5288

turboFei force-pushed the work_dir_classpath branch from fa6fa38 to 11b5288 Compare June 6, 2023 04:04

github-actions bot added the KUBERNETES label Jun 6, 2023

dongjoon-hyun approved these changes Jun 7, 2023

View reviewed changes

dongjoon-hyun closed this in 0c7b430 Jun 7, 2023

turboFei deleted the work_dir_classpath branch June 8, 2023 02:38

Yikun added a commit to Yikun/spark-docker that referenced this pull request Sep 14, 2023

Backport apache/spark#40392 and apache/spark#41201

24cbf40

Signed-off-by: Yikun Jiang <[email protected]>

[SPARK-43540][K8S][CORE] Add working directory into classpath on the driver in K8S cluster mode #41201

[SPARK-43540][K8S][CORE] Add working directory into classpath on the driver in K8S cluster mode #41201

Uh oh!

Conversation

turboFei commented May 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

dongjoon-hyun commented May 17, 2023

Uh oh!

Uh oh!

holdenk commented May 17, 2023

Uh oh!

pralabhkumar commented May 18, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei May 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei May 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

turboFei commented May 23, 2023

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented May 31, 2023

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Jun 7, 2023

Uh oh!

turboFei commented Jun 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

turboFei commented May 17, 2023 •

edited

Loading

turboFei May 18, 2023 •

edited

Loading

turboFei May 22, 2023 •

edited

Loading

turboFei commented Jun 8, 2023 •

edited

Loading