[WIP][SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode #32397

ocworld · 2021-04-29T12:56:44Z

What changes were proposed in this pull request?

Supporting '--packages' in the k8s cluster mode

Why are the changes needed?

In spark 3, '--packages' in the k8s cluster mode is not supported. I expected that managing dependencies by using packages like spark 2.

Spark 2.4.5

https://github.com/apache/spark/blob/v2.4.5/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

     if (!isMesosCluster && !isStandAloneCluster) {
      // Resolve maven dependencies if there are any and add classpath to jars. Add them to py-files
      // too for packages that include Python code
      val resolvedMavenCoordinates = DependencyUtils.resolveMavenDependencies(
        args.packagesExclusions, args.packages, args.repositories, args.ivyRepoPath,
        args.ivySettingsPath)
      
      if (!StringUtils.isBlank(resolvedMavenCoordinates)) {
        args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)
        if (args.isPython || isInternal(args.primaryResource)) {
          args.pyFiles = mergeFileLists(args.pyFiles, resolvedMavenCoordinates)
        }
      } 
      
      // install any R packages that may have been passed through --jars or --packages.
      // Spark Packages may contain R source code inside the jar.
      if (args.isR && !StringUtils.isBlank(args.jars)) {
        RPackageUtils.checkAndBuildRPackage(args.jars, printStream, args.verbose)
      }
    }

Spark 3.0.2

https://github.com/apache/spark/blob/v3.0.2/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

       if (!StringUtils.isBlank(resolvedMavenCoordinates)) {
        // In K8s client mode, when in the driver, add resolved jars early as we might need
        // them at the submit time for artifact downloading.
        // For example we might use the dependencies for downloading
        // files from a Hadoop Compatible fs eg. S3. In this case the user might pass:
        // --packages com.amazonaws:aws-java-sdk:1.7.4:org.apache.hadoop:hadoop-aws:2.7.6
        if (isKubernetesClusterModeDriver) {
          val loader = getSubmitClassLoader(sparkConf)
          for (jar <- resolvedMavenCoordinates.split(",")) {
            addJarToClasspath(jar, loader)
          }
        } else if (isKubernetesCluster) {
          // We need this in K8s cluster mode so that we can upload local deps
          // via the k8s application, like in cluster mode driver
          childClasspath ++= resolvedMavenCoordinates.split(",")
        } else {
          args.jars = mergeFileLists(args.jars, resolvedMavenCoordinates)
          if (args.isPython || isInternal(args.primaryResource)) {
            args.pyFiles = mergeFileLists(args.pyFiles, resolvedMavenCoordinates)
          }
        }
      }

unlike spark2, in spark 3, jars are not added in any place.

Does this PR introduce any user-facing change?

Unlike spark 2, resolved jars are added not in cluster mode spark submit but in driver.

It's because in spark 3, the feature is added that is uploading jars with prefix "file://" to s3.
So, if resolved jars are added in spark submit, every jars from packages are uploading to s3! When I tested it, it is very bad experience to me.

How was this patch tested?

In my k8s environment, i tested the code.

ocworld · 2021-04-29T13:02:02Z

@AhnLab-OSS

ocworld · 2021-04-29T13:07:09Z

Can I see logs about failure of build and test?

dongjoon-hyun · 2021-05-01T23:10:44Z

Thank you for your contribution, @ocworld .

dongjoon-hyun · 2021-05-01T23:10:49Z

ok to test

dongjoon-hyun · 2021-05-01T23:14:08Z

Can I see logs about failure of build and test?

Apache Spark GitHub Action is running in your GitHub Account.

If you click the failure link, you can see the error log in your repo. The current failure occurs because your master branch is too outdate. Please follow the guideline to sync your master branch.

https://github.com/apache/spark/pull/32397/checks?check_run_id=2466811890

Unable to detect the workflow run for testing the changes in your PR.

If you did not enable GitHub Actions in your forked repository, please enable it. See also Disabling or limiting GitHub Actions for a repository for more details.
It is possible your branch is based on the old master branch in Apache Spark, please sync your branch to the latest master branch. For example as below:
git fetch upstream
git rebase upstream/master
git push origin YOUR_BRANCH --force

In addition, I also triggered Jenkins machine too. For those users, we have another test machine additionally.

SparkQA · 2021-05-02T00:30:20Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42657/

SparkQA · 2021-05-02T00:30:21Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42657/

ocworld · 2021-05-02T01:26:42Z

@dongjoon-hyun Thank you for your test and reply.
Though rebasing from upstream, checking build and test is failed now.
The jenkins build is also failed with logs.

Setting status of fe0a4b5625ffceaa6c81e9f63637214d08369642 to FAILURE with url https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42657/ and message: 'Build finished. '
FileNotFoundException means that the credentials Jenkins is using is probably wrong. Or the user account does not have write access to the repo.
org.kohsuke.github.GHFileNotFoundException: https://api.github.com/repos/apache/spark/statuses/fe0a4b5625ffceaa6c81e9f63637214d08369642 {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#create-a-commit-status"}
	at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:492)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:420)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:363)
	at org.kohsuke.github.Requester.fetch(Requester.java:74)
	at org.kohsuke.github.GHRepository.createCommitStatus(GHRepository.java:1906)
	at org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus.createCommitStatus(GhprbSimpleStatus.java:283)
	at org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus.onBuildComplete(GhprbSimpleStatus.java:241)
	at org.jenkinsci.plugins.ghprb.GhprbBuilds.onCompleted(GhprbBuilds.java:205)
	at org.jenkinsci.plugins.ghprb.GhprbBuildListener.onCompleted(GhprbBuildListener.java:28)
	at hudson.model.listeners.RunListener.fireCompleted(RunListener.java:209)
	at hudson.model.Run.execute(Run.java:1952)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:429)
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/apache/spark/statuses/fe0a4b5625ffceaa6c81e9f63637214d08369642
	at sun.reflect.GeneratedConstructorAccessor1029.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
	at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268)
	at org.kohsuke.github.GitHubHttpUrlConnectionClient$HttpURLConnectionResponseInfo.bodyStream(GitHubHttpUrlConnectionClient.java:197)
	at org.kohsuke.github.GitHubResponse$ResponseInfo.getBodyAsString(GitHubResponse.java:326)
	at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
	at org.kohsuke.github.Requester.lambda$fetch$1(Requester.java:74)
	at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:461)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:412)
	... 12 more
Caused by: java.io.FileNotFoundException: https://api.github.com/repos/apache/spark/statuses/fe0a4b5625ffceaa6c81e9f63637214d08369642
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:352)
	at org.kohsuke.github.GitHubHttpUrlConnectionClient.getResponseInfo(GitHubHttpUrlConnectionClient.java:69)
	at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:400)
	... 12 more

SparkQA · 2021-05-02T02:06:44Z

Test build #138136 has finished for PR 32397 at commit fe0a4b5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-02T02:29:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42658/

SparkQA · 2021-05-02T02:29:20Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42658/

SparkQA · 2021-05-02T04:13:11Z

Test build #138137 has finished for PR 32397 at commit 31783a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2021-05-03T04:54:15Z

@ocworld can you take a look https://github.com/apache/spark/pull/32397/checks?check_run_id=2484554770 and enable Github Actions in your forked repository?

ocworld · 2021-05-04T11:48:28Z

@HyukjinKwon Github actions setting is enabled as allow all actions in my forked repository.

ocworld · 2021-05-04T11:54:49Z

@HyukjinKwon

HyukjinKwon · 2021-05-04T12:26:20Z

Can you rebase and force push in this PR to retrigger the build?

ocworld · 2021-05-04T12:41:32Z

@HyukjinKwon I've already rebase and force push. So, now, everything is up-to-date.
After merging others' some pr, i'll retry it.

HyukjinKwon · 2021-05-04T12:45:17Z

did you do something like #32400 (comment)? The GitHub Actions should run the build in your forked repository but it seems not.

ocworld · 2021-05-04T12:54:19Z

@HyukjinKwon Oh! you're right. Workflow in actions tab was not activated. It is enabled now. Thank you.

SparkQA · 2021-05-05T02:48:01Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42679/

SparkQA · 2021-05-05T02:48:03Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42679/

SparkQA · 2021-05-05T03:03:09Z

Test build #138158 has finished for PR 32397 at commit bc366a9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mridulm · 2021-05-06T15:21:15Z

+CC @shardulm94, @xkrogen

dongjoon-hyun

Could you elaborate more about the procedure how you test in the PR description?

# How was this patch tested?
In my k8s environment, i tested the code.

The best way to protect this feature is having a test coverage at BasicTestsSuite or DepsTestsSuite. Do you think you can add a new test case test, @ocworld ?

SparkQA · 2021-05-11T14:58:27Z

Kubernetes integration test unable to build dist.

exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/42903/

ocworld · 2021-05-11T15:13:52Z

@dongjoon-hyun I reviewed "BasicTestsSuite" and "DepsTestsSuite"

I think it is hard to add test case in it.

It's because it is needed to get spark.jars in the running application's spark context on the driver pod. I don't have any idea about it, now.

Do you have any idea about test?

SparkQA · 2021-05-11T16:52:45Z

Test build #138380 has finished for PR 32397 at commit 76bdccc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-05-28T00:30:11Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43556/

SparkQA · 2021-08-01T03:17:53Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46433/

SparkQA · 2021-08-20T00:41:26Z

Test build #142659 has finished for PR 32397 at commit 554127b.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2021-08-20T01:21:43Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47160/

SparkQA · 2021-08-20T02:15:35Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47160/

GaruGaru · 2021-11-02T16:23:02Z

Any update on this ?

grcanosa · 2022-02-23T19:24:47Z

Any news on when this is going to be merged?

ocworld · 2022-02-25T13:40:40Z

Any news on when this is going to be merged?

It is not merged now. I've tried to add test.

github-actions · 2022-06-06T00:17:26Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

rahulbhatia2702 · 2022-09-14T23:10:46Z

is there a solution for this?

AlexandrePieroux · 2022-10-20T14:53:22Z

is there a solution for this?

In my case I just build the docker images (from the official repo) with the dependencies included by hand.

ex:
FROM apache/spark-py:v3.3.0
ARG spark_uid=185
USER 0
RUN chown ${spark_uid} -R /opt/spark

then

RUN curl https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.12/3.1.2/spark-sql-kafka-0-10_2.12-3.1.2.jar --output /opt/spark/jars/spark-sql-kafka-0-10_2.12-3.1.2.jar

and at the end

USER ${spark_uid}

Still looking for the day this is included in spark release.

jbguerraz · 2022-11-21T11:31:31Z

@dongjoon-hyun may be this PR could be re-opened and stale removed since this is still an issue ? Thank you!

holdenk · 2022-11-28T23:40:00Z

The repo that submitted the PR is deleted so we can't re-open it. But feel free to recreate and go for it.

ocworld · 2022-11-29T00:04:36Z

I'll reopen the pr soon.

ocworld · 2022-11-29T00:33:35Z

@jbguerraz @holdenk The pr is recreated in #38828

ulan-yisaev · 2022-12-19T20:57:42Z

RUN curl https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_2.12/3.1.2/spark-sql-kafka-0-10_2.12-3.1.2.jar --output /opt/spark/jars/spark-sql-kafka-0-10_2.12-3.1.2.jar

Thank you so much for the workaround, and it's a pity that this bug took two days of my life :)

ocworld · 2023-01-24T15:23:55Z

@jbguerraz @GaruGaru The pr(#38828) is merged now

jbguerraz · 2023-01-24T19:50:32Z

Thank you @ocworld :-)

github-actions bot added the CORE label Apr 29, 2021

ocworld mentioned this pull request May 6, 2021

On Spark 3.x, jars listed in spark.jars and spark.jars.packages are not added to sparkContext kubeflow/spark-operator#1226

Closed

dongjoon-hyun reviewed May 6, 2021

View reviewed changes

Keunhyun Oh and others added 3 commits August 20, 2021 09:23

[SPARK-35084] supporting "--packages" in k8s cluster mode

c961bde

Add a commit to trigger the test

8f697c4

Trigger the build

554127b

GaruGaru mentioned this pull request Nov 2, 2021

Application Dependency not working (spec.deps.packages is ignored) and (sparkConf:spark.jars.packages not propagated to executor) kubeflow/spark-operator#1132

Open

ocworld added 3 commits December 28, 2021 20:33

Merge branch 'apache:master' into SPARK-35084-k8s-packages

e4cc8f8

Merge branch 'apache:master' into SPARK-35084-k8s-packages

c87ecf3

Merge branch 'apache:master' into SPARK-35084-k8s-packages

f2ebf58

ocworld changed the title ~~[SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode~~ [WIP][SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode Feb 9, 2022

Merge branch 'apache:master' into SPARK-35084-k8s-packages

32400c8

github-actions bot added the Stale label Jun 6, 2022

github-actions bot closed this Jun 7, 2022

roczei mentioned this pull request Nov 30, 2022

[SPARK-35084][CORE] Fixing --packages in k8s client mode with the driver running inside a POD #38828

Closed

[WIP][SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode #32397

[WIP][SPARK-35084][CORE] Spark 3: supporting "--packages" in k8s cluster mode #32397

Uh oh!

Conversation

ocworld commented Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

ocworld commented Apr 29, 2021

Uh oh!

ocworld commented Apr 29, 2021

Uh oh!

dongjoon-hyun commented May 1, 2021

Uh oh!

dongjoon-hyun commented May 1, 2021

Uh oh!

dongjoon-hyun commented May 1, 2021

Uh oh!

SparkQA commented May 2, 2021

Uh oh!

SparkQA commented May 2, 2021

Uh oh!

ocworld commented May 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented May 2, 2021

Uh oh!

SparkQA commented May 2, 2021

Uh oh!

SparkQA commented May 2, 2021

Uh oh!

SparkQA commented May 2, 2021

Uh oh!

HyukjinKwon commented May 3, 2021

Uh oh!

ocworld commented May 4, 2021

Uh oh!

ocworld commented May 4, 2021

Uh oh!

HyukjinKwon commented May 4, 2021

Uh oh!

ocworld commented May 4, 2021

Uh oh!

HyukjinKwon commented May 4, 2021

Uh oh!

ocworld commented May 4, 2021

Uh oh!

SparkQA commented May 5, 2021

Uh oh!

SparkQA commented May 5, 2021

Uh oh!

SparkQA commented May 5, 2021

Uh oh!

mridulm commented May 6, 2021

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 11, 2021

Uh oh!

ocworld commented May 11, 2021

Uh oh!

SparkQA commented May 11, 2021

Uh oh!

SparkQA commented May 28, 2021

Uh oh!

SparkQA commented Aug 1, 2021

Uh oh!

SparkQA commented Aug 20, 2021

Uh oh!

SparkQA commented Aug 20, 2021

Uh oh!

SparkQA commented Aug 20, 2021

Uh oh!

GaruGaru commented Nov 2, 2021

Uh oh!

grcanosa commented Feb 23, 2022

ocworld commented Apr 29, 2021 •

edited

Loading

ocworld commented May 2, 2021 •

edited

Loading

AlexandrePieroux commented Oct 20, 2022 •

edited

Loading