[SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods #40392

dongjoon-hyun · 2023-03-13T06:23:35Z

What changes were proposed in this pull request?

Like SPARK_EXECUTOR_POD_IP, this PR aims to add a new environment variable ENV_DRIVER_POD_IP to all executor pods.

$ kubectl get pod pi-exec-1 -oyaml | grep -C1 SPARK_DRIVER_POD_IP
      value: "0"
    - name: SPARK_DRIVER_POD_IP
      value: 10.1.0.99

Why are the changes needed?

This is helpful for some executor pods to connect driver pods via IP.

Does this PR introduce any user-facing change?

No, this is a new environment variable.

How was this patch tested?

Pass the CIs with the newly added test case.

dongjoon-hyun · 2023-03-13T08:57:32Z

Hi, @HyukjinKwon . Could you review this PR when you have some time?

pan3793 · 2023-03-13T09:43:52Z

... for some executor pods to connect driver pods via IP.

Hi @dongjoon-hyun, I think it's quite useful, but in #39160 (review), you left a concern

... I have a concern. Currently, Apache Spark uses K8s Service entity via DriverServiceFeatureStep to access Spark driver pod in K8s environment.

do you still concern that now?

dongjoon-hyun · 2023-03-13T10:20:10Z

@pan3793 .

The goal of PR is different from your PR's goal.

Your PR tried to add SPARK_DRIVER_POD_NAME to Driver Pod to expose it to 3rd party pods.
This PR aims to add SPARK_DRIVER_POD_IP to Executor Pod in order to help internal communications between Spark executors and Spark driver.

In addition, this is a kind of propagation of the information from the driver pod to the executor pods instead of exposing the executor pods' internal information.

dongjoon-hyun · 2023-03-13T15:38:47Z

Hi, @viirya . Could you review this PR when you have some time?

viirya · 2023-03-13T16:24:56Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Constants.scala

  val UI_PORT_NAME = "spark-ui"

  // Environment Variables
+  val ENV_DRIVER_POD_IP = "SPARK_DRIVER_POD_IP"


Is it different to DRIVER_HOST_ADDRESS? I saw K8s uses DRIVER_HOST_ADDRESS to derive driver url for env var ENV_DRIVER_URL.

Thank you for review, @viirya .

In K8s, DRIVER_HOST_ADDRESS is protected by DriverServiceFeatureStep here.

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala

Line 109 in 7fccd67

val DRIVER_HOST_KEY = config.DRIVER_HOST_ADDRESS.key

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala

Lines 38 to 40 in 7fccd67

require(kubernetesConf.getOption(DRIVER_HOST_KEY).isEmpty,

s"$DRIVER_HOST_KEY is not supported in Kubernetes mode, as the driver's hostname will be " +

"managed via a Kubernetes service.")

It's because we inject like this systematically.

spark/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverServiceFeatureStep.scala

Lines 67 to 68 in 7fccd67

val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"

Map(DRIVER_HOST_KEY -> driverHostname,

However, when DNS doesn't work, we need IP which is unknown from the executor pods so far.

viirya

It is not used in this PR, will you make other change to k8s executor pods to use this env var?

dongjoon-hyun · 2023-03-13T17:05:44Z

Yes, correct, @viirya ! Thank you for the approval.

dongjoon-hyun · 2023-03-13T17:10:15Z

Merged to master for Apache Spark 3.5.0.

… pods ### What changes were proposed in this pull request? Like `SPARK_EXECUTOR_POD_IP`, this PR aims to add a new environment variable `ENV_DRIVER_POD_IP` to all executor pods. ```bash $ kubectl get pod pi-exec-1 -oyaml | grep -C1 SPARK_DRIVER_POD_IP value: "0" - name: SPARK_DRIVER_POD_IP value: 10.1.0.99 ``` ### Why are the changes needed? This is helpful for some executor pods to connect driver pods via IP. ### Does this PR introduce _any_ user-facing change? No, this is a new environment variable. ### How was this patch tested? Pass the CIs with the newly added test case. Closes apache#40392 from dongjoon-hyun/SPARK-42769. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

Signed-off-by: Yikun Jiang <[email protected]>

…ARK_DRIVER_POD_IP}}:{{SPARK_UI_PORT}}` ### Why are the changes needed? We are using [virtual-kubelet](https://github.com/virtual-kubelet/virtual-kubelet) for spark on kubernetes, and spark kubernetes pods would be allocated across kubernetes clusters. And we use the driver POD ip as driver host, see apache/spark#40392, which is supported since spark-3.5. The kubernetes context and namespace are virtual and we can not build the app URL by spark driver svc. And the spark driver pod IP is accessible for our use case, so raise this PR to build the spark app url by spark driver pod id and spark ui port. ### How was this patch tested? UT. <img width="1532" height="626" alt="image" src="https://github.com/user-attachments/assets/5cb54602-9e79-40b7-b51c-0b873c17560b" /> <img width="710" height="170" alt="image" src="https://github.com/user-attachments/assets/6d1c9580-62d6-423a-a04f-dc6cdcee940a" /> ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7141 from turboFei/app_url_v2. Closes #7141 1277952 [Wang, Fei] VAR d15e6be [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala 1535e00 [Wang, Fei] spark driver pod ip Lead-authored-by: Wang, Fei <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

…//{{SPARK_DRIVER_POD_IP}}:{{SPARK_UI_PORT}}` ### Why are the changes needed? We are using [virtual-kubelet](https://github.com/virtual-kubelet/virtual-kubelet) for spark on kubernetes, and spark kubernetes pods would be allocated across kubernetes clusters. And we use the driver POD ip as driver host, see apache/spark#40392, which is supported since spark-3.5. The kubernetes context and namespace are virtual and we can not build the app URL by spark driver svc. And the spark driver pod IP is accessible for our use case, so raise this PR to build the spark app url by spark driver pod id and spark ui port. ### How was this patch tested? UT. <img width="1532" height="626" alt="image" src="https://github.com/user-attachments/assets/5cb54602-9e79-40b7-b51c-0b873c17560b" /> <img width="710" height="170" alt="image" src="https://github.com/user-attachments/assets/6d1c9580-62d6-423a-a04f-dc6cdcee940a" /> ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#7141 from turboFei/app_url_v2. Closes apache#7141 1277952 [Wang, Fei] VAR d15e6be [Cheng Pan] Update kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala 1535e00 [Wang, Fei] spark driver pod ip Lead-authored-by: Wang, Fei <[email protected]> Co-authored-by: Cheng Pan <[email protected]> Signed-off-by: Cheng Pan <[email protected]>

[SPARK-42769][K8S] Add ENV_DRIVER_POD_IP env variable to executor pods

27174e5

github-actions bot added the KUBERNETES label Mar 13, 2023

Revise test case name

c153ba2

dongjoon-hyun changed the title ~~[SPARK-42769][K8S] Add ENV_DRIVER_POD_IP env variable to executor pods~~ [SPARK-42769][K8S] Add SPARK_DRIVER_POD_IP env variable to executor pods Mar 13, 2023

pan3793 mentioned this pull request Mar 13, 2023

[SPARK-41667][K8S] Expose env var SPARK_DRIVER_POD_NAME in Driver Pod #39160

Closed

viirya reviewed Mar 13, 2023

View reviewed changes

viirya approved these changes Mar 13, 2023

View reviewed changes

dongjoon-hyun closed this in 1d2c338 Mar 13, 2023

dongjoon-hyun deleted the SPARK-42769 branch March 13, 2023 17:10

Yikun added a commit to Yikun/spark-docker that referenced this pull request Sep 14, 2023

Backport apache/spark#40392

c4ffb7c

Signed-off-by: Yikun Jiang <[email protected]>

Yikun added a commit to Yikun/spark-docker that referenced this pull request Sep 14, 2023

Backport apache/spark#40392 and apache/spark#41201

24cbf40

Signed-off-by: Yikun Jiang <[email protected]>

turboFei mentioned this pull request Jul 18, 2025

Support to get spark app url with pattern http://{{SPARK_DRIVER_POD_IP}}:{{SPARK_UI_PORT}} apache/kyuubi#7141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods #40392

[SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods #40392

Uh oh!

dongjoon-hyun commented Mar 13, 2023 •

edited

Loading

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

pan3793 commented Mar 13, 2023

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

viirya Mar 13, 2023

Uh oh!

dongjoon-hyun Mar 13, 2023

Uh oh!

viirya left a comment

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	require(kubernetesConf.getOption(DRIVER_HOST_KEY).isEmpty,
	s"$DRIVER_HOST_KEY is not supported in Kubernetes mode, as the driver's hostname will be " +
	"managed via a Kubernetes service.")

	val driverHostname = s"$resolvedServiceName.${kubernetesConf.namespace}.svc"
	Map(DRIVER_HOST_KEY -> driverHostname,

[SPARK-42769][K8S] Add SPARK_DRIVER_POD_IP env variable to executor pods #40392

[SPARK-42769][K8S] Add SPARK_DRIVER_POD_IP env variable to executor pods #40392

Uh oh!

Conversation

dongjoon-hyun commented Mar 13, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

pan3793 commented Mar 13, 2023

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

viirya Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Mar 13, 2023

Choose a reason for hiding this comment

Uh oh!

viirya left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

dongjoon-hyun commented Mar 13, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods #40392

[SPARK-42769][K8S] Add `SPARK_DRIVER_POD_IP` env variable to executor pods #40392

dongjoon-hyun commented Mar 13, 2023 •

edited

Loading