[SPARK-53157][CORE] Decouple driver and executor polling intervals #51885

ForVic · 2025-08-06T21:33:12Z

What changes were proposed in this pull request?

Add a config spark.driver.metrics.pollingInterval, and schedule driver polling interval / heartbeat at that schedule.

Why are the changes needed?

Decouple driver and executor heartbeat intervals. Due to sampling frequencies in memory metric reporting intervals we do not have a 100% accurate view of stats at drivers and executors. This is particularly observed at the driver, where we don't have the benefit of a larger sample size of metrics from N executors in application.

Here we can provide a way increase (or change in general) the rate of collection of metrics at the driver, to aid in overcoming the sampling problem, without requiring users to also increase executor heartbeat frequencies.

Does this PR introduce any user-facing change?

Yes, introduces a spark config

How was this patch tested?

Verified that metric collection was improved when sampling rates were increased, and verified that the number of events were expected when rate was changed.

Methodology for validating that increased driver heartbeat intervals would improve memory collection:

Using a 6gb driver heap, wrote a job to broadcast a table, gradually increasing the size of the table until OOM.
Increased driver memory to 10gb, large enough for the same broadcast to succeed.
Repeated this job and tracked the peak memory usage that was written to event log.
After repeated experiments, witnessed that the median peak heap typical usage was tracked at <=5GiB.
Added my change, and decreased the heartbeat interval.
Re-ran same jobs with 10gb heap, and saw that the typical peak memory usage tracked was ~8GiB, more accurately reflecting the increased memory needs.

Was this patch authored or co-authored using generative AI tooling?

No

ForVic · 2025-08-06T21:44:22Z

cc @mridulm @robreeves @shardulm94

mridulm

~~LGTM~~

Scratch that - added a comment below

mridulm

+CC @dongjoon-hyun for thoughts as well.

mridulm · 2025-08-24T20:35:55Z

core/src/main/scala/org/apache/spark/internal/config/package.scala

      .createWithDefaultString("10s")

+  private[spark] val DRIVER_HEARTBEAT_INTERVAL =
+    ConfigBuilder("spark.driver.heartbeatInterval")


Given this is mostly just for reporting metrics, I am more inclined to rename this from heartbeat to something which more clearly conveys the intent.

how about spark.driver.metricsReportingInterval

+1 for @mridulm 's comment.

Oh I see we already differentiate, spark.executor.metrics.pollingInterval, I will call this spark.driver.metrics.pollingInterval ?

dongjoon-hyun · 2025-08-26T20:45:22Z

Thank you for pinging me, @mridulm . I'm catching up the community reviews this week.

dongjoon-hyun

In general, the background of proposal sounds reasonable to me. I'll revisit later when the PR has the final code.

dongjoon-hyun · 2025-08-27T18:10:12Z

core/src/main/scala/org/apache/spark/internal/config/package.scala

+  private[spark] val DRIVER_METRICS_POLLING_INTERVAL =
+    ConfigBuilder("spark.driver.metrics.pollingInterval")
+      .version("4.1.0")
+      .fallbackConf(EXECUTOR_HEARTBEAT_INTERVAL)


Could you add more fields like spark.executor.metrics.pollingInterval, e.g., .doc and .timeConf? In addition, it would be great if we can place this new configuration to somewhere near the driver configs live instead of here.

spark/core/src/main/scala/org/apache/spark/internal/config/package.scala

Lines 369 to 376 in 86dad83

private[spark] val EXECUTOR_METRICS_POLLING_INTERVAL =

ConfigBuilder("spark.executor.metrics.pollingInterval")

.doc("How often to collect executor metrics (in milliseconds). " +

"If 0, the polling is done on executor heartbeats. " +

"If positive, the polling is done at this interval.")

.version("3.0.0")

.timeConf(TimeUnit.MILLISECONDS)

.createWithDefaultString("0")

Added doc string, but we can't add both .timeConf and .fallbackConf and if user has non-default EXECUTOR_HEARTBEAT_INTERVAL, we should fallback to matching behavior, thoughts?

core/src/main/scala/org/apache/spark/internal/config/package.scala

dongjoon-hyun

To @mridulm , sorry but I'm not sure this PR is going into the right direction. I'll leave this to you for the rest of reviews because it seems that LinkedIn requires this feature.

mridulm · 2025-09-01T05:00:09Z

Would love to hear your thoughts on the PR @dongjoon-hyun !
If this is not beneficial for the community, I would rather keep this internal than make a change which will not benefits all users.

pan3793

Increasing the heartbeat reporting frequency of the driver alone will improve the accuracy of driver metrics without introducing much overhead, and benefit diagnostic/automatic tuning tools.

mridulm · 2025-09-10T00:42:34Z

Thanks @pan3793.
I am inclined to merge this Dongjoon, unless you think it is not a great idea !
Will circle back to this later this week, based on inputs. Thanks !

mridulm · 2025-09-12T15:38:20Z

Merged to master.
Thanks for working on this @ForVic !
Thanks for the reviews and feedback @dongjoon-hyun , @pan3793 :-)

### What changes were proposed in this pull request? This PR aims to document newly added `core` module configurations as a part of Apache Spark 4.1.0 preparation. ### Why are the changes needed? To help the users use new features easily. - #47856 - #51130 - #51163 - #51604 - #51630 - #51708 - #51885 - #52091 - #52382 ### Does this PR introduce _any_ user-facing change? No behavior change because this is a documentation update. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52626 from dongjoon-hyun/SPARK-53926. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? Add a config `spark.driver.metrics.pollingInterval`, and schedule driver polling interval / heartbeat at that schedule. ### Why are the changes needed? Decouple driver and executor heartbeat intervals. Due to sampling frequencies in memory metric reporting intervals we do not have a 100% accurate view of stats at drivers and executors. This is particularly observed at the driver, where we don't have the benefit of a larger sample size of metrics from N executors in application. Here we can provide a way increase (or change in general) the rate of collection of metrics at the driver, to aid in overcoming the sampling problem, without requiring users to also increase executor heartbeat frequencies. ### Does this PR introduce _any_ user-facing change? Yes, introduces a spark config ### How was this patch tested? Verified that metric collection was improved when sampling rates were increased, and verified that the number of events were expected when rate was changed. Methodology for validating that increased driver heartbeat intervals would improve memory collection: 1. Using a 6gb driver heap, wrote a job to broadcast a table, gradually increasing the size of the table until OOM. 2. Increased driver memory to 10gb, large enough for the same broadcast to succeed. 3. Repeated this job and tracked the peak memory usage that was written to event log. 4. After repeated experiments, witnessed that the median peak heap typical usage was tracked at <=5GiB. 5. Added my change, and decreased the heartbeat interval. 6. Re-ran same jobs with 10gb heap, and saw that the typical peak memory usage tracked was ~8GiB, more accurately reflecting the increased memory needs. ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51885 from ForVic/vsunderl/driver_polling_interval. Authored-by: ForVic <[email protected]> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>

### What changes were proposed in this pull request? This PR aims to document newly added `core` module configurations as a part of Apache Spark 4.1.0 preparation. ### Why are the changes needed? To help the users use new features easily. - apache#47856 - apache#51130 - apache#51163 - apache#51604 - apache#51630 - apache#51708 - apache#51885 - apache#52091 - apache#52382 ### Does this PR introduce _any_ user-facing change? No behavior change because this is a documentation update. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52626 from dongjoon-hyun/SPARK-53926. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

[LIHADOOP-81008] Add a Driver Metrics Polling Interval (apache#462)

65c2553

github-actions bot added the CORE label Aug 6, 2025

change version introduced

4402779

ForVic marked this pull request as ready for review August 6, 2025 21:40

mridulm approved these changes Aug 24, 2025

View reviewed changes

mridulm reviewed Aug 24, 2025

View reviewed changes

mridulm self-requested a review August 24, 2025 20:36

dongjoon-hyun reviewed Aug 26, 2025

View reviewed changes

ForVic changed the title ~~[SPARK-53157][CORE] Decouple driver and executor heartbeat polling intervals~~ [SPARK-53157][CORE] Decouple driver and executor polling intervals Aug 26, 2025

Rename config to spark.driver.metrics.pollingInterval

ec588a0

dongjoon-hyun reviewed Aug 27, 2025

View reviewed changes

Add doc string

0d2031f

dongjoon-hyun reviewed Aug 27, 2025

View reviewed changes

core/src/main/scala/org/apache/spark/internal/config/package.scala Outdated Show resolved Hide resolved

dongjoon-hyun reviewed Aug 27, 2025

View reviewed changes

doc string

cedc2a4

pan3793 approved these changes Sep 1, 2025

View reviewed changes

mridulm closed this in 04a40ab Sep 12, 2025

dongjoon-hyun mentioned this pull request Oct 15, 2025

[SPARK-53926][DOCS] Document newly added core module configurations #52626

Closed

dongjoon-hyun changed the title ~~[SPARK-53157][CORE] Decouple driver and executor polling intervals~~ [SPARK-53157][CORE] Decouple driver and executor polling intervals Oct 15, 2025

	private[spark] val EXECUTOR_METRICS_POLLING_INTERVAL =
	ConfigBuilder("spark.executor.metrics.pollingInterval")
	.doc("How often to collect executor metrics (in milliseconds). " +
	"If 0, the polling is done on executor heartbeats. " +
	"If positive, the polling is done at this interval.")
	.version("3.0.0")
	.timeConf(TimeUnit.MILLISECONDS)
	.createWithDefaultString("0")

[SPARK-53157][CORE] Decouple driver and executor polling intervals #51885

[SPARK-53157][CORE] Decouple driver and executor polling intervals #51885

Uh oh!

Conversation

ForVic commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

ForVic commented Aug 6, 2025

Uh oh!

mridulm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mridulm left a comment

Choose a reason for hiding this comment

Uh oh!

mridulm Aug 24, 2025

Choose a reason for hiding this comment

Uh oh!

ForVic Aug 25, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

ForVic Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 26, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

ForVic Aug 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

mridulm commented Sep 1, 2025

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

mridulm commented Sep 10, 2025

Uh oh!

mridulm commented Sep 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ForVic commented Aug 6, 2025 •

edited

Loading

mridulm left a comment •

edited

Loading