[SPARK-45771][CORE] Enable `spark.eventLog.rolling.enabled` by default #43638

dongjoon-hyun · 2023-11-02T16:01:54Z

What changes were proposed in this pull request?

This PR aims to enable spark.eventLog.rolling.enabled by default for Apache Spark 4.0.0.

Why are the changes needed?

Since Apache Spark 3.0.0, we have been using event log rolling not only for long-running jobs, but also for some failed jobs to archive the partial event logs incrementally.

[SPARK-28869][CORE] Roll over event log files #25670

Does this PR introduce any user-facing change?

No because spark.eventLog.enabled is disabled by default.
For the users with spark.eventLog.enabled=true, yes, spark-events directory will have different layouts. However, all 3.3+ Spark History Server can read both old and new event logs. I believe that the event log users are already using this configuration to avoid the loss of event logs for long-running jobs and some failed jobs.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun · 2023-11-02T16:44:16Z

core/src/test/scala/org/apache/spark/deploy/history/EventLogTestHelper.scala

This is consistent with the existing function description.

spark/core/src/test/scala/org/apache/spark/deploy/history/EventLogTestHelper.scala

Line 35 in 5970d35

* Get a SparkConf with event logging enabled. It doesn't enable rolling event logs, so caller

dongjoon-hyun · 2023-11-02T19:36:50Z

AppVeyor failure (SparkR) is irrelevant to this PR.

Could you review this PR when you have some time, @viirya?

viirya · 2023-11-02T19:55:45Z

core/src/test/scala/org/apache/spark/deploy/history/EventLogFileWritersSuite.scala

-    buildWriterAndVerify(conf, classOf[SingleEventLogFileWriter])
+    buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])

    conf.set(EVENT_LOG_ENABLE_ROLLING, true)


Is it redundant then?

Suggested change

conf.set(EVENT_LOG_ENABLE_ROLLING, true)

Or we want to:

Suggested change

conf.set(EVENT_LOG_ENABLE_ROLLING, true)

conf.set(EVENT_LOG_ENABLE_ROLLING, false)

buildWriterAndVerify(conf, classOf[SingleEventLogFileWriter])

viirya · 2023-11-02T19:56:06Z

core/src/test/scala/org/apache/spark/deploy/history/EventLogFileWritersSuite.scala

+    buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])

    conf.set(EVENT_LOG_ENABLE_ROLLING, true)
    buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])


Suggested change

buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])

viirya · 2023-11-02T19:57:27Z

core/src/test/scala/org/apache/spark/scheduler/EventLoggingListenerSuite.scala

  test("SPARK-31764: isBarrier should be logged in event log") {
    val conf = new SparkConf()
    conf.set(EVENT_LOG_ENABLED, true)
+    conf.set(EVENT_LOG_ENABLE_ROLLING, false)


Is it failed without setting to false?

Yes, this test case tries to read the event log file.

dongjoon-hyun · 2023-11-02T20:08:03Z

Just for your confirmation. I keep the existing test structure.

    buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])

    conf.set(EVENT_LOG_ENABLE_ROLLING, true)
    buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])

    conf.set(EVENT_LOG_ENABLE_ROLLING, false)
    buildWriterAndVerify(conf, classOf[SingleEventLogFileWriter])

Do you mean to simplify like the following.

    buildWriterAndVerify(conf, classOf[RollingEventLogFilesWriter])

    conf.set(EVENT_LOG_ENABLE_ROLLING, false)
    buildWriterAndVerify(conf, classOf[SingleEventLogFileWriter])

viirya · 2023-11-02T20:13:16Z

Oh, got it, existing one looks good. From the diff, I cannot see it so I thought SingleEventLogFileWriter isn't tested.

dongjoon-hyun · 2023-11-02T20:13:40Z

Thank you for your confirmation! Merged to master for Apache Spark 4.0.0.

### What changes were proposed in this pull request? This PR aims to enable `spark.eventLog.rolling.enabled` by default for Apache Spark 4.0.0. ### Why are the changes needed? Since Apache Spark 3.0.0, we have been using event log rolling not only for **long-running jobs**, but also for **some failed jobs** to archive the partial event logs incrementally. - apache#25670 ### Does this PR introduce _any_ user-facing change? - No because `spark.eventLog.enabled` is disabled by default. - For the users with `spark.eventLog.enabled=true`, yes, `spark-events` directory will have different layouts. However, all 3.3+ `Spark History Server` can read both old and new event logs. I believe that the event log users are already using this configuration to avoid the loss of event logs for long-running jobs and some failed jobs. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#43638 from dongjoon-hyun/SPARK-45771. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

### What changes were proposed in this pull request? Add `Spark History Server` example. ### Why are the changes needed? Since Apache Spark 4.0, Spark rolls the event logs by default and compressed them by default. - apache/spark#43638 - apache/spark#43036 However, we still need more configurations to allow SHS manages the event log directories. This PR aims to provide an example of `Spark History Server` with the configuration. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #249 from dongjoon-hyun/SPARK-52481. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

…in `History Server` ### What changes were proposed in this pull request? This PR aims to support `On-Demand Log Loading` in `History Server` by looking up the **rolling event log locations** even Spark listing didn't finish to load the event log files. ```scala val EVENT_LOG_ROLLING_ON_DEMAND_LOAD_ENABLED = ConfigBuilder("spark.history.fs.eventLog.rolling.onDemandLoadEnabled") .doc("Whether to look up rolling event log locations on demand manner before listing files.") .version("4.1.0") .booleanConf .createWithDefault(true) ``` Previously, Spark History Server will show `Application ... Not Found` page if a job is requested before scanning it even if the file exists in the correct location. So, this PR doesn't introduce any regressions because this aims to introduce a kind of fallback logic to improve error handling . <img width="686" height="359" alt="Screenshot 2025-07-22 at 14 08 21" src="https://github.com/user-attachments/assets/fccb413c-5a57-4918-86c0-28ae81d54873" /> ### Why are the changes needed? Since Apache Spark 3.0, we have been using event log rolling not only for **long-running jobs**, but also for **some failed jobs** to archive the partial event logs incrementally. - #25670 Since Apache Spark 4.0, event log rolling is enabled by default. - #43638 On top of that, this PR aims to reduce storage cost at Apache Spark 4.1. By supporting `On-Demand Loading for rolled event logs`, we can use larger values for `spark.history.fs.update.interval` instead of the default `10s`. Although Spark History logs are consumed in various ways, It has a big benefit because most of successful Spark jobs's logs are not visited by the users. ### Does this PR introduce _any_ user-facing change? No. This is a new feature. ### How was this patch tested? Pass the CIs with newly added test case. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #51604 from dongjoon-hyun/SPARK-52914. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added the CORE label Nov 2, 2023

dongjoon-hyun changed the title ~~[SPARK-45771][CORE] Enable spark.eventLog.rolling.enabled by default~~ [SPARK-45771][CORE] Enable spark.eventLog.rolling.enabled by default Nov 2, 2023

dongjoon-hyun force-pushed the SPARK-45771 branch from 3798eb7 to 2aa7f1e Compare November 2, 2023 16:09

github-actions bot added the DOCS label Nov 2, 2023

dongjoon-hyun marked this pull request as draft November 2, 2023 16:29

dongjoon-hyun force-pushed the SPARK-45771 branch from 2aa7f1e to be50e5d Compare November 2, 2023 16:40

dongjoon-hyun commented Nov 2, 2023

View reviewed changes

[SPARK-45771][CORE] Enable spark.eventLog.rolling.enabled by default

b918798

dongjoon-hyun force-pushed the SPARK-45771 branch from be50e5d to b918798 Compare November 2, 2023 16:53

dongjoon-hyun marked this pull request as ready for review November 2, 2023 17:14

viirya reviewed Nov 2, 2023

View reviewed changes

viirya approved these changes Nov 2, 2023

View reviewed changes

dongjoon-hyun closed this in 653b31e Nov 2, 2023

dongjoon-hyun deleted the SPARK-45771 branch November 2, 2023 21:07

dongjoon-hyun mentioned this pull request Jun 14, 2025

[SPARK-52481] Add Spark History Server example apache/spark-kubernetes-operator#249

Closed

dongjoon-hyun mentioned this pull request Jul 22, 2025

[SPARK-52914][CORE] Support On-Demand Log Loading for rolling logs in History Server #51604

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-45771][CORE] Enable `spark.eventLog.rolling.enabled` by default #43638

[SPARK-45771][CORE] Enable `spark.eventLog.rolling.enabled` by default #43638

Uh oh!

dongjoon-hyun commented Nov 2, 2023 •

edited

Loading

Uh oh!

dongjoon-hyun Nov 2, 2023

Uh oh!

dongjoon-hyun commented Nov 2, 2023

Uh oh!

viirya Nov 2, 2023 •

edited

Loading

Uh oh!

viirya Nov 2, 2023

Uh oh!

viirya Nov 2, 2023

Uh oh!

viirya Nov 2, 2023

Uh oh!

dongjoon-hyun Nov 2, 2023

Uh oh!

dongjoon-hyun commented Nov 2, 2023 •

edited

Loading

Uh oh!

viirya commented Nov 2, 2023

Uh oh!

dongjoon-hyun commented Nov 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	conf.set(EVENT_LOG_ENABLE_ROLLING, true)
	conf.set(EVENT_LOG_ENABLE_ROLLING, false)
	buildWriterAndVerify(conf, classOf[SingleEventLogFileWriter])

[SPARK-45771][CORE] Enable spark.eventLog.rolling.enabled by default #43638

[SPARK-45771][CORE] Enable spark.eventLog.rolling.enabled by default #43638

Uh oh!

Conversation

dongjoon-hyun commented Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 2, 2023

Uh oh!

viirya Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

viirya Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

viirya Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 2, 2023

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viirya commented Nov 2, 2023

Uh oh!

dongjoon-hyun commented Nov 2, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[SPARK-45771][CORE] Enable `spark.eventLog.rolling.enabled` by default #43638

[SPARK-45771][CORE] Enable `spark.eventLog.rolling.enabled` by default #43638

dongjoon-hyun commented Nov 2, 2023 •

edited

Loading

viirya Nov 2, 2023 •

edited

Loading

dongjoon-hyun commented Nov 2, 2023 •

edited

Loading