Skip to content

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wayneguow wants to merge 11 commits into from
Closed

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bf2b681
rebase master
wayneguow Jan 31, 2023
a77cd1c
remove blank
wayneguow Jan 31, 2023
5a8246c
update docs
wayneguow Jan 31, 2023
b89264f
change target to Spark 3.5
wayneguow Feb 5, 2023
74bb68a
add to deprecatedConfigs
wayneguow Feb 14, 2023
3236f5f
update docs
wayneguow May 6, 2023
b90ed1f
remove
wayneguow May 12, 2023
46d88f4
fix docs
wayneguow May 30, 2024
753854b
fix docs
wayneguow May 30, 2024
02843cf
update version
wayneguow May 30, 2024
28f843d
separate typo
wayneguow Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
separate typo
  • Loading branch information
@wayneguow
wayneguow committed Jun 4, 2024
commit 28f843d84bc55825bd47841be93c29d9ff7b2a4c
2 changes: 1 addition & 1 deletion docs/core-migration-guide.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ license: |

- Since Spark 4.0, the MDC (Mapped Diagnostic Context) key for Spark task names in Spark logs has been changed from `mdc.taskName` to `task_name`. To use the key `mdc.taskName`, you can set `spark.log.legacyTaskNameMdc.enabled` to `true`.

- Since Spark 4.0, Spark performs speculative executions less aggressively with `spark.speculation.multiplier=3` and `spark.speculation.quantile=0.9`. To restore the legacy behavior, you can set `spark.speculation.multiplier=1.5` and `spark.speculation.quantile=0.75`.
- Since Spark 4.0, Spark performs speculative executions less agressively with `spark.speculation.multiplier=3` and `spark.speculation.quantile=0.9`. To restore the legacy behavior, you can set `spark.speculation.multiplier=1.5` and `spark.speculation.quantile=0.75`.

- Since Spark 4.0, `spark.shuffle.unsafe.file.output.buffer` is deprecated though still works. Use `spark.shuffle.localDisk.file.output.buffer` instead.

Expand Down