Skip to content

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wayneguow wants to merge 11 commits into from
Closed

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bf2b681
rebase master
wayneguow Jan 31, 2023
a77cd1c
remove blank
wayneguow Jan 31, 2023
5a8246c
update docs
wayneguow Jan 31, 2023
b89264f
change target to Spark 3.5
wayneguow Feb 5, 2023
74bb68a
add to deprecatedConfigs
wayneguow Feb 14, 2023
3236f5f
update docs
wayneguow May 6, 2023
b90ed1f
remove
wayneguow May 12, 2023
46d88f4
fix docs
wayneguow May 30, 2024
753854b
fix docs
wayneguow May 30, 2024
02843cf
update version
wayneguow May 30, 2024
28f843d
separate typo
wayneguow Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add to deprecatedConfigs
  • Loading branch information
@wayneguow
wayneguow committed May 30, 2024
commit 74bb68a9cf418d2e68072012b689d2894362557d
4 changes: 3 additions & 1 deletion core/src/main/scala/org/apache/spark/SparkConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,7 +647,9 @@ private[spark] object SparkConf extends Logging {
DeprecatedConfig("spark.yarn.blacklist.executor.launch.blacklisting.enabled", "3.1.0",
"Please use spark.yarn.executor.launch.excludeOnFailure.enabled"),
DeprecatedConfig("spark.network.remoteReadNioBufferConversion", "3.5.2",
"Please open a JIRA ticket to report it if you need to use this configuration.")
"Please open a JIRA ticket to report it if you need to use this configuration."),
DeprecatedConfig("spark.shuffle.unsafe.file.output.buffer", "3.5.2",
"Please use spark.shuffle.localDisk.file.output.buffer")
)

Map(configs.map { cfg => (cfg.key -> cfg) } : _*)
Expand Down