Skip to content

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wayneguow wants to merge 11 commits into from
Closed

[SPARK-42252][CORE] Add spark.shuffle.localDisk.file.output.buffer and deprecate spark.shuffle.unsafe.file.output.buffer #39819

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
bf2b681
rebase master
wayneguow Jan 31, 2023
a77cd1c
remove blank
wayneguow Jan 31, 2023
5a8246c
update docs
wayneguow Jan 31, 2023
b89264f
change target to Spark 3.5
wayneguow Feb 5, 2023
74bb68a
add to deprecatedConfigs
wayneguow Feb 14, 2023
3236f5f
update docs
wayneguow May 6, 2023
b90ed1f
remove
wayneguow May 12, 2023
46d88f4
fix docs
wayneguow May 30, 2024
753854b
fix docs
wayneguow May 30, 2024
02843cf
update version
wayneguow May 30, 2024
28f843d
separate typo
wayneguow Jun 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update docs
  • Loading branch information
@wayneguow
wayneguow committed May 30, 2024
commit 5a8246ce6c2f70d31f09a19be743c74ca0d918c1
12 changes: 10 additions & 2 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1031,11 +1031,19 @@ Apart from these, the following properties are also available, and may be useful
<td><code>spark.shuffle.unsafe.file.output.buffer</code></td>
<td>32k</td>
<td>
The file system for this buffer size after each partition is written in unsafe shuffle writer.
In KiB unless otherwise specified.
Deprecated since Spark 3.5, please use <code>spark.shuffle.localDisk.file.output.buffer</code>.
</td>
<td>2.3.0</td>
</tr>
<tr>
<td><code>spark.shuffle.localDisk.file.output.buffer</code></td>
<td>32k</td>
<td>
The file system for this buffer size after each partition is written in in all local disk shuffle writers.
In KiB unless otherwise specified.
</td>
<td>3.5.0</td>
</tr>
<tr>
<td><code>spark.shuffle.spill.diskWriteBufferSize</code></td>
<td>1024 * 1024</td>
Expand Down