Skip to content

[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cxzl25 wants to merge 12 commits into from
Closed

[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
647afb8
[SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill
amuraru Sep 15, 2019
044a232
spill by size
cxzl25 Aug 23, 2024
27c6478
config
cxzl25 Aug 23, 2024
a3f5d16
codegen
cxzl25 Aug 23, 2024
14a7f2d
add test
cxzl25 Aug 27, 2024
8374d18
use _elementsRead
cxzl25 Apr 21, 2025
57f0711
checkValue fallbackConf
cxzl25 Apr 21, 2025
d734bfa
log
cxzl25 May 3, 2025
7f78264
shouldSpill val
cxzl25 May 6, 2025
b28bd7b
config version 4.1.0
cxzl25 May 26, 2025
d083773
extra line
cxzl25 Jun 26, 2025
8225e94
test
cxzl25 Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use _elementsRead
  • Loading branch information
@cxzl25
cxzl25 committed Jun 26, 2025
commit 8374d18f7aab6017626fd3563b4c4e930a0c9d84
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/util/collection/Spillable.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ private[spark] abstract class Spillable[C](taskMemoryManager: TaskMemoryManager)
if (_elementsRead > numElementsForceSpillThreshold
|| currentMemory > maxSizeForceSpillThreshold) {
Copy link
Contributor

@mridulm mridulm Apr 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By moving _elementsRead > numElementsForceSpillThreshold here, we would actually reduce some unnecessary allocations .... nice !

shouldSpill = true
} else if (elementsRead % 32 == 0 && currentMemory >= myMemoryThreshold) {
} else if (_elementsRead % 32 == 0 && currentMemory >= myMemoryThreshold) {
// Claim up to double our current memory from the shuffle memory pool
val amountToRequest = 2 * currentMemory - myMemoryThreshold
val granted = acquireMemory(amountToRequest)
Expand All @@ -101,7 +101,7 @@ private[spark] abstract class Spillable[C](taskMemoryManager: TaskMemoryManager)
// Actually spill
if (shouldSpill) {
_spillCount += 1
logSpillage(currentMemory, elementsRead)
logSpillage(currentMemory, _elementsRead)
spill(collection)
_elementsRead = 0
_memoryBytesSpilled += currentMemory
Expand Down