Skip to content

[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cxzl25 wants to merge 12 commits into from
Closed

[SPARK-49386][CORE][SQL] Add memory based thresholds for shuffle spill #47856

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
647afb8
[SPARK-27734][CORE][SQL] Add memory based thresholds for shuffle spill
amuraru Sep 15, 2019
044a232
spill by size
cxzl25 Aug 23, 2024
27c6478
config
cxzl25 Aug 23, 2024
a3f5d16
codegen
cxzl25 Aug 23, 2024
14a7f2d
add test
cxzl25 Aug 27, 2024
8374d18
use _elementsRead
cxzl25 Apr 21, 2025
57f0711
checkValue fallbackConf
cxzl25 Apr 21, 2025
d734bfa
log
cxzl25 May 3, 2025
7f78264
shouldSpill val
cxzl25 May 6, 2025
b28bd7b
config version 4.1.0
cxzl25 May 26, 2025
d083773
extra line
cxzl25 Jun 26, 2025
8225e94
test
cxzl25 Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
codegen
  • Loading branch information
@cxzl25
cxzl25 committed Jun 26, 2025
commit a3f5d1668dbb4c9c69f745603569aee4a4bb7fec
2 changes: 1 addition & 1 deletion sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeJoinExec.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,7 +233,7 @@ case class SortMergeJoinExec(

// Inline mutable state since not many join operations in a task
val matches = ctx.addMutableState(clsName, "matches",
v => s"$v = new $clsName($inMemoryThreshold, $spillThreshold, ${spillSizeThreshold}L));",
v => s"$v = new $clsName($inMemoryThreshold, $spillThreshold, ${spillSizeThreshold}L);",
forceInline = true)
// Copy the streamed keys as class members so they could be used in next function call.
val matchedKeyVars = copyKeys(ctx, streamedKeyVars)
Expand Down