Skip to content

[SPARK-10342] [SPARK-10309] [SPARK-10474] [SPARK-10929] [SQL] Cooperative memory management #9241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 17 commits into from
Closed

[SPARK-10342] [SPARK-10309] [SPARK-10474] [SPARK-10929] [SQL] Cooperative memory management #9241

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
d0ada7b
force spilling
Oct 26, 2015
ee6b9a4
fix build
Oct 27, 2015
49b8135
fix Spillable
Oct 27, 2015
7087f2f
fix tests
Oct 27, 2015
ce24f03
add more tests
Oct 28, 2015
8470fc9
address coments
Oct 28, 2015
827d4f0
add tests for TaskMemoryManager
Oct 28, 2015
a3e01d0
counting the longArray in BytesToBytesMap
Oct 29, 2015
7bf76e5
preserve the memory in BytsToBytesMap
Oct 29, 2015
51278f8
use the memory of longArray for in-memory sorter
Oct 29, 2015
27ff4fc
improve spilling
Oct 29, 2015
4491013
release from other consumers first
Oct 29, 2015
c044afe
fix npe
Oct 29, 2015
afc8c7c
address comments
Oct 29, 2015
cda4b2a
address more comments
Oct 29, 2015
4ee1f42
improve OOM handling
Oct 30, 2015
e943e74
fix tests
Oct 30, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix npe
  • Loading branch information
Davies Liu committed Oct 29, 2015
commit c044afe62a36d98934d4e9f36665e2a4837783c7
2 changes: 1 addition & 1 deletion core/src/main/java/org/apache/spark/memory/TaskMemoryManager.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ public long acquireExecutionMemory(long required, MemoryConsumer consumer) {
}

// call spill() on itself
if (got < required) {
if (got < required && consumer != null) {
try {
long released = consumer.spill(required - got, consumer);
if (released > 0) {
Expand Down