Skip to content

[SPARK-9419] ShuffleMemoryManager and MemoryStore should track memory on a per-task, not per-thread, basis #7734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 11 commits into from
Closed

[SPARK-9419] ShuffleMemoryManager and MemoryStore should track memory on a per-task, not per-thread, basis #7734

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
c9e8e54
Use TaskAttemptIds to track unroll memory
JoshRosen Jul 28, 2015
2e1e0f8
Use TaskAttemptIds to track shuffle memory
JoshRosen Jul 28, 2015
1b0083b
Roll back fix in PySpark, which is no longer necessary
JoshRosen Jul 28, 2015
5e2f01e
Fix capitalization
JoshRosen Jul 28, 2015
fa78ee8
Move Executor's cleanup into Task so that TaskContext is defined when…
JoshRosen Jul 28, 2015
f57f3f2
More thread -> task changes
JoshRosen Jul 28, 2015
7b0f04b
Fix ShuffleMemoryManagerSuite
JoshRosen Jul 28, 2015
44f6497
Fix long line.
JoshRosen Jul 28, 2015
ed25d3b
Address minor PR review comments
JoshRosen Jul 29, 2015
57c9b4e
Merge remote-tracking branch 'origin/master' into memory-tracking-fixes
JoshRosen Jul 29, 2015
b4b1702
Propagate TaskContext to writer threads.
JoshRosen Jul 29, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Roll back fix in PySpark, which is no longer necessary
  • Loading branch information
@JoshRosen
JoshRosen committed Jul 28, 2015
commit 1b0083b078772ed9eedf12f35bd7f6f7187aa50d
5 changes: 0 additions & 5 deletions core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,11 +263,6 @@ private[spark] class PythonRDD(
if (!worker.isClosed) {
Utils.tryLog(worker.shutdownOutput())
}
} finally {
// Release memory used by this thread for shuffles
env.shuffleMemoryManager.releaseMemoryForThisTask()
// Release memory used by this thread for unrolling blocks
env.blockManager.memoryStore.releaseUnrollMemoryForThisTask()
}
Copy link
Contributor Author

@JoshRosen JoshRosen Jul 29, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @davies for this PySpark change.

}
}
Expand Down