Skip to content

[SPARK-21923][CORE]Avoid calling reserveUnrollMemoryForThisTask for every record #19135

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ConeyLiu wants to merge 9 commits into from
Closed

[SPARK-21923][CORE]Avoid calling reserveUnrollMemoryForThisTask for every record #19135

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
d20fa43
avoid call reserveUnrollMemoryForThisTask every record
ConeyLiu Sep 3, 2017
77b1580
Merge branch 'master' into memorystore
ConeyLiu Sep 5, 2017
9b0eed9
small fix
ConeyLiu Sep 6, 2017
8c25b2b
small fix
ConeyLiu Sep 6, 2017
4b1f9dd
change reserve more memory to check every batch records
ConeyLiu Sep 11, 2017
8059aff
small fix
ConeyLiu Sep 11, 2017
e7ebbd2
make parameter configurable and request memory with factor
ConeyLiu Sep 15, 2017
7f97cc2
address comments
ConeyLiu Sep 17, 2017
e1dc7a4
address comments
ConeyLiu Sep 18, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
small fix
  • Loading branch information
@ConeyLiu
ConeyLiu committed Sep 6, 2017
commit 8c25b2b3fc2c2f76f258201db1e03d56a3bc0e94
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -362,7 +362,7 @@ private[spark] class MemoryStore(

def reserveAdditionalMemoryIfNecessary(): Unit = {
if (bbos.size > unrollMemoryUsedByThisBlock) {
val amountToRequest = bbos.size - unrollMemoryUsedByThisBlock + additionalMemoryRequest
val amountToRequest = bbos.size - unrollMemoryUsedByThisBlock + additionalMemoryRequest
keepUnrolling = reserveUnrollMemoryForThisTask(blockId, amountToRequest, memoryMode)
if (keepUnrolling) {
unrollMemoryUsedByThisBlock += amountToRequest
Expand Down