Skip to content

[SPARK-22097][CORE]Request an accurate memory after we unrolled the block #19316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ConeyLiu wants to merge 2 commits into from
Closed

[SPARK-22097][CORE]Request an accurate memory after we unrolled the block #19316

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bfe162e
close the serializationStream after check the memory request
ConeyLiu Sep 22, 2017
3af730d
small fix
ConeyLiu Sep 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -388,7 +388,13 @@ private[spark] class MemoryStore(
// perform one final call to attempt to allocate additional memory if necessary.
if (keepUnrolling) {
Copy link
Contributor

@cloud-fan cloud-fan Oct 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we merge this if into the next if?

Copy link
Contributor Author

@ConeyLiu ConeyLiu Oct 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We maybe need request the extra-memory (bbos.size - unrollMemoryUsedByThisBlock). So the keepUnrolling maybe change in line 393.

serializationStream.close()
reserveAdditionalMemoryIfNecessary()
if (bbos.size > unrollMemoryUsedByThisBlock) {
val amountToRequest = bbos.size - unrollMemoryUsedByThisBlock
Copy link
Contributor

@jiangxb1987 jiangxb1987 Oct 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not just fix that in reserveAdditionalMemoryIfNecessary ?

Copy link
Contributor Author

@ConeyLiu ConeyLiu Oct 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because we request val amountToRequest = (bbos.size * memoryGrowthFactor - unrollMemoryUsedByThisBlock).toLong in reserveAdditionalMemoryIfNecessary to avoid requesting for every records. But here we just need request the precise memory for the last request.

keepUnrolling = reserveUnrollMemoryForThisTask(blockId, amountToRequest, memoryMode)
if (keepUnrolling) {
unrollMemoryUsedByThisBlock += amountToRequest
}
}
}

if (keepUnrolling) {
Expand Down