Skip to content

[SPARK-20801] Record accurate size of blocks in MapStatus when it's above threshold. #18031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jinxing64 wants to merge 3 commits into from
+90 −10
Closed

[SPARK-20801] Record accurate size of blocks in MapStatus when it's above threshold. #18031

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bfea9f5
Record accurate size of blocks in MapStatus when it's above threshold.
May 18, 2017
ca65544
remove spark.shuffle.accurateBlockThresholdByTimesAverage
May 22, 2017
66aa56f
Refine docs
May 22, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Refine docs
  • Loading branch information
jinxing committed May 22, 2017
commit 66aa56fd4d23645ec1d2f0e253ea4888b4882f9d
8 changes: 4 additions & 4 deletions core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,8 +127,8 @@ private[spark] class CompressedMapStatus(

/**
* A [[MapStatus]] implementation that stores the accurate size of huge blocks, which are larger
* than both spark.shuffle.accurateBlockThreshold. It stores the average size of other non-empty
* blocks, plus a bitmap for tracking which blocks are empty.
* than spark.shuffle.accurateBlockThreshold. It stores the average size of other non-empty blocks,
* plus a bitmap for tracking which blocks are empty.
*
* @param loc location where the task is being executed
* @param numNonEmptyBlocks the number of non-empty blocks
Expand Down Expand Up @@ -211,8 +211,8 @@ private[spark] object HighlyCompressedMapStatus {
val size = uncompressedSizes(i)
if (size > 0) {
numNonEmptyBlocks += 1
// Remove the huge blocks from the calculation for average size and have accurate size for
// smaller blocks.
// Huge blocks are not included in the calculation for average size, thus size for smaller
// blocks is more accurate.
if (size < threshold) {
totalSize += size
Copy link
Contributor

@cloud-fan cloud-fan May 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be put in the else branch

} else {
Expand Down