Skip to content

[SPARK-8029] robust shuffle writer (for 1.5 branch) #9686

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davies wants to merge 3 commits into from
Closed

[SPARK-8029] robust shuffle writer (for 1.5 branch) #9686

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7e83298
robust shuffle writer
Nov 13, 2015
a6d569e
fix consolidate
Nov 13, 2015
e95fcf5
fix test
Nov 13, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix test
  • Loading branch information
Davies Liu committed Nov 13, 2015
commit e95fcf5c8ef849e737fb7fc4ef6f5b41331348f0
1 change: 1 addition & 0 deletions core/src/test/scala/org/apache/spark/shuffle/hash/HashShuffleManagerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ class HashShuffleManagerSuite extends SparkFunSuite with LocalSparkContext {
// an object is written. So if the codepaths assume writeObject is end of data, this should
// flush those bugs out. This was common bug in ExternalAppendOnlyMap, etc.
conf.set("spark.serializer.objectStreamReset", "1")
conf.set("spark.shuffle.consolidateFiles", "true")
Copy link
Contributor Author

@davies davies Nov 13, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aarondav Do you remember why we did not enable consolidation in this test? Since the title is "consolidated shuffle can write to shuffle group without messing existing offsets/lengths" (introduced in #1678)

conf.set("spark.serializer", "org.apache.spark.serializer.JavaSerializer")
conf.set("spark.shuffle.manager", "org.apache.spark.shuffle.hash.HashShuffleManager")

Expand Down