Skip to content

[SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion #30312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
otterc wants to merge 35 commits into from
Closed

[SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion #30312

Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
b968e0e
LIHADOOP-48527 Enable Shuffle writer to push blocks to remote shuffle…
May 8, 2020
510f504
LIHADOOP-53496 Not logging all block push exceptions on the client
otterc May 15, 2020
2d9d27d
LIHADOOP-54374 Separate the configurations for connection creation ti…
zhouyejoe Jun 25, 2020
3d74277
LIHADOOP-54370 Not to retry on certain exceptions when pushing blocks
otterc Jun 24, 2020
02cff3b
Made the code compile, added more tests, introduced ShuffleBlockPushId
otterc Nov 6, 2020
770c25c
Renamed isPushBasedShuffleEnabled to isPushShuffleEnabled and fixed t…
otterc Nov 10, 2020
d429fb3
Added Since tags all the newly introduced classes
otterc Nov 10, 2020
3d10b20
Explicitly converting ArrayBuffer to Seq so that build with SBT succeeds
otterc Nov 11, 2020
c19bcf2
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleWriter.scala
otterc Nov 12, 2020
f411944
Addressed the typos and other comments
otterc Nov 12, 2020
bd3649f
Moved push of data out of ShuffleWriter and addressed other minor com…
otterc Nov 18, 2020
bf4b277
LIHADOOP-48527 Enable Shuffle writer to push blocks to remote shuffle…
May 8, 2020
a8a350c
LIHADOOP-54370 Not to retry on certain exceptions when pushing blocks
otterc Jun 24, 2020
0b951a7
Made the code compile, added more tests, introduced ShuffleBlockPushId
otterc Nov 6, 2020
7d16198
Added Since tags all the newly introduced classes
otterc Nov 10, 2020
415c2d0
Moved push of data out of ShuffleWriter and addressed other minor com…
otterc Nov 18, 2020
28d8098
Renamed PushShuffleSupport to PushShufleComponent and changed initiat…
otterc Nov 18, 2020
5b725a2
Renamed PushShuffleComponent to PushShuffleWriterComponent
otterc Nov 19, 2020
a2f6635
Not changing the default value of connectionCreationTimeout
otterc Nov 21, 2020
4a2aef7
Renamed PushShuffleWriterComponent to ShuffleBlockPusher and addresse…
otterc Nov 23, 2020
1e9fb08
Addressed review comments
otterc Nov 25, 2020
462af7a
Fixed the documentation for numPushThreads
otterc Nov 25, 2020
a88ffd6
Changed the initiateBlockPush method, added tests, and stop pushing c…
otterc Dec 1, 2020
08386f3
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc Dec 2, 2020
d5370db
Addressed review comments
otterc Dec 2, 2020
fd0e98b
Addressed review comments
otterc Dec 3, 2020
8c9a482
Changed the default of maxBlockSizeToPush to 1M
otterc Dec 4, 2020
f8632b3
Removed the stopPushing flag which is not needed
otterc Dec 4, 2020
6d0fade
Addressed review comments
otterc Dec 4, 2020
7d70f82
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc Dec 17, 2020
23cc502
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc Dec 17, 2020
bcebb13
Addressed Mridul's comments
otterc Dec 17, 2020
762ac1e
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc Dec 19, 2020
6aae02a
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc Dec 19, 2020
21ea881
Coverting blockSize to Int during preparation of push request
otterc Dec 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe… 
…r.scala

Co-authored-by: wuyi <[email protected]>
  • Loading branch information
@otterc @Ngone51
otterc and Ngone51 committed Dec 4, 2020
commit 08386f3549789a70c01cf0743117c4b0694963f3
4 changes: 1 addition & 3 deletions core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,8 +101,7 @@ private[spark] class ShuffleBlockPusher(
dataFile: File,
partitionLengths: Array[Long],
dep: ShuffleDependency[_, _, _],
partitionId: Int
): Unit = {
partitionId: Int): Unit = {
val numPartitions = dep.partitioner.numPartitions
val mergerLocs = dep.getMergerLocs.map(loc => BlockManagerId("", loc.host, loc.port))
val transportConf = SparkTransportConf.fromSparkConf(conf, "shuffle")
Expand Down Expand Up @@ -466,4 +465,3 @@ private[spark] object ShuffleBlockPusher {
}
}
}