-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-32917][SHUFFLE][CORE] Adds support for executors to push shuffle blocks after successful map task completion #30312
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
b968e0e
LIHADOOP-48527 Enable Shuffle writer to push blocks to remote shuffle…
510f504
LIHADOOP-53496 Not logging all block push exceptions on the client
otterc 2d9d27d
LIHADOOP-54374 Separate the configurations for connection creation ti…
zhouyejoe 3d74277
LIHADOOP-54370 Not to retry on certain exceptions when pushing blocks
otterc 02cff3b
Made the code compile, added more tests, introduced ShuffleBlockPushId
otterc 770c25c
Renamed isPushBasedShuffleEnabled to isPushShuffleEnabled and fixed t…
otterc d429fb3
Added Since tags all the newly introduced classes
otterc 3d10b20
Explicitly converting ArrayBuffer to Seq so that build with SBT succeeds
otterc c19bcf2
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleWriter.scala
otterc f411944
Addressed the typos and other comments
otterc bd3649f
Moved push of data out of ShuffleWriter and addressed other minor com…
otterc bf4b277
LIHADOOP-48527 Enable Shuffle writer to push blocks to remote shuffle…
a8a350c
LIHADOOP-54370 Not to retry on certain exceptions when pushing blocks
otterc 0b951a7
Made the code compile, added more tests, introduced ShuffleBlockPushId
otterc 7d16198
Added Since tags all the newly introduced classes
otterc 415c2d0
Moved push of data out of ShuffleWriter and addressed other minor com…
otterc 28d8098
Renamed PushShuffleSupport to PushShufleComponent and changed initiat…
otterc 5b725a2
Renamed PushShuffleComponent to PushShuffleWriterComponent
otterc a2f6635
Not changing the default value of connectionCreationTimeout
otterc 4a2aef7
Renamed PushShuffleWriterComponent to ShuffleBlockPusher and addresse…
otterc 1e9fb08
Addressed review comments
otterc 462af7a
Fixed the documentation for numPushThreads
otterc a88ffd6
Changed the initiateBlockPush method, added tests, and stop pushing c…
otterc 08386f3
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc d5370db
Addressed review comments
otterc fd0e98b
Addressed review comments
otterc 8c9a482
Changed the default of maxBlockSizeToPush to 1M
otterc f8632b3
Removed the stopPushing flag which is not needed
otterc 6d0fade
Addressed review comments
otterc 7d70f82
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc 23cc502
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc bcebb13
Addressed Mridul's comments
otterc 762ac1e
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc 6aae02a
Update core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPushe…
otterc 21ea881
Coverting blockSize to Int during preparation of push request
otterc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Addressed review comments
- Loading branch information
commit 6d0fadea97b4bcbf5769fcf0aa674c7981fd66cb
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -334,6 +334,8 @@ class ShuffleBlockPusherSuite extends SparkFunSuite with BeforeAndAfterEach { | |
| } | ||
|
|
||
| def runPendingTasks(): Unit = { | ||
|
||
| // This ensures that all the submitted tasks - updateStateAndCheckIfPushMore and pushUpToMax | ||
| // are run synchronously. | ||
| while (!tasks.isEmpty) { | ||
| tasks.take().run() | ||
| } | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be
2maccording to the comment?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The comment actually means that it should be higher than
2m. If it is 2m than each block will be loaded in memory which increases memory overhead. I will make the comment more clear.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done