Skip to content

[SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode #30062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
otterc wants to merge 35 commits into from
Closed

[SPARK-32916][SHUFFLE] Implementation of shuffle service that leverages push-based shuffle in YARN deployment mode #30062

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
df48e01
LIHADOOP-48527 Magnet shuffle service block transfer netty protocol
Victsm May 9, 2020
6472267
LIHADOOP-53438 Using different appId for the tests in RemoteBlockPush…
otterc May 12, 2020
1c78e1d
LIHADOOP-53496 Not logging all block push exceptions on the client
otterc May 15, 2020
71f3246
LIHADOOP-53700 Separate configuration for caching the merged index fi…
zhouyejoe Jun 1, 2020
221178f
LIHADOOP-53940 Logging the data file and index file path when shuffle…
otterc Jun 10, 2020
55b4a5f
LIHADOOP-54059 LIHADOOP-53496 Handle the inconsistencies between loc…
otterc Jun 15, 2020
f9d0e86
LIHADOOP-54379 Sorting the disks both on shuffle service and executors
otterc Jun 24, 2020
548e2c0
LIHADOOP-52494 Magnet fallback to origin shuffle blocks when fetch of…
otterc Jul 24, 2020
50efba9
LIHADOOP-55372 reduced the default for minChunkSizeInMergedShuffleFile
otterc Aug 26, 2020
8a6e01b
LIHADOOP-55315 Avoid network when fetching merged shuffle file in loc…
zhouyejoe Sep 9, 2020
ae5ffac
LIHADOOP-55654 Duplicate application init calls trigger NPE and wrong…
zhouyejoe Sep 12, 2020
e51042b
Further prune changes that should go into a later PR.
Victsm Sep 23, 2020
83aca99
LIHADOOP-54379 Sorting the disks both on shuffle service and executors
otterc Jun 24, 2020
04e0efe
LIHADOOP-55022 Disable the merged shuffle file cleanup in stopApplica…
zhouyejoe Aug 11, 2020
71dfd48
Tests and cleanup
otterc Oct 6, 2020
0c411c1
LIHADOOP-55948 Failure in the push stream should not change the curre…
otterc Oct 1, 2020
d029463
Minor style corrections
otterc Oct 15, 2020
8f3839f
Fixed style issues
otterc Oct 15, 2020
1cd2d03
Renamed variables, methods, fixed indentation, addressed other review…
otterc Oct 19, 2020
3356c19
Addressing review comments
otterc Oct 23, 2020
d879beb
Changed the partitions map and addressed other review comments
otterc Oct 26, 2020
48ae819
Added support for subdirs under merge_manager dirs and removed the ya…
otterc Oct 28, 2020
9b031f7
Addressed test failure and other review comments in RemoteBlockPushRe…
otterc Oct 29, 2020
807cc7b
Minor change in finalization
otterc Oct 29, 2020
5b169bc
Removing the partition from inner map after the files are closed
otterc Oct 30, 2020
9ece587
Server side configuration to specify the implementation of MergedShuf…
otterc Oct 30, 2020
d13c7ad
Change the Push block stream to not encode shuffle Id, map index, and…
otterc Nov 2, 2020
63843bb
Fixed typos, address review comments, made NoOp the default impl, and…
otterc Nov 2, 2020
d35aa4b
Addressed review comments
otterc Nov 3, 2020
ba92311
Fix IndexOutOfBoundsException and avoid instantiating AppsPathInfo mu…
otterc Nov 4, 2020
9be25b3
Removed duplicate declaration of shuffle push prefix
otterc Nov 4, 2020
ba51796
Added UT for collision with 2 streams
otterc Nov 4, 2020
1f4fcfe
Removed unnecessary TODOs, marked MergedShuffleFileManager evolving, …
otterc Nov 4, 2020
28edaae
Changed the serialization on chunktracker and removed serializedSizeI…
otterc Nov 6, 2020
cb1881c
Use RandomAccessFile
otterc Nov 8, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
LIHADOOP-48527 Magnet shuffle service block transfer netty protocol 
The following changes are included in this patch.
In addition, fixed a potential block duplicate issue when speculative execution is enabled, and improved test coverage.

commit 7e134c2b75c8882474a67c15036087eb8a02caef
Author: Chandni Singh <[email protected]>
Date:   Tue Apr 21 21:54:46 2020 -0700

    LIHADOOP-52202 Create merge_manager under app local dirs

    RB=2069854
    G=superfriends-reviewers
    R=mshen,yezhou
    A=chsingh

commit 63dcda9309fc06c5f1fb6e7268df1d7416db49c7
Author: Chandni Singh <[email protected]>
Date:   Wed Apr 1 12:15:21 2020 -0700

    LIHADOOP-51889 Divide remote fetches into smaller chunks

    RB=2029681

commit 35298465155ec10e6ee2caf1adc0e78717dc6fed
Author: Chandni Singh <[email protected]>
Date:   Thu Mar 19 17:47:40 2020 -0700

    LIHADOOP-51889 Writing last chunk offsets to merge index file

    RB=2016700
    BUG=LIHADOOP-51889
    G=superfriends-reviewers
    R=mshen
    A=mshen

commit bbb53ec0fdfa0aebda954ede17a9f6e217607a53
Author: Min Shen <[email protected]>
Date:   Thu Dec 19 08:46:34 2019 -0800

    Shuffle server and client properly handles merged block fetch failure.
    Use file length as merged shuffle block size when serving merged shuffle block.

commit 6718074c6a6a98b1d66d4fdff6bf08fb266ce32e
Author: Min Shen <[email protected]>
Date:   Mon Nov 18 14:19:20 2019 -0800

    Netty protocol for DAGScheduler control message

commit 52e4dfade2e004fbc39fc60937342a9a57872680
Author: Min Shen <[email protected]>
Date:   Sun Sep 8 18:44:09 2019 -0700

    Netty protocol for remote block push, pass 3

commit e9db4cc1ae56e9722a598b0011a10e55e84bf19c
Author: Min Shen <[email protected]>
Date:   Thu Sep 5 18:29:24 2019 -0700

    Netty protocol for remote block push, pass 2

commit 7627ecf54292edda4a133e596f53306e7af76100
Author: Min Shen <[email protected]>
Date:   Fri Aug 30 08:54:08 2019 -0700

    Netty protocol for remote block push, pass 1
  • Loading branch information
@Victsm @otterc
Victsm authored and otterc committed Nov 6, 2020
commit df48e011372c33a2cd2331267c9eab3dace91a2c
Loading