Skip to content

[SPARK-19753][CORE] Un-register all shuffle output on a host in case of slave lost or fetch failure #18150

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sitalkedia wants to merge 16 commits into from
Closed

[SPARK-19753][CORE] Un-register all shuffle output on a host in case of slave lost or fetch failure #18150

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
4643325
[SPARK-19753][CORE] All shuffle files on a host should be removed in …
Feb 25, 2017
6145d2a
Do not un-register shuffle files in case of executor lost
Mar 1, 2017
ef86d12
no-op when external shuffle service is disabled
Mar 1, 2017
263d55b
fix check style
Mar 1, 2017
a548dc3
Addressed review comments and fixed a bug
Mar 16, 2017
b5f1949
Fix build
Mar 16, 2017
705a5aa
Fix test failures
Mar 17, 2017
996ad20
refactoring & comments
squito Mar 17, 2017
78f9094
Review comments
Mar 18, 2017
3b53f7e
Minor changes as per review comments
Mar 24, 2017
e5f532e
Make the behaviour configurable and minor review comments
Jun 9, 2017
1189e40
Minor changes
Jun 9, 2017
9d700e2
set the behavior to false by defaut
Jun 9, 2017
b26e99d
Fix test
Jun 10, 2017
74f285a
Rebase with master
Jun 14, 2017
ba2ca2a
remove extra while space
Jun 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Minor changes
  • Loading branch information
Sital Kedia committed Jun 13, 2017
commit 1189e409ddec9faf38de2c1b4d0c00e68c162e46
6 changes: 3 additions & 3 deletions core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,13 +188,13 @@ class DAGScheduler(
private val disallowStageRetryForTest = sc.getConf.getBoolean("spark.test.noStageRetry", false)

/**
* Number of consecutive stage attempts allowed before a stage is aborted.
* If enabled, fetch failure will cause all the output on that host to be unregistered.
*/
private[scheduler] val unRegisterOutputOnHostOnFetchFailure =
Copy link
Contributor

@jiangxb1987 jiangxb1987 Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please place correct comment over the value.

sc.getConf.getBoolean("spark.fetch.failure.unRegister.output.on.host", true)
sc.getConf.getBoolean("spark.files.fetchFailure.unRegisterOutputOnHost", true)

/**
Copy link
Contributor

@jiangxb1987 jiangxb1987 Jun 9, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

*
* Number of consecutive stage attempts allowed before a stage is aborted.
*/
private[scheduler] val maxConsecutiveStageAttempts =
sc.getConf.getInt("spark.stage.maxConsecutiveAttempts",
Expand Down