[SPARK-17451] [CORE] CoarseGrainedExecutorBackend should inform driver before self-kill #15013
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…r before self-kill
Contributor
Author
|
cc @zsxwing for review |
|
Test build #65104 has finished for PR 15013 at commit
|
|
Test build #65111 has finished for PR 15013 at commit
|
|
|
||
| if (notifyDriver) { | ||
| logInfo(s"Notifying the driver before exiting the executor") | ||
| rpcEnv.asyncSetupEndpointRefByURI(driverUrl).flatMap { ref => |
Member
There was a problem hiding this comment.
not need to connect to driver if driver is None. It's usually because we cannot connect to driver. Then the codes become very simple:
if (notifyDriver && driver.nonEmpty) {
driver.get.ask[Boolean](RemoveExecutor(executorId, new ExecutorLossReason(reason))).onFailure { case e =>
logWarning(s"Unable to notify the driver due to " + e.getMessage, e)
}(ThreadUtils.sameThread)
}
|
Test build #65113 has finished for PR 15013 at commit
|
|
Test build #65124 has finished for PR 15013 at commit
|
Contributor
Author
|
Done with all change. Ready for review. |
Contributor
Author
|
@zsxwing : ping |
Member
|
retest this please. LGTM. Let's run the test again! |
|
Test build #3264 has finished for PR 15013 at commit
|
|
Test build #3265 has finished for PR 15013 at commit
|
Member
|
LGTM. Thanks! Merging to master. |
wgtmac
pushed a commit
to wgtmac/spark
that referenced
this pull request
Sep 19, 2016
… before self-kill ## What changes were proposed in this pull request? Jira : https://issues.apache.org/jira/browse/SPARK-17451 `CoarseGrainedExecutorBackend` in some failure cases exits the JVM. While this does not have any issue, from the driver UI there is no specific reason captured for this. In this PR, I am adding functionality to `exitExecutor` to notify driver that the executor is exiting. ## How was this patch tested? Ran the change over a test env and took down shuffle service before the executor could register to it. In the driver logs, where the job failure reason is mentioned (ie. `Job aborted due to stage ...` it gives the correct reason: Before: `ExecutorLostFailure (executor ZZZZZZZZZ exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.` After: `ExecutorLostFailure (executor ZZZZZZZZZ exited caused by one of the running tasks) Reason: Unable to create executor due to java.util.concurrent.TimeoutException: Timeout waiting for task.` Author: Tejas Patil <[email protected]> Closes apache#15013 from tejasapatil/SPARK-17451_inform_driver.
zzcclp
added a commit
to zzcclp/spark
that referenced
this pull request
Sep 20, 2016
…driver before self-kill apache#15013
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Jira : https://issues.apache.org/jira/browse/SPARK-17451
CoarseGrainedExecutorBackendin some failure cases exits the JVM. While this does not have any issue, from the driver UI there is no specific reason captured for this. In this PR, I am adding functionality toexitExecutorto notify driver that the executor is exiting.How was this patch tested?
Ran the change over a test env and took down shuffle service before the executor could register to it. In the driver logs, where the job failure reason is mentioned (ie.
Job aborted due to stage ...it gives the correct reason:Before:
ExecutorLostFailure (executor ZZZZZZZZZ exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.After:
ExecutorLostFailure (executor ZZZZZZZZZ exited caused by one of the running tasks) Reason: Unable to create executor due to java.util.concurrent.TimeoutException: Timeout waiting for task.