[SPARK-13465] Add a task failure listener to TaskContext #11340
Conversation
|
test this pleaes |
|
Test build #2577 has finished for PR 11340 at commit
|
|
Test build #51862 has finished for PR 11340 at commit
|
This reverts commit 9efd83c.
|
Test build #51865 has finished for PR 11340 at commit
|
|
Test build #2579 has finished for PR 11340 at commit
|
|
Test build #2582 has finished for PR 11340 at commit
|
| private[spark] def markTaskFailed(error: Throwable): Unit = { | ||
| val errorMsgs = new ArrayBuffer[String](2) | ||
| // Process complete callbacks in the reverse order of registration | ||
| onFailureCallbacks.reverse.foreach { listener => |
There was a problem hiding this comment.
Existing: Should we store these callback as reversed order, so don't need to reverse them for every callback, especially for onCompleteCallbacks.
There was a problem hiding this comment.
Not sure how much it matters at all since it is an one time only operation and usually the number of callbacks is in the range of 0 to 2. Prepending efficiently would require us to not use the arraybuffer too.
| * executing the callback in TaskCompletionListener. | ||
| */ | ||
| private[spark] | ||
| class TaskCompletionListenerException(errorMessages: Seq[String]) extends Exception { |
There was a problem hiding this comment.
this was moved into JavaTaskCompletionListenerImpl as a static class.
|
@davies I updated the patch. |
|
Test build #52027 has finished for PR 11340 at commit
|
|
LGTM, merging into master, thanks! |
## What changes were proposed in this pull request?
TaskContext supports task completion callback, which gets called regardless of task failures. However, there is no way for the listener to know if ther
e is an error. This patch adds a new listener that gets called when a task fails.
## How was the this patch tested?
New unit test case and integration test case covering the code path
Author: Reynold Xin <[email protected]>
Closes apache#11340 from rxin/SPARK-13465.
4 participants
What changes were proposed in this pull request?
TaskContext supports task completion callback, which gets called regardless of task failures. However, there is no way for the listener to know if there is an error. This patch adds a new listener that gets called when a task fails.
How was the this patch tested?
New unit test case and integration test case covering the code path