-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-22897][CORE]: Expose stageAttemptId in TaskContext #20082
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
5753ee0
f02bc1e
59e4a9c
291bbbc
72a3abf
9266cd8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -159,28 +159,28 @@ class TaskContextSuite extends SparkFunSuite with BeforeAndAfter with LocalSpark | |
| assert(attemptIdsWithFailedTask.toSet === Set(0, 1)) | ||
| } | ||
|
|
||
| test("TaskContext.stageAttemptId getter") { | ||
| test("TaskContext.stageAttemptNumber getter") { | ||
| sc = new SparkContext("local[1,2]", "test") | ||
|
|
||
| // Check stage attemptIds are 0 for initial stage | ||
| val stageAttemptIds = sc.parallelize(Seq(1, 2), 2).mapPartitions { _ => | ||
| Seq(TaskContext.get().stageAttemptId()).iterator | ||
| // Check stageAttemptNumbers are 0 for initial stage | ||
| val stageAttemptNumbers = sc.parallelize(Seq(1, 2), 2).mapPartitions { _ => | ||
| Seq(TaskContext.get().stageAttemptNumber()).iterator | ||
| }.collect() | ||
| assert(stageAttemptIds.toSet === Set(0)) | ||
| assert(stageAttemptNumbers.toSet === Set(0)) | ||
|
|
||
| // Check stage attemptIds that are resubmitted when tasks have FetchFailedException | ||
| val stageAttemptIdsWithFailedStage = | ||
| // Check stageAttemptNumbers that are resubmitted when tasks have FetchFailedException | ||
| val stageAttemptNumbersWithFailedStage = | ||
| sc.parallelize(Seq(1, 2, 3, 4), 4).repartition(1).mapPartitions { _ => | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You don't need |
||
| val stageAttemptId = TaskContext.get().stageAttemptId() | ||
| if (stageAttemptId < 2) { | ||
| val stageAttemptNumber = TaskContext.get().stageAttemptNumber() | ||
| if (stageAttemptNumber < 2) { | ||
| // Throw FetchFailedException to explicitly trigger stage resubmission. A normal exception | ||
| // will only trigger task resubmission in the same stage. | ||
| throw new FetchFailedException(null, 0, 0, 0, "Fake") | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Emmm... just throw an
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Related to repartition part. I use FetchFailedException to explicitly trigger a stage resubmission. Otherwise, the task would be resubmitted in the same stage if IIRC.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. oh, right~
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please add comment to explain that
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will do. |
||
| } | ||
| Seq(stageAttemptId).iterator | ||
| Seq(stageAttemptNumber).iterator | ||
| }.collect() | ||
|
|
||
| assert(stageAttemptIdsWithFailedStage.toSet === Set(2)) | ||
| assert(stageAttemptNumbersWithFailedStage.toSet === Set(2)) | ||
| } | ||
|
|
||
| test("accumulators are updated on exception failures") { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How much work we need to rename the internal
stageAttemptIdtostageAttemptNumber?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The modification may not be too much (100+ occurrences in 20+ files), however it may break eventLog's JsonProtocol backward compatibility(not sure)..
@squito you may have more knowledge on this since you introduced
stageAttemptId.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ah, so
stageAttemptIdis already exposed in developer API, we can't change it.