Skip to content

[SPARK-10381] Fix mixup of taskAttemptNumber & attemptId in OutputCommitCoordinator #8544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 9 commits into from
Closed

[SPARK-10381] Fix mixup of taskAttemptNumber & attemptId in OutputCommitCoordinator #8544

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c6220fd
Add failing integration test.
JoshRosen Sep 1, 2015
a09814b
Fix bug by always using task attempt number.
JoshRosen Sep 1, 2015
0059c95
Add MiMa excludes.
JoshRosen Sep 1, 2015
81b86a1
Merge remote-tracking branch 'origin/master' into SPARK-10381
JoshRosen Sep 14, 2015
7af9c6b
Only add MiMa changes for master (will add others during backport)
JoshRosen Sep 15, 2015
035d660
More attempt -> attemptNumber renaming, per Andrew's suggestion.
JoshRosen Sep 15, 2015
0d40f83
Revert "Only add MiMa changes for master (will add others during back…
JoshRosen Sep 15, 2015
4dcb78a
Merge remote-tracking branch 'origin/master' into SPARK-10381
JoshRosen Sep 15, 2015
edbbf6f
Merge remote-tracking branch 'origin/master' into SPARK-10381
JoshRosen Sep 15, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'origin/master' into SPARK-10381
  • Loading branch information
@JoshRosen
JoshRosen committed Sep 14, 2015
commit 81b86a1b542e65d16e2ff3db103339f1b5b7cec0
6 changes: 6 additions & 0 deletions core/src/main/scala/org/apache/spark/TaskEndReason.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,12 @@ case class TaskCommitDenied(
attemptNumber: Int) extends TaskFailedReason {
Copy link
Contributor Author

@JoshRosen JoshRosen Sep 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't want this patch's backporting to be blocked over binary incompatibility concerns, hence my decision to use the attempt number here. The unique-within-a-SparkContext TaskAttemptIDs are Longs, which would require a change to this interface.

The parameter renaming is potentially source-incompatible. If we think that this is a concern, then I can roll back to the old incorrect name and can add a comment.

override def toErrorString: String = s"TaskCommitDenied (Driver denied task commit)" +
s" for job: $jobID, partition: $partitionID, attemptNumber: $attemptNumber"
/**
* If a task failed because its attempt to commit was denied, do not count this failure
* towards failing the stage. This is intended to prevent spurious stage failures in cases
* where many speculative tasks are launched and denied to commit.
*/
override def shouldEventuallyFailJob: Boolean = false
}

/**
Expand Down
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.