Skip to content

[SPARK-23559][SS] Add epoch ID to DataWriterFactory. #20710

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jose-torres wants to merge 11 commits into from
Closed

[SPARK-23559][SS] Add epoch ID to DataWriterFactory. #20710

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
42bca60
change api
jose-torres Feb 21, 2018
84cfa21
add docs
jose-torres Feb 21, 2018
a18a57b
don't use epochId in batch sources
jose-torres Feb 22, 2018
b2ee7f3
change comment
jose-torres Feb 22, 2018
55b38db
fix docs
jose-torres Mar 2, 2018
544eb1b
fix docs
jose-torres Mar 2, 2018
f5948e8
clarify DataWriter lifecycle
jose-torres Mar 3, 2018
4588616
fix foreach
jose-torres Mar 3, 2018
79495b1
various doc fixes
jose-torres Mar 3, 2018
9fb74e2
describe what's an epoch
jose-torres Mar 3, 2018
215c225
clarify retry docs
jose-torres Mar 3, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
clarify retry docs
  • Loading branch information
@jose-torres
jose-torres committed Mar 3, 2018
commit 215c225c5a1623cfa02f617201e21067bbf6088a
8 changes: 4 additions & 4 deletions sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@
* succeeds), a {@link WriterCommitMessage} will be sent to the driver side and pass to
* {@link DataSourceWriter#commit(WriterCommitMessage[])} with commit messages from other data
* writers. If this data writer fails(one record fails to write or {@link #commit()} fails), an
* exception will be sent to the driver side, and Spark may retry this writing task for some times,
* each time {@link DataWriterFactory#createDataWriter(int, int, long)} gets a different
* `attemptNumber`, and finally call {@link DataSourceWriter#abort(WriterCommitMessage[])} if all
* retry fail.
* exception will be sent to the driver side, and Spark may retry this writing task a few times.
* In each retry, {@link DataWriterFactory#createDataWriter(int, int, long)} will receive a
* different `attemptNumber`. Spark will call {@link DataSourceWriter#abort(WriterCommitMessage[])}
Copy link
Contributor

@tdas tdas Mar 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not clear to me. Isnt it the case that abort will be called every time a task attempt ends in an error?
This seems to give the impression that abort is called only after N failed attempts have been made.

Copy link
Contributor Author

@jose-torres jose-torres Mar 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The local abort will be called every time a task attempt fails. The global abort referenced here is called only when the job fails.

* when the configured number of retries is exhausted.
*
* Besides the retry mechanism, Spark may launch speculative tasks if the existing writing task
* takes too long to finish. Different from retried tasks, which are launched one by one after the
Expand Down