Skip to content

[SPARK-23559][SS] Add epoch ID to DataWriterFactory. #20710

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jose-torres wants to merge 11 commits into from
Closed

[SPARK-23559][SS] Add epoch ID to DataWriterFactory. #20710

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
42bca60
change api
jose-torres Feb 21, 2018
84cfa21
add docs
jose-torres Feb 21, 2018
a18a57b
don't use epochId in batch sources
jose-torres Feb 22, 2018
b2ee7f3
change comment
jose-torres Feb 22, 2018
55b38db
fix docs
jose-torres Mar 2, 2018
544eb1b
fix docs
jose-torres Mar 2, 2018
f5948e8
clarify DataWriter lifecycle
jose-torres Mar 3, 2018
4588616
fix foreach
jose-torres Mar 3, 2018
79495b1
various doc fixes
jose-torres Mar 3, 2018
9fb74e2
describe what's an epoch
jose-torres Mar 3, 2018
215c225
clarify retry docs
jose-torres Mar 3, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix docs
  • Loading branch information
@jose-torres
jose-torres committed Mar 2, 2018
commit 544eb1b296bceb213965bf3c5dc1a6264c5b7acd
7 changes: 4 additions & 3 deletions sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
import org.apache.spark.annotation.InterfaceStability;

/**
* A data writer returned by {@link DataWriterFactory#createDataWriter(int, int)} and is
* A data writer returned by {@link DataWriterFactory#createDataWriter(int, int, long)} and is
* responsible for writing data for an input RDD partition.
*
* One Spark task has one exclusive data writer, so there is no thread-safe concern.
Expand All @@ -36,8 +36,9 @@
* {@link DataSourceWriter#commit(WriterCommitMessage[])} with commit messages from other data
* writers. If this data writer fails(one record fails to write or {@link #commit()} fails), an
* exception will be sent to the driver side, and Spark will retry this writing task for some times,
Copy link
Contributor

@tdas tdas Mar 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark may retry... (in continuous we dont retry the task)

Copy link
Contributor

@tdas tdas Mar 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for some times --> for a few times

Copy link
Contributor

@tdas tdas Mar 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Break this sentence. very long.

* each time {@link DataWriterFactory#createDataWriter(int, int)} gets a different `attemptNumber`,
* and finally call {@link DataSourceWriter#abort(WriterCommitMessage[])} if all retry fail.
* each time {@link DataWriterFactory#createDataWriter(int, int, long)} gets a different
* `attemptNumber`, and finally call {@link DataSourceWriter#abort(WriterCommitMessage[])} if all
* retry fail.
*
* Besides the retry mechanism, Spark may launch speculative tasks if the existing writing task
* takes too long to finish. Different from retried tasks, which are launched one by one after the
Expand Down