-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-23559][SS] Add epoch ID to DataWriterFactory. #20710
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
42bca60
84cfa21
a18a57b
b2ee7f3
55b38db
544eb1b
f5948e8
4588616
79495b1
9fb74e2
215c225
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -49,8 +49,8 @@ public interface DataWriterFactory<T> extends Serializable { | |
| * tasks with the same task id running at the same time. Implementations can | ||
| * use this attempt number to distinguish writers of different task attempts. | ||
| * @param epochId A monotonically increasing id for streaming queries that are split in to | ||
| * discrete periods of execution. For queries that execute as a single batch, this | ||
| * id will always be zero. | ||
| * discrete periods of execution. For non-streaming queries, | ||
| * this ID will always be 0. | ||
| */ | ||
| DataWriter<T> createDataWriter(int partitionId, int attemptNumber, long epochId); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why are we using the same interface for streaming and batch here? Is there a compelling reason to do so instead of adding
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The guarantees are identical, and in the current execution model, each epoch is in fact processed by a single batch job. |
||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add clear lifecycle semantics.