Skip to content

[SPARK-23559][SS] Create StreamingDataWriterFactory for epoch ID. #20752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jose-torres wants to merge 5 commits into from
Closed

[SPARK-23559][SS] Create StreamingDataWriterFactory for epoch ID. #20752

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c6d4ff5
Create StreamingDataWriterFactory for epoch ID.
jose-torres Mar 6, 2018
3cf5479
remove override in StreamWriter - incompatible with SupportsWriteInte…
jose-torres Mar 6, 2018
d4d765b
add header
jose-torres Mar 6, 2018
9c276f3
correct for no override
jose-torres Mar 6, 2018
0c68fd1
fix docs again
jose-torres Mar 6, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
fix docs again
  • Loading branch information
@jose-torres
jose-torres committed Mar 6, 2018
commit 0c68fd1bfdbd45988263ebc9c627b1b27f7af281
4 changes: 2 additions & 2 deletions sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
import org.apache.spark.annotation.InterfaceStability;

/**
* A data writer returned by {@link DataWriterFactory#createDataWriter(int, int, long)} and is
* A data writer returned by {@link DataWriterFactory#createDataWriter(int, int)} and is
* responsible for writing data for an input RDD partition.
*
* One Spark task has one exclusive data writer, so there is no thread-safe concern.
Expand All @@ -39,7 +39,7 @@
* {@link DataSourceWriter#commit(WriterCommitMessage[])} with commit messages from other data
* writers. If this data writer fails(one record fails to write or {@link #commit()} fails), an
* exception will be sent to the driver side, and Spark may retry this writing task a few times.
* In each retry, {@link DataWriterFactory#createDataWriter(int, int, long)} will receive a
* In each retry, {@link DataWriterFactory#createDataWriter(int, int)} will receive a
* different `attemptNumber`. Spark will call {@link DataSourceWriter#abort(WriterCommitMessage[])}
* when the configured number of retries is exhausted.
*
Expand Down
2 changes: 2 additions & 0 deletions ...ain/java/org/apache/spark/sql/sources/v2/writer/streaming/StreamingDataWriterFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,11 @@

package org.apache.spark.sql.sources.v2.writer.streaming;

import org.apache.spark.annotation.InterfaceStability;
import org.apache.spark.sql.sources.v2.writer.DataWriter;
import org.apache.spark.sql.sources.v2.writer.DataWriterFactory;

@InterfaceStability.Evolving
public interface StreamingDataWriterFactory<T> extends DataWriterFactory<T> {
/**
* Returns a data writer to do the actual writing work.
Expand Down