Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
cbf7837
add tests
jose-torres Dec 5, 2017
10ac599
writer impl
jose-torres Dec 5, 2017
5594f7a
rm useless writer
jose-torres Dec 5, 2017
a364fa6
rm weird docs
jose-torres Dec 5, 2017
41d732c
shuffle around public interfaces
jose-torres Dec 5, 2017
60c12c2
fix imports
jose-torres Dec 5, 2017
57db413
put deserialize in reader so we don't have to port SerializedOffset
jose-torres Dec 5, 2017
93b6976
off by one errors grr
jose-torres Dec 5, 2017
abd20db
document right semantics
jose-torres Dec 5, 2017
3629d27
document checkpoint location
jose-torres Dec 6, 2017
d021f31
add getStart to continuous and clarify semantics
jose-torres Dec 6, 2017
052808a
cleanup offset set/get docs
jose-torres Dec 6, 2017
7a7638b
cleanup reader docs
jose-torres Dec 6, 2017
81efbee
explain getOffset
jose-torres Dec 6, 2017
a9c43d9
fix fmt
jose-torres Dec 6, 2017
3f03f50
fix doc
jose-torres Dec 6, 2017
6338043
note interfaces are temporary
jose-torres Dec 6, 2017
8949571
fix wording
jose-torres Dec 6, 2017
df9f224
lifecycle
jose-torres Dec 6, 2017
ddaee34
fix offset semantic implementation
jose-torres Dec 6, 2017
1608444
remove unneeded restriction
jose-torres Dec 7, 2017
49525b4
deserializeOffset
jose-torres Dec 7, 2017
a9fbf33
add copyright headers
jose-torres Dec 7, 2017
22d07cd
rebase against datasource package change
jose-torres Dec 7, 2017
0b68873
refer properly to sink
jose-torres Dec 8, 2017
f924a8a
widen tolerance interval in continuous rate source test
jose-torres Dec 8, 2017
4d166de
widen tolerance interval
jose-torres Dec 8, 2017
7c46b33
clarify start offset
jose-torres Dec 11, 2017
8a2a4f1
add docs
jose-torres Dec 13, 2017
8809bf9
fix problems
jose-torres Dec 13, 2017
0974ac3
even timestamps
jose-torres Dec 13, 2017
c1a6322
unify offsets
jose-torres Dec 13, 2017
3cb6cee
undo spurious rename
jose-torres Dec 13, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
explain getOffset
  • Loading branch information
jose-torres committed Dec 7, 2017
commit 81efbeeca9c41a52bf2112c11f4ec4f042dc2b52
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,8 @@ public interface MicroBatchReadSupport extends DataSourceV2 {
* @param options the options for the returned data source reader, which is an immutable
* case-insensitive string-to-string map.
*/
MicroBatchReader createMicroBatchReader(Optional<StructType> schema, String checkpointLocation, DataSourceV2Options options);
MicroBatchReader createMicroBatchReader(
Optional<StructType> schema,
String checkpointLocation,
DataSourceV2Options options);
}
Original file line number Diff line number Diff line change
Expand Up @@ -8,22 +8,12 @@
* A variation on {@link DataReader} for use with streaming in continuous processing mode.
*/
public interface ContinuousDataReader<T> extends DataReader<T> {
/**
* Proceed to next record, returning false only if the read is interrupted.
*
* @throws IOException if failure happens during disk/network IO like reading files.
*/
boolean next() throws IOException;

/**
* Return the current record. This method should return same value until `next` is called.
*/
T get();

/**
* Get the offset of the current record.
*
* The execution engine will use this offset as a restart checkpoint.
* The execution engine will call this method along with get() to keep track of the current
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better to use a real java doc link, e.g. {@link DataReader#get}

* offset. When an epoch ends, the offset of the previous record in each partition will be saved
* as a restart checkpoint.
*/
PartitionOffset getOffset();
}