Skip to content

[SPARK-22732] Add Structured Streaming APIs to DataSourceV2 #19925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jose-torres wants to merge 33 commits into from
Closed

[SPARK-22732] Add Structured Streaming APIs to DataSourceV2 #19925

Changes from 1 commit
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
cbf7837
add tests
jose-torres Dec 5, 2017
10ac599
writer impl
jose-torres Dec 5, 2017
5594f7a
rm useless writer
jose-torres Dec 5, 2017
a364fa6
rm weird docs
jose-torres Dec 5, 2017
41d732c
shuffle around public interfaces
jose-torres Dec 5, 2017
60c12c2
fix imports
jose-torres Dec 5, 2017
57db413
put deserialize in reader so we don't have to port SerializedOffset
jose-torres Dec 5, 2017
93b6976
off by one errors grr
jose-torres Dec 5, 2017
abd20db
document right semantics
jose-torres Dec 5, 2017
3629d27
document checkpoint location
jose-torres Dec 6, 2017
d021f31
add getStart to continuous and clarify semantics
jose-torres Dec 6, 2017
052808a
cleanup offset set/get docs
jose-torres Dec 6, 2017
7a7638b
cleanup reader docs
jose-torres Dec 6, 2017
81efbee
explain getOffset
jose-torres Dec 6, 2017
a9c43d9
fix fmt
jose-torres Dec 6, 2017
3f03f50
fix doc
jose-torres Dec 6, 2017
6338043
note interfaces are temporary
jose-torres Dec 6, 2017
8949571
fix wording
jose-torres Dec 6, 2017
df9f224
lifecycle
jose-torres Dec 6, 2017
ddaee34
fix offset semantic implementation
jose-torres Dec 6, 2017
1608444
remove unneeded restriction
jose-torres Dec 7, 2017
49525b4
deserializeOffset
jose-torres Dec 7, 2017
a9fbf33
add copyright headers
jose-torres Dec 7, 2017
22d07cd
rebase against datasource package change
jose-torres Dec 7, 2017
0b68873
refer properly to sink
jose-torres Dec 8, 2017
f924a8a
widen tolerance interval in continuous rate source test
jose-torres Dec 8, 2017
4d166de
widen tolerance interval
jose-torres Dec 8, 2017
7c46b33
clarify start offset
jose-torres Dec 11, 2017
8a2a4f1
add docs
jose-torres Dec 13, 2017
8809bf9
fix problems
jose-torres Dec 13, 2017
0974ac3
even timestamps
jose-torres Dec 13, 2017
c1a6322
unify offsets
jose-torres Dec 13, 2017
3cb6cee
undo spurious rename
jose-torres Dec 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rm useless writer
  • Loading branch information
@jose-torres
jose-torres committed Dec 7, 2017
commit 5594f7a5770b1a39ea86003fc6aa8ef9d93bc3c1
1 change: 0 additions & 1 deletion sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/MemorySinkV2Suite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,6 @@ class MemorySinkV2Suite extends StreamTest with BeforeAndAfter {

test("microbatch writer") {
val sink = new MemorySinkV2
val writer = new ContinuousMemoryWriter(sink, OutputMode.Append())
new MemoryWriter(sink, 0, OutputMode.Append()).commit(
Array(
MemoryWriterCommitMessage(0, Seq(Row(1), Row(2))),
Expand Down