Skip to content

Initial draft of Streaming Dataframe infrastructure #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
marmbrus merged 24 commits into from
Jan 7, 2016
Merged

Initial draft of Streaming Dataframe infrastructure #21

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
e238911
first draft
marmbrus Dec 10, 2015
d2706b5
working on state
marmbrus Dec 11, 2015
7a3590f
working on stateful streaming
marmbrus Dec 11, 2015
c8a9238
now with event time windows
marmbrus Dec 12, 2015
dddd192
some refactoring after talking to ali
marmbrus Dec 13, 2015
a0a1e7b
docs
marmbrus Dec 15, 2015
15bed31
start kinesis
marmbrus Dec 15, 2015
89464a9
some renaming
marmbrus Dec 17, 2015
d133dbb
WIP: file source
marmbrus Dec 28, 2015
e3c4c83
Merge remote-tracking branch 'origin/master' into streaming-infra
marmbrus Jan 5, 2016
90fa6d3
remove half-baked stateful implementation
marmbrus Jan 5, 2016
b1c1dc6
cleanup
marmbrus Jan 5, 2016
9205068
more docs
marmbrus Jan 5, 2016
eab186d
rollback changes
marmbrus Jan 6, 2016
cd575db
some feedback
marmbrus Jan 6, 2016
d233eae
comments
marmbrus Jan 6, 2016
3423dce
Merge remote-tracking branch 'marmbrus/streaming-df' into streaming-i…
marmbrus Jan 6, 2016
b0b20e5
Update circle.yml
marmbrus Jan 6, 2016
f5d9642
Update SparkBuild.scala
marmbrus Jan 6, 2016
f8911e4
Update circle.yml
marmbrus Jan 6, 2016
6387781
Add newlines to satisfy Scalastyle
JoshRosen Jan 6, 2016
6242bc2
revert CI changes
marmbrus Jan 6, 2016
95fd978
update based on TD's comments
marmbrus Jan 6, 2016
f928595
style and docs
marmbrus Jan 6, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
some feedback
  • Loading branch information
@marmbrus
marmbrus committed Jan 6, 2016
commit cd575db5bb4ab1e954f80a3b0d044ab492394443
5 changes: 3 additions & 2 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StreamExecution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,8 @@ class StreamExecution(
microBatchThread.setDaemon(true)
microBatchThread.start()

var lastExecution: QueryExecution = null
@volatile
private[sql] var lastExecution: QueryExecution = null

/**
* Checks to see if any new data is present in any of the sources. When new data is available,
Expand Down Expand Up @@ -128,7 +129,7 @@ class StreamExecution(
*/
def stop(): Unit = {
shouldRun = false
while (microBatchThread.isAlive) { Thread.sleep(100) }
if (microBatchThread.isAlive) { microBatchThread.join() }
}

/**
Expand Down