-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-2377] Python API for Streaming #2538
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 1 commit
Commits
Show all changes
365 commits
Select commit
Hold shift + click to select a range
1fd12ae
WIP
giwa c05922c
WIP: added PythonTestInputStream
giwa 1f68b78
WIP
giwa 3dda31a
WIP added test case
giwa 7f96294
added basic operation test cases
giwa fa75d71
delete waste file
giwa 8efa266
fixed PEP-008 violation
giwa 3a671cc
remove export PYSPARK_PYTHON in spark submit
giwa 774f18d
removed unnesessary changes
giwa 33c0f94
edited the comment to add more precise description
giwa 4f2d7e6
added mapValues and flatMapVaules WIP for glom and mapPartitions test
giwa 9767712
WIP: solved partitioned and None is not recognized
giwa 35933e1
broke something
giwa 7051a84
all tests are passed if numSlice is 2 and the numver of each input is…
giwa 99e4bb3
basic function test cases are passed
giwa 580fbc2
modified streaming test case to add coment
giwa 94f2b65
remove waste duplicated code
giwa e9fab72
added saveAsTextFiles and saveAsPickledFiles
giwa 4aa99e4
added TODO coments
giwa 6d8190a
add comments
giwa 14d4c0e
removed wasted print in DStream
giwa 97742fe
added sparkContext as input parameter in StreamingContext
giwa e162822
added gorupByKey testcase
giwa e70f706
added testcase for combineByKey
giwa f1798c4
merge with master
giwa 185fdbf
merge with master
giwa 199e37f
adopted the latest compression way of python command
giwa 58150f5
Changed the test case to focus the test operation
giwa 09a28bf
improve testcases
giwa 268a6a5
Changed awaitTermination not to call awaitTermincation in Scala. Just…
giwa 4dedd2d
change test case not to use awaitTermination
giwa 171edeb
clean up
giwa f0ea311
clean up code
giwa 1d84142
remove unimplement test
giwa 583e66d
move tests for streaming inside streaming directory
giwa b7dab85
improve test case
giwa 0d30109
fixed pep8 violation
giwa 24f95db
clen up examples
giwa 9c85e48
clean up exmples
giwa 7339df2
fixed typo
giwa 9d1de23
revert pom.xml
giwa 4f82c89
remove duplicated import
giwa 50fd6f9
revert pom.xml
giwa 93f7637
fixed explanaiton
giwa acfcaeb
revert pom.xml
giwa 3b27bd4
remove the last brank line
giwa 2ea769e
added comment in dstream._test_output
giwa c97377c
delete inproper comments
giwa 67473a9
delete not implemented functions
giwa d9d59fe
Fix scalastyle errors
1fd6bc7
Merge pull request #2 from mattf/giwa-master
giwa 4afa390
clean up code
giwa d68b568
clean up code
giwa da09768
added StreamingContext.remember
giwa f5bfb70
added StreamingContext.sparkContext
giwa fdc9125
added comment for StreamingContext.sparkContext
giwa ee50c5a
added atexit to handle callback server
giwa f7bc8f9
WIP:added more test for StreamingContext
giwa 150b94c
added some StreamingContextTestSuite
giwa 454981d
initial commit for pySparkStreaming
giwa b406252
comment PythonDStream.PairwiseDStream
87438e2
modify dstream.py to fix indent error
d7b4d6f
added reducedByKey not working yet
1a0f065
implementing transform function in Python
17a74c6
modified the code base on comment in https://github.com/tdas/spark/pu…
494cae5
remove not implemented DStream functions in python
e1df940
revert pom.xml
5bac7ec
revert streaming/pom.xml
d2099d8
sorted the import following Spark coding convention
224fc5e
add empty line
bb7ccf3
remove unused import in python
f746109
initial commit for socketTextStream
0d1b954
fied input of socketTextDStream
ccfd214
added doctest for pyspark.streaming.duration
b31446a
fixed typo of network_workdcount.py
dc6995d
delete old file
c455c8d
added reducedByKey not working yet
6f98e50
reduceByKey is working
15feea9
edit python sparkstreaming example
d3ee86a
added count operation but this implementation need double check
72b9738
fix map function
bab31c1
clean up code
0a8bbbb
clean up codes
678e854
remove waste file
b1d2a30
Implemented DStream.foreachRDD in the Python API using Py4J callback …
tdas 05e991b
Added missing file
tdas 9ab8952
Added extra line.
tdas 84a9668
tried to restart callback server
3b498e1
Kill py4j callback server properly
b349649
Removed the waste line
giwa 3c45cd2
implemented reduce and count function in Dstream
giwa d2c01ba
clean up examples
giwa c462bb3
added stop in StreamingContext
giwa 4d40d63
clean up dstream.py
giwa 29c2bc5
initial commit for testcase
giwa fe648e3
WIP
giwa 8a0fbbc
update comment
giwa 1523b66
WIP
giwa 1df77f5
WIP: added PythonTestInputStream
giwa 9ad6855
WIP
giwa ce2acd2
WIP added test case
giwa 878bad7
added basic operation test cases
giwa f21cab3
delete waste file
giwa 3d37822
fixed PEP-008 violation
giwa 253a863
removed unnesessary changes
giwa bb10956
edited the comment to add more precise description
giwa 270a9e1
added mapValues and flatMapVaules WIP for glom and mapPartitions test
giwa bcdec33
WIP: solved partitioned and None is not recognized
giwa ff14070
broke something
giwa 3000b2b
all tests are passed if numSlice is 2 and the numver of each input is…
giwa 13fb44c
basic function test cases are passed
giwa 18c8723
modified streaming test case to add coment
giwa f76c182
remove waste duplicated code
giwa 74535d4
added saveAsTextFiles and saveAsPickledFiles
giwa 16aa64f
added TODO coments
giwa e54f986
add comments
giwa 10b5b04
removed wasted print in DStream
giwa 10ab87b
added sparkContext as input parameter in StreamingContext
giwa 5625bdc
added gorupByKey testcase
giwa c214199
added testcase for combineByKey
giwa 0b99bec
initial commit for pySparkStreaming
giwa 41886c2
comment PythonDStream.PairwiseDStream
66fcfff
modify dstream.py to fix indent error
38adf95
added reducedByKey not working yet
4bcb318
implementing transform function in Python
247fd74
modified the code base on comment in https://github.com/tdas/spark/pu…
dd6de81
initial commit for socketTextStream
f485b1d
fied input of socketTextDStream
0df7111
delete old file
58591d2
reduceByKey is working
98c2a00
added count operation but this implementation need double check
eb4bf48
fix map function
6197a11
clean up code
2ad7bd3
clean up codes
fe02547
remove waste file
4f07163
Implemented DStream.foreachRDD in the Python API using Py4J callback …
tdas 54b5358
tried to restart callback server
88f7506
Kill py4j callback server properly
1b83354
Removed the waste line
giwa 92e333e
implemented reduce and count function in Dstream
giwa 0b09cff
added stop in StreamingContext
giwa 932372a
clean up dstream.py
giwa 376e3ac
WIP
giwa 1934726
update comment
giwa 019ef38
WIP
giwa 5c04a5f
WIP: added PythonTestInputStream
giwa bd3ba53
WIP
giwa 9cde7c9
WIP added test case
giwa b3b0362
added basic operation test cases
giwa 99410be
delete waste file
giwa c1d546e
fixed PEP-008 violation
giwa af610d3
removed unnesessary changes
giwa 953deb0
edited the comment to add more precise description
giwa f67cf57
added mapValues and flatMapVaules WIP for glom and mapPartitions test
giwa 1e126bf
WIP: solved partitioned and None is not recognized
giwa 795b2cd
broke something
giwa 8dcda84
all tests are passed if numSlice is 2 and the numver of each input is…
giwa c5ecfc1
basic function test cases are passed
giwa 2a06cdb
remove waste duplicated code
giwa 99ce042
added saveAsTextFiles and saveAsPickledFiles
giwa ddd4ee1
added TODO coments
giwa af336b7
add comments
giwa 455e5af
removed wasted print in DStream
giwa 58e41ff
merge with master
giwa e80647e
adopted the latest compression way of python command
giwa c00e091
change test case not to use awaitTermination
giwa 3166d31
clean up
giwa f198d14
clean up code
giwa b171ec3
fixed pep8 violation
giwa f04882c
clen up examples
giwa 62dc7a3
clean up exmples
giwa 7dc7391
fixed typo
giwa 6ae3caa
revert pom.xml
giwa fa4af88
remove duplicated import
giwa 066ba90
revert pom.xml
giwa 8ed93af
fixed explanaiton
giwa fbed8da
revert pom.xml
giwa bebb3f3
remove the last brank line
giwa b0f2015
added comment in dstream._test_output
giwa f385976
delete inproper comments
giwa c0a06bc
delete not implemented functions
giwa 2fdf0de
Fix scalastyle errors
d542743
clean up code
giwa d39f102
added StreamingContext.remember
giwa 63c881a
added StreamingContext.sparkContext
giwa d5f5fcb
added comment for StreamingContext.sparkContext
giwa 8ffdbf1
added atexit to handle callback server
giwa 4a59e1e
WIP:added more test for StreamingContext
giwa 2d32a74
added some StreamingContextTestSuite
giwa e685853
meged with rebased 1.1 branch
giwa 5cdb6fa
changed for SCCallSiteSync
giwa 550dfd9
WIP fixing 1.1 merge
giwa df098fc
Merge branch 'master' into giwa
davies 7f53086
support transform(), refactor and cleanup
davies 7339be0
delete tests
davies bd27874
fix scala style
davies 9a57685
fix python style
davies eec401e
refactor, combine TransformedRDD, fix reuse PythonRDD, fix union
davies bd13026
fix examples
davies d357b70
support windowed dstream
davies c28f520
support updateStateByKey
davies 3f0fb4b
refactor fix tests
davies c499ba0
remove Time and Duration
davies 604323f
enable streaming tests
davies b32774c
move java_import into streaming
davies 74df565
fix print and docs
davies 26ea396
refactor
davies 7001b51
refactor of queueStream()
davies fce0ef5
rafactor of foreachRDD()
davies e059ca2
move check of window into Python
davies 847f9b9
add more docs, add first(), take()
davies b983f0f
address comments
davies 98ac6c2
support ssc.transform()
davies c40c52d
change first(), take(n) to has the same behavior as RDD
davies 6ebceca
add more tests
davies 19797f9
clean up
davies 338580a
change _first(), _take(), _collect() as private API
davies 069a94c
fix the number of partitions during window()
davies e00136b
address comments
davies eed6e2a
rollback not needed changes
davies b98d63f
change private[spark] to private[python]
davies 9a16bd1
change number of partitions during tests
davies 8466916
support checkpoint
davies a13ff34
address comments
davies fa7261b
refactor
davies 6f0da2f
recover from checkpoint
davies d328aca
fix serializer in queueStream
davies ff88bec
rename RDDFunction to TransformFunction
davies 7797c70
refactor
davies bd8a4c2
fix scala style
davies 7a88f9f
rollback RDD.setContext(), use textFileStream() to test checkpointing
davies c2b31cb
Merge branch 'master' of github.com:apache/spark into streaming
davies 54bd92b
improve tests
davies 4d0ea8b
clear reference of SparkEnv after stop
davies 6bb9d91
Merge branch 'master' of github.com:apache/spark into streaming
davies c7bbbce
fix sphinx docs
davies 8071541
Merge branch 'env' into streaming
davies be5e5ff
merge branch of env, make tests stable.
davies d05871e
remove reuse of PythonRDD
davies 37fe06f
use random port for callback server
davies e108ec1
address comments
davies 52c535b
remove fix for sum()
davies 8380064
Merge branch 'master' of github.com:apache/spark into streaming
davies 6db00da
Merge branch 'master' of github.com:apache/spark into streaming
davies bebeb4a
address all comments
davies 02d0575
add wrapper for foreachRDD()
davies 182be73
Merge branch 'master' of github.com:apache/spark into streaming
davies 3e2492b
change updateStateByKey() to easy API
davies 331ecce
fix example
davies 64561e4
fix tests
davies File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
add empty line
- Loading branch information
commit 224fc5e271ef1a58c7b9a7e6f77652d6a0691d7c
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
extra space.