[SPARK-8373][PySpark]Add emptyRDD to pyspark and fix the issue when calling sum on an empty RDD #6826
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #34927 has finished for PR 6826 at commit
|
Member
Author
|
retest this please |
Member
There was a problem hiding this comment.
emptyRDD[T] should produce an RDD of T, not byte arrays right?
Member
Author
There was a problem hiding this comment.
T in the return type is not used. Actually, it can be an arbitrary type.
|
Test build #34935 has finished for PR 6826 at commit
|
Member
Author
|
/cc @davies |
Contributor
|
LGTM |
|
Test build #34937 has finished for PR 6826 at commit
|
Contributor
There was a problem hiding this comment.
Is this API used anywhere? If the answer is "no", we should remove it.
Contributor
There was a problem hiding this comment.
I think this is just bringing it to parity with the scala SparkContext API, which we currently also only use in tests. This is probably fine.
Contributor
There was a problem hiding this comment.
This class is PythonRDD, is not JavaSparkContext.
Member
Author
asfgit
pushed a commit
that referenced
this pull request
Jun 17, 2015
… calling sum on an empty RDD This PR fixes the sum issue and also adds `emptyRDD` so that it's easy to create a test case. Author: zsxwing <[email protected]> Closes #6826 from zsxwing/python-emptyRDD and squashes the following commits: b36993f [zsxwing] Update the return type to JavaRDD[T] 71df047 [zsxwing] Add emptyRDD to pyspark and fix the issue when calling sum on an empty RDD (cherry picked from commit 0fc4b96) Signed-off-by: Andrew Or <[email protected]>
asfgit
pushed a commit
that referenced
this pull request
Jun 18, 2015
This is a follow-up PR to remove unused `PythonRDD.emptyRDD` added by #6826 Author: zsxwing <[email protected]> Closes #6867 from zsxwing/remove-PythonRDD-emptyRDD and squashes the following commits: b66d363 [zsxwing] Remove PythonRDD.emptyRDD
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 19, 2015
… calling sum on an empty RDD This PR fixes the sum issue and also adds `emptyRDD` so that it's easy to create a test case. Author: zsxwing <[email protected]> Closes apache#6826 from zsxwing/python-emptyRDD and squashes the following commits: b36993f [zsxwing] Update the return type to JavaRDD[T] 71df047 [zsxwing] Add emptyRDD to pyspark and fix the issue when calling sum on an empty RDD
nemccarthy
pushed a commit
to nemccarthy/spark
that referenced
this pull request
Jun 19, 2015
This is a follow-up PR to remove unused `PythonRDD.emptyRDD` added by apache#6826 Author: zsxwing <[email protected]> Closes apache#6867 from zsxwing/remove-PythonRDD-emptyRDD and squashes the following commits: b66d363 [zsxwing] Remove PythonRDD.emptyRDD
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR fixes the sum issue and also adds
emptyRDDso that it's easy to create a test case.