Skip to content

Spark 1165 rdd.intersection in python and java #80

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ScrapCodes wants to merge 4 commits into from
+75 −0
Closed

Spark 1165 rdd.intersection in python and java #80

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d6effee
SPARK-1165 Implemented RDD.intersection in python.
ScrapCodes Mar 5, 2014
d0c71f3
SPARK-1165 RDD.intersection in java
ScrapCodes Mar 5, 2014
1fea813
correct the lines wrapping
ScrapCodes Mar 7, 2014
9b015e9
Added a note, shuffle is required for intersection.
ScrapCodes Mar 8, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
SPARK-1165 Implemented RDD.intersection in python.
  • Loading branch information
@ScrapCodes
ScrapCodes committed Mar 5, 2014
commit d6effee4ee967f15210d0d57526beab4e3f9c8e2
13 changes: 13 additions & 0 deletions python/pyspark/rdd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -319,6 +319,19 @@ def union(self, other):
return RDD(self_copy._jrdd.union(other_copy._jrdd), self.ctx,
self.ctx.serializer)

def intersection(self, other):
"""
Return the intersection of this RDD and another one.

>>> rdd1 = sc.parallelize([1, 10, 2, 3, 4, 5])
>>> rdd2 = sc.parallelize([1, 6, 2, 3, 7, 8])
>>> rdd1.intersection(rdd2).collect()
[1, 2, 3]
"""
return self.map(lambda v: (v, None)).cogroup(
other.map(lambda v: (v, None))).filter(
lambda x: (len(x[1][0]) != 0) and (len(x[1][1]) != 0)).keys()
Copy link
Contributor

@mateiz mateiz Mar 6, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably slightly nicer to write this like this:

return self.map(lambda v: (v, None)) \
    .cogroup(other.map(lambda v: (v, None))) \
    .filter(lambda x: (len(x[1][0]) != 0) and (len(x[1][1]) != 0)) \
    .keys()

Or put parens around the whole thing to avoid Python thinking lines ended (but we used the backslash style before).

Other than that it looks good.


def _reserialize(self):
if self._jrdd_deserializer == self.ctx.serializer:
return self
Expand Down