Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
collectPartitions()
  • Loading branch information
davies committed Aug 5, 2014
commit 4ffae0031e1f00641845fc5e9e3b62f54e7c56ad
10 changes: 8 additions & 2 deletions python/pyspark/rdd.py
Original file line number Diff line number Diff line change
Expand Up @@ -737,10 +737,16 @@ def _collect_iterator_through_file(self, iterator):

def collectPartitions(self, partitions):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the Scala API, this is marked as a private API used only for tests. Is there a non-test usecase for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will help for debug, you can collect parts of the RDD to investigate with them.

It also be helpful if we have an API called slice(start, [end]) to select parts of the partitions. DPark has this kind of API, it help us a lot, Narrow down the data to do fast debugging.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Josh, let's delete this for now. We can open a separate JIRA about making it public and maybe discuss there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW I do like a slice-based API in general, that might be what we propose publicly.

"""
Return an array that contains all of the elements in a specific
Return a list of list that contains all of the elements in a specific
partition of this RDD.

>>> rdd = sc.parallelize(range(8), 4)
>>> rdd.collectPartitions([1, 3])
[[2, 3], [6, 7]]
"""
raise NotImplementedError

return [self.ctx.runJob(self, lambda it: it, [p], True)
for p in partitions]

def reduce(self, f):
"""
Expand Down