Skip to content

Spark 1246 add min max to stat counter #144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dwmclary wants to merge 12 commits into from
Closed

Spark 1246 add min max to stat counter #144

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
4916016
added histogram method, added max and min to statscounter
dwmclary Mar 11, 2014
eaf89d9
added correct doctest for histogram
dwmclary Mar 11, 2014
29981f2
fixed indentation on doctest comment
dwmclary Mar 11, 2014
37a7dea
cleaned up boundaries for histogram -- uses real min/max when buckets…
dwmclary Mar 11, 2014
1e7056d
added underscore to getBucket
dwmclary Mar 12, 2014
ed67136
broke min/max out into separate transaction, added to rdd.py
dwmclary Mar 14, 2014
a5c13b0
Added min and max to Scala and Java RDD, added min and max to StatCou…
dwmclary Mar 14, 2014
1a97558
added max and min to StatCounter output, updated doc
dwmclary Mar 15, 2014
21dd366
added max and min to StatCounter output, updated doc
dwmclary Mar 15, 2014
5d96799
added max and min to StatCounter repr for pyspark
dwmclary Mar 15, 2014
82cde0e
flipped incorrectly assigned inf values in StatCounter
dwmclary Mar 15, 2014
fd3fd4b
fixed error, updated test
dwmclary Mar 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added underscore to getBucket
  • Loading branch information
@dwmclary
dwmclary committed Mar 12, 2014
commit 1e7056db8a9ce0364e46ad0e269bc1de9f9925f4
4 changes: 2 additions & 2 deletions python/pyspark/rdd.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,7 +612,7 @@ def sampleVariance(self):
"""
return self.stats().sampleVariance()

def getBuckets(self, bucketCount):
def _getBuckets(self, bucketCount):
#use the statscounter as a quick way of getting max and min
mm_stats = self.stats()
min = mm_stats.min()
Expand All @@ -637,7 +637,7 @@ def histogram(self, bucketCount, buckets=None):
max = float("inf")
evenBuckets = False
if not buckets:
b = self.getBuckets(bucketCount)
b = self._getBuckets(bucketCount)
buckets = b["buckets"]
min = b["min"]
max = b["max"]
Expand Down