Skip to content

[SPARK-21741][ML][PySpark] Python API for DataFrame-based multivariate summarizer #20695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WeichenXu123 wants to merge 10 commits into from
Closed

[SPARK-21741][ML][PySpark] Python API for DataFrame-based multivariate summarizer #20695

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
488d45a
init pr
WeichenXu123 Feb 28, 2018
7d3cb1b
update doctest
WeichenXu123 Feb 28, 2018
001ff46
update doctest
WeichenXu123 Feb 28, 2018
b3e9ddd
update version
WeichenXu123 Mar 1, 2018
e64f795
address bago comment
WeichenXu123 Mar 21, 2018
21edbcd
merge master & resolve conflicts
WeichenXu123 Apr 11, 2018
f7cec51
Merge branch 'master' into py_summarizer
WeichenXu123 Apr 13, 2018
20968c1
address comments
WeichenXu123 Apr 13, 2018
b91dbeb
fix python style
WeichenXu123 Apr 13, 2018
9a4a0ca
address comments
WeichenXu123 Apr 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update doctest
  • Loading branch information
@WeichenXu123
WeichenXu123 committed Feb 28, 2018
commit 001ff4685b7b40e14e678a004eee2a88ee6e1356
4 changes: 4 additions & 0 deletions python/pyspark/ml/stat.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,24 +154,28 @@ class Summarizer(object):
+-----------------------------------+
|[[1.0,1.0,1.0], 1] |
+-----------------------------------+
<BLANKLINE>
>>> df.select(summarizer.summary(df.features)).show(truncate=False)
+--------------------------------+
|aggregate_metrics(features, 1.0)|
+--------------------------------+
|[[1.0,1.5,2.0], 2] |
+--------------------------------+
<BLANKLINE>
>>> df.select(Summarizer.mean(df.features, df.weight)).show(truncate=False)
+--------------+
|mean(features)|
+--------------+
|[1.0,1.0,1.0] |
+--------------+
<BLANKLINE>
>>> df.select(Summarizer.mean(df.features)).show(truncate=False)
+--------------+
|mean(features)|
+--------------+
|[1.0,1.5,2.0] |
+--------------+
<BLANKLINE>

.. versionadded:: 2.3.0

Expand Down