[SPARK-2515][mllib] Chi Squared test #1733

dorx · 2014-08-02T03:50:39Z

No description provided.

SparkQA · 2014-08-02T03:54:02Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17744/consoleFull

SparkQA · 2014-08-02T03:54:50Z

QA results for PR 1733:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17744/consoleFull

SparkQA · 2014-08-06T02:19:42Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17974/consoleFull

dorx · 2014-08-06T02:21:06Z

@mengxr @jkbradley @falaki
PR ready for review now.

SparkQA · 2014-08-06T02:24:45Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17975/consoleFull

mengxr · 2014-08-06T02:30:42Z

remove space between @ and jkbradley ~ :)

mengxr · 2014-08-06T03:18:04Z

mllib/src/main/scala/org/apache/spark/mllib/stat/Statistics.scala

Shall we call it chiSqTest (following R's)? We need test in the method name because X_2 is also a distribution. I feel chiSqTest may be better than chiSquaredTest because it is also called chi-square test without d.

chiSqTest sounds good.

SparkQA · 2014-08-06T03:19:22Z

QA results for PR 1733:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17974/consoleFull

mengxr · 2014-08-06T03:22:40Z

mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala

documentation

SparkQA · 2014-08-06T03:24:48Z

QA results for PR 1733:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17975/consoleFull

mengxr · 2014-08-06T03:52:21Z

I think we should either allow user to input the raw observations or use Map[_, Long] for input frequencies. I'm going to take a look at R's implementation ...

mengxr · 2014-08-06T06:26:11Z

@dorx I checked R's implementation and finally figured out what is going on.

When only a vector x is given, it is treated as a vector containing frequency counts for categories and tested against multinomial distribution.
When a matrix x is given, it is treated as a contingency table and the test is for independence.
When both x and y are given, both vectors are treated as factors (categorical values) and the test is for independence.

I want to suggest the following APIs:

// test observed frequencies against multinomial distribution with
// `p = (1/n, 1/n, ..., 1/n)`
def chiSqTest(counts: Vector)

// test observed frequencies against the given multinomial distribution
def chiSqTest(counts: Vector, p: Vector)

// test independence using the given contingency table 
def chiSqTest(counts: Matrix)

// test independence using the given observed pairs (assuming categorical values)
def chiSqTest[V1, V2](observations: RDD[(V1, V2)])

mengxr · 2014-08-06T15:04:12Z

The previous proposal may be hard to implement in Python. Another solution would be separate goodness-of-fit test from independence test, e.g., chiSqGofTest and chiSqIndTest.

def chiSqGofTest(counts: Vector)

def chiSqGofTest(counts: Vector, p: Vector)

def chiSqIndTest(counts: Matrix)

def chiSqIndTest[V1, V2](observations: RDD[(V1, V2)])

We can also add direct RDD support, which may be unnecessary:

def chiSqGofTest[V](observations: RDD[V], p: Map[V, Double])

Since we only support pearson, we can hide method in the public API for now.

SparkQA · 2014-08-08T20:09:34Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18217/consoleFull

SparkQA · 2014-08-08T21:00:01Z

QA results for PR 1733:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18217/consoleFull

SparkQA · 2014-08-08T21:04:40Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18226/consoleFull

SparkQA · 2014-08-08T21:59:12Z

QA results for PR 1733:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18226/consoleFull

mengxr · 2014-08-09T03:22:18Z

mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSquaredTest.scala

could be initialized as a null

SparkQA · 2014-08-11T21:34:45Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18333/consoleFull

SparkQA · 2014-08-11T22:26:05Z

QA results for PR 1733:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18333/consoleFull

mengxr · 2014-08-11T22:29:58Z

mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSqTest.scala

p-value should be 0 here (strongly against the null)

SparkQA · 2014-08-11T23:14:53Z

QA tests have started for PR 1733. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18340/consoleFull

SparkQA · 2014-08-12T00:06:30Z

QA results for PR 1733:
- This patch PASSES unit tests.
- This patch merges cleanly
- This patch adds no public classes

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18340/consoleFull

Author: Doris Xin <[email protected]> Closes #1733 from dorx/chisquare and squashes the following commits: cafb3a7 [Doris Xin] fixed p-value for extreme case. d286783 [Doris Xin] Merge branch 'master' into chisquare e95e485 [Doris Xin] reviewer comments. 7dde711 [Doris Xin] ChiSqTestResult renaming and changed to Class 80d03e2 [Doris Xin] Reviewer comments. c39eeb5 [Doris Xin] units passed with updated API e90d90a [Doris Xin] Merge branch 'master' into chisquare 7eea80b [Doris Xin] WIP d64c2fb [Doris Xin] Merge branch 'master' into chisquare 5686082 [Doris Xin] facelift bc7eb2e [Doris Xin] unit passed; still need docs and some refactoring 50703a5 [Doris Xin] merge master 4e4e361 [Doris Xin] WIP e6b83f3 [Doris Xin] reviewer comments 3d61582 [Doris Xin] input names 706d436 [Doris Xin] Added API for RDD[Vector] 6598379 [Doris Xin] API and code structure. ff17423 [Doris Xin] WIP (cherry picked from commit 32638b5) Signed-off-by: Xiangrui Meng <[email protected]>

mengxr · 2014-08-12T02:23:13Z

LGTM. Merged into both master and branch-1.1. Thanks!

Author: Doris Xin <[email protected]> Closes apache#1733 from dorx/chisquare and squashes the following commits: cafb3a7 [Doris Xin] fixed p-value for extreme case. d286783 [Doris Xin] Merge branch 'master' into chisquare e95e485 [Doris Xin] reviewer comments. 7dde711 [Doris Xin] ChiSqTestResult renaming and changed to Class 80d03e2 [Doris Xin] Reviewer comments. c39eeb5 [Doris Xin] units passed with updated API e90d90a [Doris Xin] Merge branch 'master' into chisquare 7eea80b [Doris Xin] WIP d64c2fb [Doris Xin] Merge branch 'master' into chisquare 5686082 [Doris Xin] facelift bc7eb2e [Doris Xin] unit passed; still need docs and some refactoring 50703a5 [Doris Xin] merge master 4e4e361 [Doris Xin] WIP e6b83f3 [Doris Xin] reviewer comments 3d61582 [Doris Xin] input names 706d436 [Doris Xin] Added API for RDD[Vector] 6598379 [Doris Xin] API and code structure. ff17423 [Doris Xin] WIP

bump boson to 0.3.1 Patch list: https://github.pie.apple.com/pie/boson/compare/8a8fbab4fd...658ed31b85

dorx added 8 commits July 25, 2014 12:31

WIP

ff17423

API and code structure.

6598379

Added API for RDD[Vector]

706d436

input names

3d61582

reviewer comments

e6b83f3

WIP

4e4e361

merge master

50703a5

unit passed; still need docs and some refactoring

bc7eb2e

dorx added 2 commits August 5, 2014 19:16

facelift

5686082

Merge branch 'master' into chisquare

d64c2fb

mengxr reviewed Aug 6, 2014
View reviewed changes

mllib/src/main/scala/org/apache/spark/mllib/stat/test/TestResult.scala Outdated

Copy link

Contributor

mengxr Aug 6, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

documentation

dorx added 2 commits August 7, 2014 13:32

WIP

7eea80b

Merge branch 'master' into chisquare

e90d90a

ChiSqTestResult renaming and changed to Class

7dde711

mengxr reviewed Aug 9, 2014
View reviewed changes

mllib/src/main/scala/org/apache/spark/mllib/stat/test/ChiSquaredTest.scala Outdated

Copy link

Contributor

mengxr Aug 9, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could be initialized as a null

reviewer comments.

e95e485

mengxr reviewed Aug 11, 2014
View reviewed changes

dorx added 2 commits August 11, 2014 15:40

Merge branch 'master' into chisquare

d286783

fixed p-value for extreme case.

cafb3a7

asfgit closed this in 32638b5 Aug 12, 2014

snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Jun 20, 2023

rdar://107970457 Bump Boson to 0.3.1 (apache#1733)

40e82dd

bump boson to 0.3.1 Patch list: https://github.pie.apple.com/pie/boson/compare/8a8fbab4fd...658ed31b85

[SPARK-2515][mllib] Chi Squared test #1733

[SPARK-2515][mllib] Chi Squared test #1733

Uh oh!

Conversation

dorx commented Aug 2, 2014

Uh oh!

SparkQA commented Aug 2, 2014

Uh oh!

SparkQA commented Aug 2, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

dorx commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

mengxr commented Aug 6, 2014

Uh oh!

mengxr Aug 6, 2014

Choose a reason for hiding this comment

Uh oh!

dorx Aug 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

mengxr Aug 6, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 6, 2014

Uh oh!

mengxr commented Aug 6, 2014

Uh oh!

mengxr commented Aug 6, 2014

Uh oh!

mengxr commented Aug 6, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

SparkQA commented Aug 8, 2014

Uh oh!

mengxr Aug 9, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

mengxr Aug 11, 2014

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Aug 11, 2014

Uh oh!

SparkQA commented Aug 12, 2014

Uh oh!

mengxr commented Aug 12, 2014

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants