Skip to content

[SPARK-8598] [MLlib] Implementation of 1-sample, two-sided, Kolmogorov Smirnov Test for RDDs #6994

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
josepablocam wants to merge 27 commits into from
Closed

[SPARK-8598] [MLlib] Implementation of 1-sample, two-sided, Kolmogorov Smirnov Test for RDDs #6994

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
13dfe4d
created test result class for ks test
Jun 24, 2015
c659ea1
created KS test class
Jun 24, 2015
4da189b
added user facing ks test functions
Jun 24, 2015
ce8e9a1
added kstest testing in HypothesisTestSuite
Jun 24, 2015
b9cff3a
made small changes to pass style check
Jun 24, 2015
f6951b6
changed style and some comments based on feedback from pull request
Jun 24, 2015
c18dc66
removed ksTestOpt from API and changed comments in HypothesisTestSuit…
Jun 24, 2015
16b5c4c
renamed dat to data and eliminated recalc of RDD size by sharing as a…
Jun 25, 2015
0b5e8ec
changed KS one sample test to perform just 1 distributed pass (in add…
Jun 25, 2015
4b8ba61
fixed off by 1/N in cases when post-constant adjustment ecdf is above…
Jun 25, 2015
6a4784f
specified what distributions are available for the convenience method…
Jun 26, 2015
992293b
Style changes as per comments and added implementation note explainin…
Jun 26, 2015
3f81ad2
renamed ks1 sample test for clarity
Jun 26, 2015
9c0f1af
additional style changes incorporated and added documentation to mlli…
Jun 26, 2015
1226b30
reindent multi-line lambdas, prior intepretation of style guide was w…
Jun 29, 2015
9026895
addressed style changes, correctness change to simpler approach, and …
Jul 7, 2015
3288e42
addressed style changes, correctness change to simpler approach, and …
Jul 7, 2015
e760ebd
line length changes to fit style check
Jul 7, 2015
7e66f57
copied implementation note to public api docs, and added @see for lin…
Jul 7, 2015
a4bc0c7
changed ksTest(data, distName) to ksTest(data, distName, params*) aft…
Jul 8, 2015
2ec2aa6
initialize to stdnormal when no params passed (and log). Change unit …
Jul 9, 2015
1bb44bd
style and doc changes. Factored out ks test into 2 separate tests
Jul 9, 2015
a48ae7b
refactor code to account for serializable RealDistribution. Reuse tes…
Jul 9, 2015
1f56371
changed ksTest in public API to kolmogorovSmirnovTest for clarity
Jul 9, 2015
0d0c201
kstTest -> kolmogorovSmirnovTest in statistics.md
Jul 9, 2015
bbb30b1
renamed KSTestResult to KolmogorovSmirnovTestResult, to stay consiste…
Jul 11, 2015
08834f4
Merge remote-tracking branch 'upstream/master'
Jul 13, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
renamed ks1 sample test for clarity
  • Loading branch information
jose.cambronero committed Jun 26, 2015
commit 3f81ad25505b38a3d929314842078ac9730a4142
2 changes: 1 addition & 1 deletion mllib/src/test/scala/org/apache/spark/mllib/stat/HypothesisTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ class HypothesisTestSuite extends SparkFunSuite with MLlibTestSparkContext {
}
}

test("kolmogorov smirnov test empirical distributions") {
test("1 sample kolmogorov smirnov test") {
Copy link
Contributor

@mengxr mengxr Jul 1, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kolmogorov smirnov -> Kolmogorov-Smirnov

Could you also add a unit test to compare with R? Here is one example:

https://github.com/apache/spark/blob/master/mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala#L214

Copy link
Author

@josepablocam josepablocam Jul 7, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added a comparison using R's ks.test (from stats library)

// Create theoretical distributions
val stdNormalDist = new NormalDistribution(0, 1)
val expDist = new ExponentialDistribution(0.6)
Expand Down