Skip to content

[SPARK-13444] [MLlib] QuantileDiscretizer chooses bad splits on large DataFrames #11377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oliverpierson wants to merge 7 commits into from
Closed

[SPARK-13444] [MLlib] QuantileDiscretizer chooses bad splits on large DataFrames #11377

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
635fb4e
fixed splits bug in QuantileDiscretizer
Feb 23, 2016
d5dbaa2
explicitly cast requiredSamples to Double
Feb 23, 2016
4892fb7
added minSamplesRequired parameter to QuantileDiscretizer
Feb 23, 2016
3b55b60
test for QuantileDiscretizer on large datasets
Feb 23, 2016
c0052e4
private-tize minSamplesRequired; updated comments
Feb 24, 2016
abea876
change QuantileDiscretizer test name to better reflect purpose
Feb 24, 2016
4356da2
remove setSeed in test suite causing build fail
Feb 25, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
change QuantileDiscretizer test name to better reflect purpose
  • Loading branch information
Oliver Pierson committed Feb 24, 2016
commit abea8765f66d061fca1d5358660fb71e9a194cc2
2 changes: 1 addition & 1 deletion mllib/src/test/scala/org/apache/spark/ml/feature/QuantileDiscretizerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ class QuantileDiscretizerSuite
}
}

test("Test splits on relatively large dataset") {
test("Test splits on dataset larger than minSamplesRequired") {
val sqlCtx = SQLContext.getOrCreate(sc)
import sqlCtx.implicits._

Expand Down