Skip to content

[SPARK-13444] [MLlib] QuantileDiscretizer chooses bad splits on large DataFrames #11319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oliverpierson wants to merge 6 commits into from
Closed

[SPARK-13444] [MLlib] QuantileDiscretizer chooses bad splits on large DataFrames #11319

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
635fb4e
fixed splits bug in QuantileDiscretizer
Feb 23, 2016
d5dbaa2
explicitly cast requiredSamples to Double
Feb 23, 2016
4892fb7
added minSamplesRequired parameter to QuantileDiscretizer
Feb 23, 2016
3b55b60
test for QuantileDiscretizer on large datasets
Feb 23, 2016
c0052e4
private-tize minSamplesRequired; updated comments
Feb 24, 2016
abea876
change QuantileDiscretizer test name to better reflect purpose
Feb 24, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
explicitly cast requiredSamples to Double
  • Loading branch information
Oliver Pierson committed Feb 23, 2016
commit d5dbaa251f55f7763712d5fb45682a2125a86bb8
4 changes: 2 additions & 2 deletions mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,8 +110,8 @@ object QuantileDiscretizer extends DefaultParamsReadable[QuantileDiscretizer] wi
val totalSamples = dataset.count()
require(totalSamples > 0,
"QuantileDiscretizer requires non-empty input dataset but was given an empty input.")
val requiredSamples = math.max(numBins * numBins, 10000.0)
val fraction = math.min(requiredSamples / dataset.count(), 1.0)
val requiredSamples = math.max(numBins * numBins, 10000)
val fraction = math.min(requiredSamples.toDouble / dataset.count(), 1.0)
dataset.sample(withReplacement = false, fraction, new XORShiftRandom(seed).nextInt()).collect()
}

Expand Down