Skip to content

[SPARK-13444] [MLlib] QuantileDiscretizer chooses bad splits on large DataFrames #11319

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
oliverpierson wants to merge 6 commits into from
Closed

[SPARK-13444] [MLlib] QuantileDiscretizer chooses bad splits on large DataFrames #11319

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
635fb4e
fixed splits bug in QuantileDiscretizer
Feb 23, 2016
d5dbaa2
explicitly cast requiredSamples to Double
Feb 23, 2016
4892fb7
added minSamplesRequired parameter to QuantileDiscretizer
Feb 23, 2016
3b55b60
test for QuantileDiscretizer on large datasets
Feb 23, 2016
c0052e4
private-tize minSamplesRequired; updated comments
Feb 24, 2016
abea876
change QuantileDiscretizer test name to better reflect purpose
Feb 24, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion mllib/src/main/scala/org/apache/spark/ml/feature/QuantileDiscretizer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ object QuantileDiscretizer extends DefaultParamsReadable[QuantileDiscretizer] wi
val totalSamples = dataset.count()
require(totalSamples > 0,
"QuantileDiscretizer requires non-empty input dataset but was given an empty input.")
val requiredSamples = math.max(numBins * numBins, 10000)
val requiredSamples = math.max(numBins * numBins, 10000.0)
Copy link
Member

@srowen srowen Feb 23, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd very slightly prefer to cast this to a double in the following line. This makes requiredSamples a double itself, which is slightly wrong.

val fraction = math.min(requiredSamples / dataset.count(), 1.0)
dataset.sample(withReplacement = false, fraction, new XORShiftRandom(seed).nextInt()).collect()
}
Expand Down