Skip to content

SPARK-1310: Start adding k-fold cross validation to MLLib [adds kFold to MLUtils & fixes bug in BernoulliSampler] #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 22 commits into from
Closed

SPARK-1310: Start adding k-fold cross validation to MLLib [adds kFold to MLUtils & fixes bug in BernoulliSampler] #18

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a751ec6
Add k-fold cross validation to MLLib
holdenk Feb 5, 2014
08f8e4d
Fix BernoulliSampler to respect complement
holdenk Feb 11, 2014
c0b7fa4
Switch FoldedRDD to use BernoulliSampler and PartitionwiseSampledRDD
holdenk Feb 11, 2014
dd0b737
Wrap long lines (oops)
holdenk Feb 11, 2014
264502a
Add a test for the bug that was found with BernoulliSampler not copyi…
holdenk Feb 11, 2014
91eae64
Consolidate things in mlutils
holdenk Feb 12, 2014
b78804e
Remove cross validation [TODO in another pull request]
holdenk Feb 12, 2014
e8741a7
CR feedback
holdenk Feb 17, 2014
5a33f1d
Code review follow up.
holdenk Feb 24, 2014
163c5b1
code review feedback 1.to -> 1 to and folds -> numFolds
holdenk Mar 4, 2014
bb5fa56
extra line sadness
holdenk Apr 9, 2014
7ebe4d5
CR feedback, remove unecessary learners (came back during merge mista…
holdenk Apr 9, 2014
c5b723f
clean up
holdenk Apr 9, 2014
e187e35
Move { up to same line as whenExecuting(random) in RandomSamplerSuite…
holdenk Apr 9, 2014
c702a96
Fix imports in MLUtils
holdenk Apr 9, 2014
2cb90b3
Fix the names in kFold
holdenk Apr 9, 2014
150889c
Fix up error messages in the MLUtilsSuite
holdenk Apr 9, 2014
90896c7
New line
holdenk Apr 9, 2014
7157ae9
CR feedback
holdenk Apr 10, 2014
6ddbf05
swap training and validation order
holdenk Apr 10, 2014
e84f2fc
Fix the test, we should be looking at the second element instead
holdenk Apr 10, 2014
208db9b
Fix a bad space
holdenk Apr 15, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Wrap long lines (oops)
  • Loading branch information
@holdenk
holdenk committed Apr 9, 2014
commit dd0b7373546d7571e8b0e0c85299bf94232cc461
6 changes: 4 additions & 2 deletions core/src/main/scala/org/apache/spark/rdd/FoldedRDD.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,8 @@ class FoldedRDD[T: ClassTag](
fold: Float,
folds: Float,
seed: Int)
extends PartitionwiseSampledRDD[T, T](prev, new BernoulliSampler((fold-1)/folds,fold/folds, false), seed) {
extends PartitionwiseSampledRDD[T, T](prev,
new BernoulliSampler((fold-1)/folds,fold/folds, false), seed) {
}

/**
Expand All @@ -48,5 +49,6 @@ class CompositeFoldedRDD[T: ClassTag](
fold: Float,
folds: Float,
seed: Int)
extends PartitionwiseSampledRDD[T, T](prev, new BernoulliSampler((fold-1)/folds, fold/folds, true), seed) {
extends PartitionwiseSampledRDD[T, T](prev,
new BernoulliSampler((fold-1)/folds, fold/folds, true), seed) {
}