Skip to content

SPARK-1310: Start adding k-fold cross validation to MLLib [adds kFold to MLUtils & fixes bug in BernoulliSampler] #18

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
holdenk wants to merge 22 commits into from
Closed

SPARK-1310: Start adding k-fold cross validation to MLLib [adds kFold to MLUtils & fixes bug in BernoulliSampler] #18

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a751ec6
Add k-fold cross validation to MLLib
holdenk Feb 5, 2014
08f8e4d
Fix BernoulliSampler to respect complement
holdenk Feb 11, 2014
c0b7fa4
Switch FoldedRDD to use BernoulliSampler and PartitionwiseSampledRDD
holdenk Feb 11, 2014
dd0b737
Wrap long lines (oops)
holdenk Feb 11, 2014
264502a
Add a test for the bug that was found with BernoulliSampler not copyi…
holdenk Feb 11, 2014
91eae64
Consolidate things in mlutils
holdenk Feb 12, 2014
b78804e
Remove cross validation [TODO in another pull request]
holdenk Feb 12, 2014
e8741a7
CR feedback
holdenk Feb 17, 2014
5a33f1d
Code review follow up.
holdenk Feb 24, 2014
163c5b1
code review feedback 1.to -> 1 to and folds -> numFolds
holdenk Mar 4, 2014
bb5fa56
extra line sadness
holdenk Apr 9, 2014
7ebe4d5
CR feedback, remove unecessary learners (came back during merge mista…
holdenk Apr 9, 2014
c5b723f
clean up
holdenk Apr 9, 2014
e187e35
Move { up to same line as whenExecuting(random) in RandomSamplerSuite…
holdenk Apr 9, 2014
c702a96
Fix imports in MLUtils
holdenk Apr 9, 2014
2cb90b3
Fix the names in kFold
holdenk Apr 9, 2014
150889c
Fix up error messages in the MLUtilsSuite
holdenk Apr 9, 2014
90896c7
New line
holdenk Apr 9, 2014
7157ae9
CR feedback
holdenk Apr 10, 2014
6ddbf05
swap training and validation order
holdenk Apr 10, 2014
e84f2fc
Fix the test, we should be looking at the second element instead
holdenk Apr 10, 2014
208db9b
Fix a bad space
holdenk Apr 15, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Code review follow up.
  • Loading branch information
@holdenk
holdenk committed Apr 9, 2014
commit 5a33f1d6d91e1ec1afb62c920abc0443f13725cf
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/util/random/RandomSampler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ class BernoulliSampler[T](lb: Double, ub: Double, complement: Boolean = false)
}
}

def cloneComplement() = new BernoulliSampler[T](lb, ub, !complement)
Copy link
Contributor

@mengxr mengxr Apr 10, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mark it private[spark] or provide doc? Also, please add return type (my bad).


override def clone = new BernoulliSampler[T](lb, ub, complement)
Copy link
Contributor

@mengxr mengxr Apr 10, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add return type.

}

Expand Down
12 changes: 6 additions & 6 deletions mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,12 +187,12 @@ object MLUtils {
*/
def kFold[T : ClassTag](rdd: RDD[T], folds: Int, seed: Int): List[Pair[RDD[T], RDD[T]]] = {
val foldsF = folds.toFloat
1.to(folds).map(fold => ((
new PartitionwiseSampledRDD(rdd, new BernoulliSampler[T]((fold-1)/foldsF,fold/foldsF,
complement = false), seed),
new PartitionwiseSampledRDD(rdd, new BernoulliSampler[T]((fold-1)/foldsF,fold/foldsF,
complement = true), seed)
))).toList
1.to(folds).map { fold =>
val sampler = new BernoulliSampler[T]((fold-1)/foldsF,fold/foldsF, complement = false)
val train = new PartitionwiseSampledRDD(rdd, sampler, seed)
val test = new PartitionwiseSampledRDD(rdd, sampler.cloneComplement(), seed)
(train, test)
}.toList
}

/**
Expand Down