Skip to content

[SPARK-21087] [ML] CrossValidator, TrainValidationSplit expose sub models after fitting: Scala #19208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
WeichenXu123 wants to merge 11 commits into from
Closed

[SPARK-21087] [ML] CrossValidator, TrainValidationSplit expose sub models after fitting: Scala #19208

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
46d3ab3
init pr
WeichenXu123 Sep 11, 2017
ae13440
fix style
WeichenXu123 Sep 12, 2017
e0f4ce6
remove code for dump models to disk
WeichenXu123 Sep 14, 2017
a33c4ea
merge master & resolve conflicts
WeichenXu123 Sep 27, 2017
e009ee1
address comment issues
WeichenXu123 Sep 27, 2017
931fa6c
address issues from comments
WeichenXu123 Nov 3, 2017
2a83fb5
fix style
WeichenXu123 Nov 3, 2017
f2ef609
address comments from joseph
WeichenXu123 Nov 6, 2017
7bacfca
address comments from joseph
WeichenXu123 Nov 7, 2017
654e4d5
fix mima
WeichenXu123 Nov 7, 2017
7e997da
address minor issues
WeichenXu123 Nov 14, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
address minor issues
  • Loading branch information
@WeichenXu123
WeichenXu123 committed Nov 14, 2017
commit 7e997da44157a9807de9c8fe8e7d2e5b66b6bfb1
5 changes: 3 additions & 2 deletions mllib/src/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,9 +399,10 @@ object CrossValidatorModel extends MLReadable[CrossValidatorModel] {
val bestModelPath = new Path(path, "bestModel").toString
val bestModel = DefaultParamsReader.loadParamsInstance[Model[_]](bestModelPath, sc)
val avgMetrics = (metadata.metadata \ "avgMetrics").extract[Seq[Double]].toArray
val shouldPersistSubModels = (metadata.metadata \ "persistSubModels").extract[Boolean]
val persistSubModels = (metadata.metadata \ "persistSubModels")
.extractOrElse[Boolean](false)

val subModels: Option[Array[Array[Model[_]]]] = if (shouldPersistSubModels) {
val subModels: Option[Array[Array[Model[_]]]] = if (persistSubModels) {
val subModelsPath = new Path(path, "subModels")
val _subModels = Array.fill(numFolds)(Array.fill[Model[_]](
estimatorParamMaps.length)(null))
Expand Down
14 changes: 8 additions & 6 deletions mllib/src/main/scala/org/apache/spark/ml/tuning/TrainValidationSplit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,9 +202,7 @@ class TrainValidationSplit @Since("1.5.0") (@Since("1.5.0") override val uid: St
}

@Since("2.0.0")
override def write: TrainValidationSplit.TrainValidationSplitWriter = {
new TrainValidationSplit.TrainValidationSplitWriter(this)
}
override def write: MLWriter = new TrainValidationSplit.TrainValidationSplitWriter(this)
}

@Since("2.0.0")
Expand Down Expand Up @@ -311,7 +309,9 @@ class TrainValidationSplitModel private[ml] (
}

@Since("2.0.0")
override def write: MLWriter = new TrainValidationSplitModel.TrainValidationSplitModelWriter(this)
override def write: TrainValidationSplitModel.TrainValidationSplitModelWriter = {
new TrainValidationSplitModel.TrainValidationSplitModelWriter(this)
}
}

@Since("2.0.0")
Expand Down Expand Up @@ -339,6 +339,7 @@ object TrainValidationSplitModel extends MLReadable[TrainValidationSplitModel] {
* If subModels are not available, then setting "persistSubModels" to "true" will cause
* an exception.
*/
@Since("2.3.0")
final class TrainValidationSplitModelWriter private[tuning] (
Copy link
Member

@jkbradley jkbradley Nov 13, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since annotation

instance: TrainValidationSplitModel) extends MLWriter {

Expand Down Expand Up @@ -385,9 +386,10 @@ object TrainValidationSplitModel extends MLReadable[TrainValidationSplitModel] {
val bestModelPath = new Path(path, "bestModel").toString
val bestModel = DefaultParamsReader.loadParamsInstance[Model[_]](bestModelPath, sc)
val validationMetrics = (metadata.metadata \ "validationMetrics").extract[Seq[Double]].toArray
val shouldPersistSubModels = (metadata.metadata \ "persistSubModels").extract[Boolean]
val persistSubModels = (metadata.metadata \ "persistSubModels")
.extractOrElse[Boolean](false)

val subModels: Option[Array[Model[_]]] = if (shouldPersistSubModels) {
val subModels: Option[Array[Model[_]]] = if (persistSubModels) {
val subModelsPath = new Path(path, "subModels")
val _subModels = Array.fill[Model[_]](estimatorParamMaps.length)(null)
for (paramIndex <- 0 until estimatorParamMaps.length) {
Expand Down
1 change: 0 additions & 1 deletion mllib/src/test/scala/org/apache/spark/ml/tuning/CrossValidatorSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,6 @@ class CrossValidatorSuite
val eval = new BinaryClassificationEvaluator
val numFolds = 3
val subPath = new File(tempDir, "testCrossValidatorSubModels")
val persistSubModelsPath = new File(subPath, "subModels").toString

val cv = new CrossValidator()
.setEstimator(lr)
Expand Down