Skip to content

[SPARK-13257] [Improvement] Scala naive Bayes example: check model after loading #11192

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
movelikeriver wants to merge 2 commits into from
Closed

[SPARK-13257] [Improvement] Scala naive Bayes example: check model after loading #11192

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8a93c66
Scala naive Bayes example: check model after loading
movelikeriver Feb 13, 2016
1da83ac
update
movelikeriver Feb 13, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions examples/src/main/scala/org/apache/spark/examples/mllib/NaiveBayesExample.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,16 @@
* limitations under the License.
*/

/*
* Usage:
Copy link
Contributor

@holdenk holdenk Apr 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this usage text probably isn't the format we want, its pretty specific to each Spark release. Maybe look at some of the other files in this directory and see how they do their usage?

*
* sbt package
*
* spark-submit --class "org.apache.spark.examples.mllib.NaiveBayesExample"
* --master local[4]
* examples/target/scala-2.11/spark-examples_2.11-2.0.0-SNAPSHOT.jar
*/

// scalastyle:off println
package org.apache.spark.examples.mllib

Expand All @@ -23,6 +33,9 @@ import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.mllib.classification.{NaiveBayes, NaiveBayesModel}
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint

import java.io.File;
Copy link
Contributor

@holdenk holdenk Apr 28, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These imports don't seem to be ordered in the way we would want them to be. You should check out https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark and https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide

import org.apache.commons.io.FileUtils;
// $example off$

object NaiveBayesExample {
Expand All @@ -46,10 +59,18 @@ object NaiveBayesExample {

val predictionAndLabel = test.map(p => (model.predict(p.features), p.label))
val accuracy = 1.0 * predictionAndLabel.filter(x => x._1 == x._2).count() / test.count()
println("model accuracy %f".format(accuracy))

// Save and load model
model.save(sc, "target/tmp/myNaiveBayesModel")
val sameModel = NaiveBayesModel.load(sc, "target/tmp/myNaiveBayesModel")
val outputDir = "target/tmp/myNaiveBayesModel"
FileUtils.forceDelete(new File(outputDir))
model.save(sc, outputDir)
val sameModel = NaiveBayesModel.load(sc, outputDir)

val samePredictionAndLabel = test.map(p => (sameModel.predict(p.features), p.label))
val sameAccuracy = 1.0 * samePredictionAndLabel.filter(x => x._1 == x._2).count() /
test.count()
println("sameModel accuracy %f".format(sameAccuracy))
// $example off$
}
}
Expand Down