Skip to content

[SPARK-9005][MLlib]Fix RegressionMetrics computation of explainedVariance #7361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
feynmanliang wants to merge 7 commits into from
Closed

[SPARK-9005][MLlib]Fix RegressionMetrics computation of explainedVariance #7361

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4c4e56f
Fix RegressionMetrics computation of explainedVariance and r2
Jul 13, 2015
c235de0
Fix RegressionMetrics tests
Jul 13, 2015
bde9761
Fix RegressionMetrics tests, relax assumption predictor is unbiased
Jul 13, 2015
db8605a
Style fix
Jul 13, 2015
08a0e1b
Fix pyspark tests
Jul 13, 2015
1a3d098
SROwen code review comments
Jul 13, 2015
f1112fc
Add explainedVariance formula
Jul 14, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix RegressionMetrics tests
  • Loading branch information
Feynman Liang committed Jul 13, 2015
commit c235de066583f2f0980de040b2aa879b5072a05a
6 changes: 3 additions & 3 deletions mllib/src/test/scala/org/apache/spark/mllib/evaluation/RegressionMetricsSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,20 +27,20 @@ class RegressionMetricsSuite extends SparkFunSuite with MLlibTestSparkContext {
val predictionAndObservations = sc.parallelize(
Seq((2.5, 3.0), (0.0, -0.5), (2.0, 2.0), (8.0, 7.0)), 2)
val metrics = new RegressionMetrics(predictionAndObservations)
assert(metrics.explainedVariance ~== 0.95717 absTol 1E-5,
assert(metrics.explainedVariance ~== 2.05729 absTol 1E-5,
"explained variance regression score mismatch")
assert(metrics.meanAbsoluteError ~== 0.5 absTol 1E-5, "mean absolute error mismatch")
assert(metrics.meanSquaredError ~== 0.375 absTol 1E-5, "mean squared error mismatch")
assert(metrics.rootMeanSquaredError ~== 0.61237 absTol 1E-5,
"root mean squared error mismatch")
assert(metrics.r2 ~== 0.94861 absTol 1E-5, "r2 score mismatch")
assert(metrics.r2 ~== 0.84582 absTol 1E-5, "r2 score mismatch")
}

test("regression metrics with complete fitting") {
val predictionAndObservations = sc.parallelize(
Seq((3.0, 3.0), (0.0, 0.0), (2.0, 2.0), (8.0, 8.0)), 2)
val metrics = new RegressionMetrics(predictionAndObservations)
assert(metrics.explainedVariance ~== 1.0 absTol 1E-5,
assert(metrics.explainedVariance ~== 2.89583 absTol 1E-5,
"explained variance regression score mismatch")
assert(metrics.meanAbsoluteError ~== 0.0 absTol 1E-5, "mean absolute error mismatch")
assert(metrics.meanSquaredError ~== 0.0 absTol 1E-5, "mean squared error mismatch")
Expand Down