Skip to content

Commit 6cb75db

Browse files
WeichenXu123srowen
authored andcommitted
[SPARK-16470][ML][OPTIMIZER] Check linear regression training whether actually reach convergence and add warning if not
## What changes were proposed in this pull request? In `ml.regression.LinearRegression`, it use breeze `LBFGS` and `OWLQN` optimizer to do data training, but do not check whether breeze's optimizer returned result actually reached convergence. The `LBFGS` and `OWLQN` optimizer in breeze finish iteration may result the following situations: 1) reach max iteration number 2) function reach value convergence 3) objective function stop improving 4) gradient reach convergence 5) search failed(due to some internal numerical error) I add warning printing code so that if the iteration result is (1) or (3) or (5) in above, it will print a warning with respective reason string. ## How was this patch tested? Manual. Author: WeichenXu <WeichenXu123@outlook.com> Closes #14122 from WeichenXu123/add_lr_not_convergence_warn.
1 parent 5b28e02 commit 6cb75db

File tree

1 file changed

+5
-0
lines changed

1 file changed

+5
-0
lines changed

mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -327,6 +327,11 @@ class LinearRegression @Since("1.3.0") (@Since("1.3.0") override val uid: String
327327
throw new SparkException(msg)
328328
}
329329

330+
if (!state.actuallyConverged) {
331+
logWarning("LinearRegression training fininshed but the result " +
332+
s"is not converged because: ${state.convergedReason.get.reason}")
333+
}
334+
330335
/*
331336
The coefficients are trained in the scaled space; we're converting them back to
332337
the original space.

0 commit comments

Comments
 (0)