[SPARK-3444] Provide an easy way to change log level

[holdenk]

Add support for changing the log level at run time through the SparkContext. Based on an earlier PR, #2433 includes CR feedback from @pwendel & @davies

[SparkQA]

Test build #31340 has finished for PR 5791 at commit d9d03f3.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class Word2Vec extends Estimator[Word2VecModel] with Word2VecBase
  • trait HasStepSize extends Params
• This patch does not change any dependencies.

[SparkQA]

Test build #31352 has finished for PR 5791 at commit fac14a0.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • final class Word2Vec extends Estimator[Word2VecModel] with Word2VecBase
  • trait HasStepSize extends Params
• This patch does not change any dependencies.

[pwendell]

Hey Holden - can you have it be that if we see an invalid level we throw an exception? Otherwise it could be confusing if a user simply has a typo or something. Something like:

val validLevels = Seq("ALL", "DEBUG", "ERROR", "FATAL", "INFO", "OFF", "TRACE", "WARN")
if (!validLevels.contains(logLevel.toUpperCase) {
  throw new InvalidArgumentException(
    s"Supplied level $logLevel did not match one of: ${validLevels.mkString(',')}")
}

[holdenk]

Do you want this instead of the default to debug behavior that is currently present?

[pwendell]

Yes - I'm suggesting that before you pass anything to log4j we make sure it's one of the valid logging levels. Therefore, the default behavior of log4j's toLevel will be irrelevant.

[pwendell]

Looks great, but left one small comment.

[SparkQA]

Test build #31596 has finished for PR 5791 at commit 338d7bf.

• This patch fails MiMa tests.
• This patch merges cleanly.
• This patch adds no public classes.

[pwendell]

Jenkins, retest this please.

[SparkQA]

Test build #31606 has finished for PR 5791 at commit 338d7bf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #31628 has finished for PR 5791 at commit 3bf3be9.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.