Skip to content

[SPARK-52426][CORE] Support redirecting stdout/stderr to logging system #51130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pan3793 wants to merge 11 commits into from
Closed

[SPARK-52426][CORE] Support redirecting stdout/stderr to logging system #51130

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
b0c4474
[SPARK-52426][CORE] Support redirecting stdout/stderr to logging system
pan3793 Jun 9, 2025
6cf48f7
nit
pan3793 Jun 9, 2025
c66f4d3
address comments
pan3793 Jun 13, 2025
9461673
nit
pan3793 Jun 13, 2025
1593faa
extract common code
pan3793 Jun 13, 2025
110e7b1
document affect shell console progress bar
pan3793 Jun 13, 2025
ed76488
nit
pan3793 Jun 13, 2025
8ccae63
Ensure ConsoleProgressBar always print to console stderr
pan3793 Jun 13, 2025
28222ec
revert docs changes
pan3793 Jun 13, 2025
5a32078
nit
pan3793 Jun 13, 2025
a3326a6
address comments
pan3793 Jun 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
revert docs changes
  • Loading branch information
@pan3793
pan3793 committed Jun 13, 2025
commit 28222ec6b39bc901a837aaa5808569ff5fce4a2b
10 changes: 1 addition & 9 deletions core/src/main/scala/org/apache/spark/deploy/RedirectConsolePlugin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,8 @@ import java.util.{Collections, Map => JMap}

import org.apache.spark.SparkContext
import org.apache.spark.api.plugin.{DriverPlugin, ExecutorPlugin, PluginContext, SparkPlugin}
import org.apache.spark.internal.{Logging, MDC, SparkLoggerFactory}
import org.apache.spark.internal.LogKeys._
import org.apache.spark.internal.{Logging, SparkLoggerFactory}
import org.apache.spark.internal.config._
import org.apache.spark.internal.config.UI.UI_SHOW_CONSOLE_PROGRESS

/**
* A built-in plugin to allow redirecting stdout/stderr to logging system (SLF4J).
Expand All @@ -51,12 +49,6 @@ class DriverRedirectConsolePlugin extends DriverPlugin with Logging {
override def init(sc: SparkContext, ctx: PluginContext): JMap[String, String] = {
if (sc.conf.get(DRIVER_REDIRECT_CONSOLE_TO_LOG_ENABLED)) {
logInfo("Redirect driver's stdout/stderr to logging system")
if (sc.conf.get(UI_SHOW_CONSOLE_PROGRESS)) {
logWarning(log"Redirect driver's stderr to logging system may affect " +
log"console progress bar, consider disabling either " +
log"${MDC(CONFIG, DRIVER_REDIRECT_CONSOLE_TO_LOG_ENABLED.key)} or " +
log"${MDC(CONFIG2, UI_SHOW_CONSOLE_PROGRESS.key)}.")
}
RedirectConsolePlugin.redirectConsoleToLog()
}
Collections.emptyMap
Expand Down
3 changes: 1 addition & 2 deletions core/src/main/scala/org/apache/spark/internal/config/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2843,8 +2843,7 @@ package object config {
ConfigBuilder("spark.driver.log.redirectConsole.enabled")
Copy link
Member

@yaooqinn yaooqinn Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, we can still configure separately for stderr/stdout if we have a config intaking a subset of {stderr, stdout}

Copy link
Member Author

@pan3793 pan3793 Jun 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, refactored the config to accept a list of console output kinds
spark.driver.log.redirectConsoleOutputs

.doc("Whether to redirect the driver's stdout/stderr to logging system. " +
s"It only takes affect when `${PLUGINS.key}` is configured with " +
"`org.apache.spark.deploy.RedirectConsolePlugin`. Note, enabling this " +
"feature may affect the shell console progress bar.")
"`org.apache.spark.deploy.RedirectConsolePlugin`.")
.version("4.1.0")
.booleanConf
.createWithDefault(true)
Expand Down