Skip to content

[SPARK-23850][sql] Add separate config for SQL options redaction. #21158

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vanzin wants to merge 2 commits into from
Closed

[SPARK-23850][sql] Add separate config for SQL options redaction. #21158

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
deb299b
[SPARK-23850][sql] Add separate config for SQL options redaction.
Apr 24, 2018
6ca25e3
Use SQLConf.get.
Apr 30, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Use SQLConf.get.
  • Loading branch information
Marcelo Vanzin committed Apr 30, 2018
commit 6ca25e3db8966c47f58d206e3f6e7bdce7786478
10 changes: 2 additions & 8 deletions ...src/main/scala/org/apache/spark/sql/execution/datasources/SaveIntoDataSourceCommand.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,12 @@

package org.apache.spark.sql.execution.datasources

import org.apache.spark.SparkEnv
import org.apache.spark.sql.{Dataset, Row, SaveMode, SparkSession}
import org.apache.spark.sql.catalyst.plans.QueryPlan
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.execution.command.RunnableCommand
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.sources.CreatableRelationProvider
import org.apache.spark.util.Utils

/**
* Saves the results of `query` in to a data source.
Expand All @@ -50,12 +49,7 @@ case class SaveIntoDataSourceCommand(
}

override def simpleString: String = {
// Redact the options according to the current session's config. If for some reason there
// is no current session, be paranoid and don't show any options.
val redacted = SparkSession.getActiveSession.map(_.sessionState.conf).map { conf =>
conf.redactOptions(options)
}.getOrElse(Map())

val redacted = SQLConf.get.redactOptions(options)
s"SaveIntoDataSourceCommand ${dataSource}, ${redacted}, ${mode}"
}
}