[SPARK-29247][SQL] Redact sensitive information in when construct HiveClientHive.state #25954
Conversation
| s""" | ||
| |Applying Hadoop/Hive/Spark and extra properties to Hive Conf: | ||
| |$k=${if (k.toLowerCase(Locale.ROOT).contains("password")) "xxx" else v} | ||
| |$k=${SQLConf.get.redactOptions(k.toLowerCase(Locale.ROOT) -> v)._2} |
There was a problem hiding this comment.
Can't we just use Map here instead of adding the copies of redact for (K, V)?
There was a problem hiding this comment.
Can't we just use Map here instead of adding the copies of
redactfor (K, V)?
In that way we need to get conf map , put all of then to hiveConf then call SQLConf.get.redactOptions, and print out log.
If we don't want to overloading a new method, it's ok.
There was a problem hiding this comment.
Yeah I don't think it's worth adding two more methods for this.
There was a problem hiding this comment.
Change back use origin method.
|
cc @wangyum |
|
ok to test |
|
Test build #111538 has finished for PR 25954 at commit
|
|
Test build #111549 has finished for PR 25954 at commit
|
There was a problem hiding this comment.
+1, LGTM. Merged to master/2.4. Thank you, @AngersZhuuuu , @HyukjinKwon , @srowen .
…eClientHive.state
### What changes were proposed in this pull request?
HiveClientImpl may be log sensitive information. e.g. url, secret and token:
```scala
logDebug(
s"""
|Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
|$k=${if (k.toLowerCase(Locale.ROOT).contains("password")) "xxx" else v}
""".stripMargin)
```
So redact it. Use SQLConf.get.redactOptions.
I add a new overloading function to fit this situation for one by one kv pair situation.
### Why are the changes needed?
Redact sensitive information when construct HiveClientImpl
### Does this PR introduce any user-facing change?
No
### How was this patch tested?
MT
Run command
` /sbin/start-thriftserver.sh`
In log we can get
```
19/09/28 08:27:02 main DEBUG HiveClientImpl:
Applying Hadoop/Hive/Spark and extra properties to Hive Conf:
hive.druid.metadata.password=*********(redacted)
```
Closes #25954 from AngersZhuuuu/SPARK-29247.
Authored-by: angerszhu <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 1d4b2f0)
Signed-off-by: Dongjoon Hyun <[email protected]>
5 participants
What changes were proposed in this pull request?
HiveClientImpl may be log sensitive information. e.g. url, secret and token:
logDebug( s""" |Applying Hadoop/Hive/Spark and extra properties to Hive Conf: |$k=${if (k.toLowerCase(Locale.ROOT).contains("password")) "xxx" else v} """.stripMargin)So redact it. Use SQLConf.get.redactOptions.
I add a new overloading function to fit this situation for one by one kv pair situation.
Why are the changes needed?
Redact sensitive information when construct HiveClientImpl
Does this PR introduce any user-facing change?
No
How was this patch tested?
MT
Run command
/sbin/start-thriftserver.shIn log we can get