Skip to content

[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the filesystem explicitly specified by the user #21734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 7 commits into from
Closed

[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the filesystem explicitly specified by the user #21734

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8885fff
Add spark.yarn.access.all.hadoopFileSystems
wangyum Jul 9, 2018
50ef3c9
accessAllFileSystem -> !accessAllFileSystem
wangyum Jul 9, 2018
da1e389
FILESYSTEMS_TO_ACCESS_ALL -> true
wangyum Jul 9, 2018
43ffeaa
NAMENODES_TO_ACCESS default to *
wangyum Jul 10, 2018
61b1339
isRequestAllDelegationTokens = filesystemsToAccess.isEmpty
wangyum Jul 10, 2018
89b9d09
Merge remote-tracking branch 'upstream/master' into SPARK-24149
wangyum Aug 25, 2018
f3aa2a4
isRequestAllDelegationTokens -> requestAllDelegationTokens
wangyum Aug 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add spark.yarn.access.all.hadoopFileSystems
  • Loading branch information
@wangyum
wangyum committed Jul 9, 2018
commit 8885ffff03efe1bbc2afa86b24f15c0413d2c05d
4 changes: 3 additions & 1 deletion resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,10 +200,12 @@ object YarnSparkHadoopUtil {
.map(new Path(_).getFileSystem(hadoopConf))
.getOrElse(FileSystem.get(hadoopConf))

val accessAllFileSystem = sparkConf.get(FILESYSTEMS_TO_ACCESS_ALL)

// Add the list of available namenodes for all namespaces in HDFS federation.
// If ViewFS is enabled, this is skipped as ViewFS already handles delegation tokens for its
// namespaces.
val hadoopFilesystems = if (stagingFS.getScheme == "viewfs") {
val hadoopFilesystems = if (accessAllFileSystem || stagingFS.getScheme == "viewfs") {
Set.empty
} else {
val nameservices = hadoopConf.getTrimmedStrings("dfs.nameservices")
Expand Down
8 changes: 8 additions & 0 deletions resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/config.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,14 @@ package object config {
"that hosts fs.defaultFS does not need to be listed here.")
.fallbackConf(NAMENODES_TO_ACCESS)

private[spark] val FILESYSTEMS_TO_ACCESS_ALL =
ConfigBuilder("spark.yarn.access.all.hadoopFileSystems")
.doc("Whether to get tokens of all filesystem configured in hdfs-site.xml. " +
"The default value is false. " +
"If true, a filesystem failed to get token may cause the entire job failed.")
.booleanConf
.createWithDefault(false)

/* Rolled log aggregation configuration. */

private[spark] val ROLLED_LOG_INCLUDE_PATTERN =
Expand Down