Skip to content

[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the filesystem explicitly specified by the user #21734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 7 commits into from
Closed

[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the filesystem explicitly specified by the user #21734

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8885fff
Add spark.yarn.access.all.hadoopFileSystems
wangyum Jul 9, 2018
50ef3c9
accessAllFileSystem -> !accessAllFileSystem
wangyum Jul 9, 2018
da1e389
FILESYSTEMS_TO_ACCESS_ALL -> true
wangyum Jul 9, 2018
43ffeaa
NAMENODES_TO_ACCESS default to *
wangyum Jul 10, 2018
61b1339
isRequestAllDelegationTokens = filesystemsToAccess.isEmpty
wangyum Jul 10, 2018
89b9d09
Merge remote-tracking branch 'upstream/master' into SPARK-24149
wangyum Aug 25, 2018
f3aa2a4
isRequestAllDelegationTokens -> requestAllDelegationTokens
wangyum Aug 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 5 additions & 9 deletions resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,8 @@ import org.apache.hadoop.yarn.api.ApplicationConstants
import org.apache.hadoop.yarn.api.records.{ApplicationAccessType, ContainerId, Priority}
import org.apache.hadoop.yarn.util.ConverterUtils

import org.apache.spark.{SecurityManager, SparkConf, SparkException}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.{SecurityManager, SparkConf}
import org.apache.spark.deploy.yarn.config._
import org.apache.spark.deploy.yarn.security.YARNHadoopDelegationTokenManager
import org.apache.spark.internal.config._
import org.apache.spark.launcher.YarnCommandBuilderUtils
import org.apache.spark.util.Utils

Expand Down Expand Up @@ -193,8 +190,7 @@ object YarnSparkHadoopUtil {
sparkConf: SparkConf,
hadoopConf: Configuration): Set[FileSystem] = {
val filesystemsToAccess = sparkConf.get(FILESYSTEMS_TO_ACCESS)
.map(new Path(_).getFileSystem(hadoopConf))
.toSet
val requestAllDelegationTokens = filesystemsToAccess.isEmpty

val stagingFS = sparkConf.get(STAGING_DIR)
.map(new Path(_).getFileSystem(hadoopConf))
Expand All @@ -203,8 +199,8 @@ object YarnSparkHadoopUtil {
// Add the list of available namenodes for all namespaces in HDFS federation.
// If ViewFS is enabled, this is skipped as ViewFS already handles delegation tokens for its
// namespaces.
val hadoopFilesystems = if (stagingFS.getScheme == "viewfs") {
Set.empty
val hadoopFilesystems = if (!requestAllDelegationTokens || stagingFS.getScheme == "viewfs") {
filesystemsToAccess.map(new Path(_).getFileSystem(hadoopConf)).toSet
} else {
val nameservices = hadoopConf.getTrimmedStrings("dfs.nameservices")
// Retrieving the filesystem for the nameservices where HA is not enabled
Expand All @@ -222,7 +218,7 @@ object YarnSparkHadoopUtil {
(filesystemsWithoutHA ++ filesystemsWithHA).toSet
}

filesystemsToAccess ++ hadoopFilesystems + stagingFS
hadoopFilesystems + stagingFS
}

}