Skip to content

[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the filesystem explicitly specified by the user #21734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 7 commits into from
Closed

[SPARK-24149][YARN][FOLLOW-UP] Only get the delegation tokens of the filesystem explicitly specified by the user #21734

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
8885fff
Add spark.yarn.access.all.hadoopFileSystems
wangyum Jul 9, 2018
50ef3c9
accessAllFileSystem -> !accessAllFileSystem
wangyum Jul 9, 2018
da1e389
FILESYSTEMS_TO_ACCESS_ALL -> true
wangyum Jul 9, 2018
43ffeaa
NAMENODES_TO_ACCESS default to *
wangyum Jul 10, 2018
61b1339
isRequestAllDelegationTokens = filesystemsToAccess.isEmpty
wangyum Jul 10, 2018
89b9d09
Merge remote-tracking branch 'upstream/master' into SPARK-24149
wangyum Aug 25, 2018
f3aa2a4
isRequestAllDelegationTokens -> requestAllDelegationTokens
wangyum Aug 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
FILESYSTEMS_TO_ACCESS_ALL -> true
  • Loading branch information
@wangyum
wangyum committed Jul 9, 2018
commit da1e389a75980234d6e11a2ea7f6b6ad4cab7062
3 changes: 2 additions & 1 deletion ...-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/YarnSparkHadoopUtilSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import org.scalatest.Matchers

import org.apache.spark.{SecurityManager, SparkConf, SparkFunSuite}
import org.apache.spark.deploy.SparkHadoopUtil
import org.apache.spark.deploy.yarn.config.FILESYSTEMS_TO_ACCESS_ALL
import org.apache.spark.internal.Logging
import org.apache.spark.util.{ResetSystemProperties, Utils}

Expand Down Expand Up @@ -143,7 +144,7 @@ class YarnSparkHadoopUtilSuite extends SparkFunSuite with Matchers with Logging
}

test("SPARK-24149: retrieve all namenodes from HDFS") {
val sparkConf = new SparkConf()
val sparkConf = new SparkConf().set(FILESYSTEMS_TO_ACCESS_ALL, true)
val basicFederationConf = new Configuration()
basicFederationConf.set("fs.defaultFS", "hdfs://localhost:8020")
basicFederationConf.set("dfs.nameservices", "ns1,ns2")
Expand Down