Skip to content

[SPARK-5158] Access kerberized HDFS from Spark standalone #17530

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
themodernlife wants to merge 18 commits into from
Closed

[SPARK-5158] Access kerberized HDFS from Spark standalone #17530

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
62a6e20
WIP
Feb 24, 2017
accfe0c
Add license header that somehow got removed
Feb 24, 2017
b8559b5
Fixup tests
Feb 24, 2017
539cc6c
WIP
Feb 26, 2017
3f76281
Push delegation token out to ExecutorRunner
Feb 27, 2017
25e7639
More wip... probably borked
Feb 28, 2017
847f604
Untested... make cluster mode work with standalone
Mar 2, 2017
4689a55
Hadoop FileInputFormat is hardcoded to request delegation tokens with…
Mar 7, 2017
3e85aa5
Still need to sort out a few things, but overall much smaller patch-set
Mar 7, 2017
f743e6b
WIP
Mar 8, 2017
31c91dc
WIP
Mar 8, 2017
1964419
Still something isn't working
Mar 8, 2017
b5bacf3
Merge master
Mar 15, 2017
83f0501
Actually use credential updater
Mar 15, 2017
917b077
Change order of configuration setting so that everything works
Apr 4, 2017
a4c22a9
Merge branch 'master' into spark-5158
Apr 4, 2017
16f9551
Remove inadvertent file
Apr 4, 2017
246c76a
Cleanup code
Apr 4, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fixup tests
  • Loading branch information
Ian Hummel committed Feb 24, 2017
commit b8559b5895c81c871b1db00b75f038082b2dd4fb
6 changes: 2 additions & 4 deletions .../ConfigurableCredentialManagerSuite.scala → .../ConfigurableCredentialManagerSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,14 @@
* limitations under the License.
*/

package org.apache.spark.deploy.yarn.security
package org.apache.spark.deploy.security

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.io.Text
import org.apache.hadoop.security.Credentials
import org.apache.hadoop.security.token.Token
import org.apache.spark.deploy.security.{ConfigurableCredentialManager, HBaseCredentialProvider, HiveCredentialProvider, ServiceCredentialProvider}
import org.scalatest.{BeforeAndAfter, Matchers}
import org.apache.spark.{SparkConf, SparkFunSuite}
import org.apache.spark.deploy.yarn.config._
import org.scalatest.{BeforeAndAfter, Matchers}

class ConfigurableCredentialManagerSuite extends SparkFunSuite with Matchers with BeforeAndAfter {
private var credentialManager: ConfigurableCredentialManager = null
Expand Down
7 changes: 3 additions & 4 deletions ...ity/HadoopFSCredentialProviderSuite.scala → ...ity/HadoopFSCredentialProviderSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,13 @@
* limitations under the License.
*/

package org.apache.spark.deploy.yarn.security
package org.apache.spark.deploy.security

import org.apache.hadoop.conf.Configuration
import org.apache.spark.deploy.security.HadoopFSCredentialProvider
import org.scalatest.{Matchers, PrivateMethodTester}
import org.apache.spark.{SparkException, SparkFunSuite}
import org.scalatest.{Matchers, PrivateMethodTester}

class HadoopFSCredentialProviderSuite
class HadoopFSCredentialProviderSuite
extends SparkFunSuite
with PrivateMethodTester
with Matchers {
Expand Down
2 changes: 1 addition & 1 deletion ...st/resources/META-INF/services/org.apache.spark.deploy.security.ServiceCredentialProvider
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
org.apache.spark.deploy.yarn.security.TestCredentialProvider
org.apache.spark.deploy.security.TestCredentialProvider