Skip to content

[SPARK-20605][Core][Yarn][Mesos] Deprecate not used AM and executor port configuration #17866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jerryshao wants to merge 2 commits into from
Closed

[SPARK-20605][Core][Yarn][Mesos] Deprecate not used AM and executor port configuration #17866

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3c9120e
deprecate not used AM and executor port configuration
jerryshao May 5, 2017
ac710c7
Address the comments
jerryshao May 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Address the comments 
Change-Id: Ia1ae58b27edce1283b507026cdc4c0bd3b35817c
  • Loading branch information
@jerryshao
jerryshao committed May 8, 2017
commit ac710c7020a4fb8b3d684e0eb5daf47befb039ec
4 changes: 2 additions & 2 deletions core/src/main/scala/org/apache/spark/SparkConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -580,8 +580,8 @@ private[spark] object SparkConf extends Logging {
DeprecatedConfig("spark.rpc", "2.0", "Not used any more."),
DeprecatedConfig("spark.scheduler.executorTaskBlacklistTime", "2.1.0",
"Please use the new blacklisting options, spark.blacklist.*"),
DeprecatedConfig("spark.yarn.am.port", "2.2.1", "Not used any more"),
DeprecatedConfig("spark.executor.port", "2.2.1", "Not used any more")
DeprecatedConfig("spark.yarn.am.port", "2.0.0", "Not used any more"),
DeprecatedConfig("spark.executor.port", "2.0.0", "Not used any more")
)

Map(configs.map { cfg => (cfg.key -> cfg) } : _*)
Expand Down
9 changes: 4 additions & 5 deletions core/src/main/scala/org/apache/spark/SparkEnv.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ object SparkEnv extends Logging {
SparkContext.DRIVER_IDENTIFIER,
bindAddress,
advertiseAddress,
port,
Option(port),
isLocal,
numCores,
ioEncryptionKey,
Expand All @@ -194,7 +194,6 @@ object SparkEnv extends Logging {
conf: SparkConf,
executorId: String,
hostname: String,
port: Int,
numCores: Int,
ioEncryptionKey: Option[Array[Byte]],
isLocal: Boolean): SparkEnv = {
Expand All @@ -203,7 +202,7 @@ object SparkEnv extends Logging {
executorId,
hostname,
hostname,
port,
None,
isLocal,
numCores,
ioEncryptionKey
Expand All @@ -220,7 +219,7 @@ object SparkEnv extends Logging {
executorId: String,
bindAddress: String,
advertiseAddress: String,
port: Int,
port: Option[Int],
isLocal: Boolean,
numUsableCores: Int,
ioEncryptionKey: Option[Array[Byte]],
Expand All @@ -243,7 +242,7 @@ object SparkEnv extends Logging {
}

val systemName = if (isDriver) driverSystemName else executorSystemName
val rpcEnv = RpcEnv.create(systemName, bindAddress, advertiseAddress, port, conf,
val rpcEnv = RpcEnv.create(systemName, bindAddress, advertiseAddress, port.getOrElse(-1), conf,
securityManager, clientMode = !isDriver)

// Figure out which port RpcEnv actually bound to in case the original port is 0 or occupied.
Expand Down
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ private[spark] object CoarseGrainedExecutorBackend extends Logging {
}

val env = SparkEnv.createExecutorEnv(
driverConf, executorId, hostname, -1, cores, cfg.ioEncryptionKey, isLocal = false)
driverConf, executorId, hostname, cores, cfg.ioEncryptionKey, isLocal = false)

env.rpcEnv.setupEndpoint("Executor", new CoarseGrainedExecutorBackend(
env.rpcEnv, driverUrl, executorId, hostname, cores, userClassPath, env))
Expand Down
2 changes: 1 addition & 1 deletion resource-managers/mesos/src/main/scala/org/apache/spark/executor/MesosExecutorBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ private[spark] class MesosExecutorBackend
Seq[(String, String)](("spark.app.id", frameworkInfo.getId.getValue))
val conf = new SparkConf(loadDefaults = true).setAll(properties)
val env = SparkEnv.createExecutorEnv(
conf, executorId, slaveInfo.getHostname, -1, cpusPerTask, None, isLocal = false)
conf, executorId, slaveInfo.getHostname, cpusPerTask, None, isLocal = false)

executor = new Executor(
executorId,
Expand Down