Skip to content

[SPARK-26843][MESOS] Use ConfigEntry for hardcoded configs for "mesos" resource manager #23743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HeartSaVioR wants to merge 7 commits into from
Closed

[SPARK-26843][MESOS] Use ConfigEntry for hardcoded configs for "mesos" resource manager #23743

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
69751cb
[SPARK-26843][MESOS] Use ConfigEntry for hardcoded configs for "mesos…
HeartSaVioR Feb 7, 2019
c156fe0
Also apply to the change of commit b4e1d145135445eeed85784dab0c2c0889…
HeartSaVioR Feb 7, 2019
c2d0e04
Address review comment from dongjoon-hyun
HeartSaVioR Feb 7, 2019
f103b4e
Address review comments from dongjoon-hyun
HeartSaVioR Feb 9, 2019
e9a180e
Copy documentation on new ConfigEntry from running-mesos.md
HeartSaVioR Feb 10, 2019
7986b0d
Minor sync-up with doc
HeartSaVioR Feb 10, 2019
c5bee33
Reduce lines in too-way-long doc
HeartSaVioR Feb 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address review comment from dongjoon-hyun
  • Loading branch information
@HeartSaVioR
HeartSaVioR committed Feb 7, 2019
commit c2d0e04b4ac11ec84dc8857c6dba3dca28bd27a6
4 changes: 2 additions & 2 deletions ...mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterScheduler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,7 @@ private[spark] class MesosClusterScheduler(
private def getDriverEnvironment(desc: MesosDriverDescription): Environment = {
// TODO(mgummelt): Don't do this here. This should be passed as a --conf
val commandEnv = adjust(desc.command.environment, "SPARK_SUBMIT_OPTS", "")(
v => s"$v -Dspark.mesos.driver.frameworkId=${getDriverFrameworkID(desc)}"
v => s"$v -D${config.DRIVER_FRAMEWORK_ID.key}=${getDriverFrameworkID(desc)}"
)

val env = desc.conf.getAllWithPrefix(config.DRIVER_ENV_PREFIX) ++ commandEnv
Expand Down Expand Up @@ -423,7 +423,7 @@ private[spark] class MesosClusterScheduler(
case "container" => true
case "host" => false
case other =>
logWarning(s"Unknown spark.mesos.appJar.local.resolution.mode $other, using host.")
logWarning(s"Unknown ${config.APPLICATION_JAR_LOCAL_RESOLUTION_MODE} $other, using host.")
false
}
Copy link
Member

@dongjoon-hyun dongjoon-hyun Feb 8, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have checkValue, we can simplify the above logic line 422 ~ 426, too.

isLocalJar && isContainerLocal
Expand Down
4 changes: 2 additions & 2 deletions ...s/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -554,8 +554,8 @@ trait MesosSchedulerUtils extends Logging {
* framework ID, the driver calls this method after the first registration.
*/
def unsetFrameworkID(sc: SparkContext) {
sc.conf.remove("spark.mesos.driver.frameworkId")
System.clearProperty("spark.mesos.driver.frameworkId")
sc.conf.remove(mesosConfig.DRIVER_FRAMEWORK_ID)
System.clearProperty(mesosConfig.DRIVER_FRAMEWORK_ID.key)
}

def mesosToTaskState(state: MesosTaskState): TaskState.TaskState = state match {
Expand Down