This repository was archived by the owner on Dec 4, 2024. It is now read-only.
[DCOS-46369] Spark Dispatcher should run Spark Drivers and Executors in the same virtual network if specified #472
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
akirillov
force-pushed
the
DCOS-46369-dispatcher-network-defaults-for-spark-applications
branch
from
dfc9ce9 to
a811b4a
Compare
samvantran
approved these changes
Jan 16, 2019
| "default_spark_dist": { | ||
| "hadoop_version": "2.7", | ||
| "uri": "https://downloads.mesosphere.com/spark/assets/spark-2.3.2-SNAPSHOT-3-bin-2.7.7.tgz" | ||
| "uri": "https://downloads.mesosphere.com/spark/assets/spark-2.3.2-SNAPSHOT-4-bin-2.7.7.tgz" |
Contributor
There was a problem hiding this comment.
not something for this PR but just making a note for us that we should probably timestamp these jars (when we build in CI)
Contributor
Author
There was a problem hiding this comment.
agreed, this is pretty painful when the changes happen in both Spark and spark-build. This should be overridden in the CI e.g. by means of env var passed to Makefile, and version update in manifest committed prior to merge.
akirillov
force-pushed
the
DCOS-46369-dispatcher-network-defaults-for-spark-applications
branch
from
a811b4a to
5fd1c2d
Compare
…in the same virtual network if specified
akirillov
force-pushed
the
DCOS-46369-dispatcher-network-defaults-for-spark-applications
branch
from
5fd1c2d to
a1759a7
Compare
vishnu2kmohan
deleted the
DCOS-46369-dispatcher-network-defaults-for-spark-applications
branch
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Resolves DCOS-46369
If Spark Dispatcher is started with a network name specified - it runs all submitted Spark Application within the same network until another network is specified.
spark.mesos.network.nameand/orspark.mesos.network.labelsare specified - they're added tomesos-cluster-dispatcher.propertieson dispatcher startHow were these changes tested?
Release Notes