Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
250cb95
Do not ignore spark.driver.extra* for client mode
andrewor14 Aug 4, 2014
a2ab1b0
Parse spark.driver.extra* in bash
andrewor14 Aug 6, 2014
0025474
Revert SparkSubmit handling of --driver-* options for only cluster mode
andrewor14 Aug 6, 2014
63ed2e9
Merge branch 'master' of github.com:apache/spark into submit-driver-e…
andrewor14 Aug 6, 2014
75ee6b4
Remove accidentally added file
andrewor14 Aug 6, 2014
8843562
Fix compilation issues...
andrewor14 Aug 6, 2014
98dd8e3
Add warning if properties file does not exist
andrewor14 Aug 6, 2014
130f295
Handle spark.driver.memory too
andrewor14 Aug 6, 2014
4edcaa8
Redirect stdout to stderr for python
andrewor14 Aug 6, 2014
e5cfb46
Collapse duplicate code + fix potential whitespace issues
andrewor14 Aug 6, 2014
4ec22a1
Merge branch 'master' of github.com:apache/spark into submit-driver-e…
andrewor14 Aug 6, 2014
ef12f74
Minor formatting
andrewor14 Aug 6, 2014
fa2136e
Escape Java options + parse java properties files properly
andrewor14 Aug 7, 2014
dec2343
Only export variables if they exist
andrewor14 Aug 7, 2014
a4df3c4
Move parsing and escaping logic to utils.sh
andrewor14 Aug 7, 2014
de765c9
Print spark-class command properly
andrewor14 Aug 7, 2014
8e552b7
Include an example of spark.*.extraJavaOptions
andrewor14 Aug 7, 2014
c13a2cb
Merge branch 'master' of github.com:apache/spark into submit-driver-e…
andrewor14 Aug 7, 2014
c854859
Add small comment
andrewor14 Aug 7, 2014
1cdc6b1
Fix bug: escape escaped double quotes properly
andrewor14 Aug 7, 2014
45a1eb9
Fix bug: escape escaped backslashes and quotes properly...
andrewor14 Aug 7, 2014
aabfc7e
escape -> split (minor)
andrewor14 Aug 7, 2014
a992ae2
Escape spark.*.extraJavaOptions correctly
andrewor14 Aug 7, 2014
c7b9926
Minor changes to spark-defaults.conf.template
andrewor14 Aug 7, 2014
5d8f8c4
Merge branch 'master' of github.com:apache/spark into submit-driver-e…
andrewor14 Aug 7, 2014
e793e5f
Handle multi-line arguments
andrewor14 Aug 8, 2014
c2273fc
Fix typo (minor)
andrewor14 Aug 8, 2014
b3c4cd5
Fix bug: count the number of quotes instead of detecting presence
andrewor14 Aug 8, 2014
4ae24c3
Fix bug: escape properly in quote_java_property
andrewor14 Aug 8, 2014
8d26a5c
Add tests for bash/utils.sh
andrewor14 Aug 8, 2014
2732ac0
Integrate BASH tests into dev/run-tests + log error properly
andrewor14 Aug 9, 2014
aeb79c7
Merge branch 'master' of github.com:apache/spark into handle-configs-…
andrewor14 Aug 9, 2014
8d4614c
Merge branch 'master' of github.com:apache/spark into handle-configs-…
andrewor14 Aug 16, 2014
56ac247
Use eval and set to simplify splitting
andrewor14 Aug 16, 2014
bd0d468
Simplify parsing config file by ignoring multi-line arguments
andrewor14 Aug 16, 2014
be99eb3
Fix tests to not include multi-line configs
andrewor14 Aug 16, 2014
371cac4
Add function prefix (minor)
andrewor14 Aug 16, 2014
fa11ef8
Parse the properties file only if the special configs exist
andrewor14 Aug 16, 2014
7396be2
Explicitly comment that multi-line properties are not supported
andrewor14 Aug 16, 2014
7a4190a
Merge branch 'master' of github.com:apache/spark into handle-configs-…
andrewor14 Aug 16, 2014
c886568
Fix lines too long + a few comments / style (minor)
andrewor14 Aug 16, 2014
0effa1e
Add code in Scala that handles special configs
andrewor14 Aug 19, 2014
a396eda
Nullify my own hard work to simplify bash
andrewor14 Aug 19, 2014
c37e08d
Revert a few more changes
andrewor14 Aug 19, 2014
3a8235d
Only parse the properties file if special configs exist
andrewor14 Aug 19, 2014
7d94a8d
Merge branch 'master' of github.com:apache/spark into handle-configs-…
andrewor14 Aug 19, 2014
b71f52b
Revert a few more changes (minor)
andrewor14 Aug 19, 2014
c84f5c8
Remove debug print statement (minor)
andrewor14 Aug 19, 2014
158f813
Remove "client mode" boolean argument
andrewor14 Aug 19, 2014
a91ea19
Fix precedence of library paths, classpath, java opts and memory
andrewor14 Aug 19, 2014
1ea6bbe
SparkClassLauncher -> SparkSubmitDriverBootstrapper
andrewor14 Aug 19, 2014
d6488f9
Merge branch 'master' of github.com:apache/spark into handle-configs-…
andrewor14 Aug 19, 2014
19464ad
SPARK_SUBMIT_JAVA_OPTS -> SPARK_SUBMIT_OPTS
andrewor14 Aug 19, 2014
8867a09
A few more naming things (minor)
andrewor14 Aug 19, 2014
9ba37e2
Don't barf when the properties file does not exist
andrewor14 Aug 19, 2014
a78cb26
Revert a few changes in utils.sh (minor)
andrewor14 Aug 19, 2014
d0f20db
Don't pass empty library paths, classpath, java opts etc.
andrewor14 Aug 19, 2014
9a778f6
Fix PySpark: actually kill driver on termination
andrewor14 Aug 20, 2014
51aeb01
Filter out JVM memory in Scala rather than Bash (minor)
andrewor14 Aug 20, 2014
ff34728
Minor comments
andrewor14 Aug 20, 2014
08fd788
Warn against external usages of SparkSubmitDriverBootstrapper
andrewor14 Aug 20, 2014
24dba60
Merge branch 'master' of github.com:apache/spark into handle-configs-…
andrewor14 Aug 20, 2014
bed4bdf
Change a few comments / messages (minor)
andrewor14 Aug 20, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK_SUBMIT_JAVA_OPTS -> SPARK_SUBMIT_OPTS
  • Loading branch information
andrewor14 committed Aug 19, 2014
commit 19464ada6d69fd233cbee455bd3af1845a4e2d9d
5 changes: 3 additions & 2 deletions bin/spark-class
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,10 @@ case "$1" in
OUR_JAVA_MEM=${SPARK_EXECUTOR_MEMORY:-$DEFAULT_MEM}
;;

# Spark submit uses SPARK_JAVA_OPTS + SPARK_SUBMIT_JAVA_OPTS + SPARK_DRIVER_MEMORY + SPARK_SUBMIT_DRIVER_MEMORY.
# Spark submit uses SPARK_JAVA_OPTS + SPARK_SUBMIT_OPTS +
# SPARK_DRIVER_MEMORY + SPARK_SUBMIT_DRIVER_MEMORY.
'org.apache.spark.deploy.SparkSubmit')
OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_SUBMIT_JAVA_OPTS"
OUR_JAVA_OPTS="$SPARK_JAVA_OPTS $SPARK_SUBMIT_OPTS"
OUR_JAVA_MEM=${SPARK_DRIVER_MEMORY:-$DEFAULT_MEM}
if [ -n "$SPARK_SUBMIT_LIBRARY_PATH" ]; then
OUR_JAVA_OPTS="$OUR_JAVA_OPTS -Djava.library.path=$SPARK_SUBMIT_LIBRARY_PATH"
Expand Down
2 changes: 1 addition & 1 deletion bin/spark-submit
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ while (($#)); do
elif [ "$1" = "--driver-class-path" ]; then
export SPARK_SUBMIT_CLASSPATH=$2
elif [ "$1" = "--driver-java-options" ]; then
export SPARK_SUBMIT_JAVA_OPTS=$2
export SPARK_SUBMIT_OPTS=$2
fi
shift
done
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ private[spark] object SparkSubmitDriverBootstrapper {
val submitDriverMemory = sys.env.get("SPARK_SUBMIT_DRIVER_MEMORY")
val submitLibraryPath = sys.env.get("SPARK_SUBMIT_LIBRARY_PATH")
val submitClasspath = sys.env.get("SPARK_SUBMIT_CLASSPATH")
val submitJavaOpts = sys.env.get("SPARK_SUBMIT_JAVA_OPTS")
val submitJavaOpts = sys.env.get("SPARK_SUBMIT_OPTS")

assume(runner != null, "RUNNER must be set")
assume(classpath != null, "CLASSPATH must be set")
Expand Down Expand Up @@ -87,7 +87,7 @@ private[spark] object SparkSubmitDriverBootstrapper {
}
val newJavaOpts =
if (submitJavaOpts.isDefined) {
// SPARK_SUBMIT_JAVA_OPTS is already captured in JAVA_OPTS
// SPARK_SUBMIT_OPTS is already captured in JAVA_OPTS
javaOpts
} else {
javaOpts + " " + confJavaOpts
Expand Down