-
Notifications
You must be signed in to change notification settings - Fork 29k
[SPARK-2849] Handle driver configs separately in client mode #1845
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
250cb95
a2ab1b0
0025474
63ed2e9
75ee6b4
8843562
98dd8e3
130f295
4edcaa8
e5cfb46
4ec22a1
ef12f74
fa2136e
dec2343
a4df3c4
de765c9
8e552b7
c13a2cb
c854859
1cdc6b1
45a1eb9
aabfc7e
a992ae2
c7b9926
5d8f8c4
e793e5f
c2273fc
b3c4cd5
4ae24c3
8d26a5c
2732ac0
aeb79c7
8d4614c
56ac247
bd0d468
be99eb3
371cac4
fa11ef8
7396be2
7a4190a
c886568
0effa1e
a396eda
c37e08d
3a8235d
7d94a8d
b71f52b
c84f5c8
158f813
a91ea19
1ea6bbe
d6488f9
19464ad
8867a09
9ba37e2
a78cb26
d0f20db
9a778f6
51aeb01
ff34728
08fd788
24dba60
bed4bdf
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
- Loading branch information
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -35,12 +35,12 @@ parse_java_property() { | |
| export JAVA_PROPERTY_VALUE | ||
| } | ||
|
|
||
| # Properly escape java options, dealing with whitespace, double quotes and backslashes. | ||
| # This accepts a string and returns the escaped list through ESCAPED_JAVA_OPTS. | ||
| escape_java_options() { | ||
| ESCAPED_JAVA_OPTS=() # return value | ||
| option_buffer="" # buffer for collecting parts of an option | ||
| opened_quotes=0 # whether we are expecting a closing double quotes | ||
| # Properly split java options, dealing with whitespace, double quotes and backslashes. | ||
| # This accepts a string and returns the resulting list through SPLIT_JAVA_OPTS. | ||
| split_java_options() { | ||
| SPLIT_JAVA_OPTS=() # return value | ||
|
||
| option_buffer="" # buffer for collecting parts of an option | ||
| opened_quotes=0 # whether we are expecting a closing double quotes | ||
| for word in $1; do | ||
| contains_quote=$(echo "$word" | sed "s/\\\\\"//g" | grep "\"") | ||
| if [[ -n "$contains_quote" ]]; then | ||
|
|
@@ -49,7 +49,7 @@ escape_java_options() { | |
| fi | ||
| if [[ $opened_quotes == 0 ]]; then | ||
| # Remove all non-escaped quotes around the value | ||
| ESCAPED_JAVA_OPTS+=("$( | ||
| SPLIT_JAVA_OPTS+=("$( | ||
| echo "$option_buffer $word" | \ | ||
| sed "s/^[[:space:]]*//" | \ | ||
| sed "s/\([^\\]\)\"/\1/g" | \ | ||
|
|
@@ -66,7 +66,7 @@ escape_java_options() { | |
| echo "Java options parse error! Expecting closing double quotes." 1>&2 | ||
| exit 1 | ||
| fi | ||
| export ESCAPED_JAVA_OPTS | ||
| export SPLIT_JAVA_OPTS | ||
| } | ||
|
|
||
| # Put double quotes around each of the given java options that is a system property. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This actually solves a more general problem than those reported in SPARK-2849 and SPARK-2914... the problem/feature is that in general we don't support quotes in any of the java option strings we have. I tested this in master and confirmed it doesn't work:
So it might be good to create another JIRA as well for this PR. One that just explains that none of the JAVA_OPTS variants we have correctly support quoted strings.