Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Remove double quotes in spark/bin batch files to avoid cutting off ar…
…guments that double quoted as contain special character.

To simply validate (for example, mysql connection string), cannot just start it :
spark-submit.cmd --jars just-to-start "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table

After this fix :
(1) Keep not working : full path has space like "D:\opengit\spark\bin - Copy\spark-submit.cmd" --jars any-jars "jdbc:mysql://localhost:3306/lzdb?user=guest&password=abc123" my_table
(2) Keep working : currently can works.
(3) Will work by this fix: argument has quotes if full path no space.
By the way, I didn't change the pyspark , R, beeline etc. scripts because they seems work fine for long.

What's more in addition, a tool to quickly change the files in spark/bin if you like : https://github.com/qualiu/lzmw
(1) Remove quotes : lzmw -i -t "(^cmd.*?/[VCE])\s+\"+(%~dp0\S+\.cmd)\"+" -o "$1 $2" --nf "pyspark|sparkR|beeline|example" -p . -R
(2) Add/Restore   : lzmw -it "\"*(%~dp0\S+\.cmd)\"*" -o "\"$1\"" -p . -R
(3) Or remove the head : lzmw -f "\.cmd$" -it "^cmd /V /E /C " -o "" -p %CD% -R
  • Loading branch information
lqm678 committed Aug 26, 2016
commit a6b1c60c83902f023e230057a2b0d6ea99a30426
2 changes: 1 addition & 1 deletion bin/spark-class.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ rem
rem This is the entry point for running a Spark class. To avoid polluting
rem the environment, it just launches a new cmd to do the real work.

cmd /V /E /C "%~dp0spark-class2.cmd" %*
cmd /V /E /C %~dp0spark-class2.cmd %*
2 changes: 1 addition & 1 deletion bin/spark-shell.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ rem
rem This is the entry point for running Spark shell. To avoid polluting the
rem environment, it just launches a new cmd to do the real work.

cmd /V /E /C "%~dp0spark-shell2.cmd" %*
cmd /V /E /C %~dp0spark-shell2.cmd %*
2 changes: 1 addition & 1 deletion bin/spark-submit.cmd
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,4 @@ rem
rem This is the entry point for running Spark submit. To avoid polluting the
rem environment, it just launches a new cmd to do the real work.

cmd /V /E /C "%~dp0spark-submit2.cmd" %*
cmd /V /E /C %~dp0spark-submit2.cmd %*