Skip to content

[SPARK-1808] Route bin/pyspark through Spark submit #799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 15 commits into from
Closed

[SPARK-1808] Route bin/pyspark through Spark submit #799

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
a371d26
Route bin/pyspark through Spark submit
andrewor14 May 16, 2014
f04aaa4
Merge branch 'master' of github.com:apache/spark into pyspark-submit
andrewor14 May 16, 2014
afe47bf
Fix spark shell
andrewor14 May 16, 2014
fe4c8a7
Update --help for bin/pyspark
andrewor14 May 16, 2014
6fba412
Deal with quotes + address various comments
andrewor14 May 16, 2014
a823661
Fix --die-on-broken-pipe not propagated properly
andrewor14 May 16, 2014
05879fa
Merge branch 'master' of github.com:apache/spark into pyspark-submit
andrewor14 May 16, 2014
06eb138
Use shlex instead of writing our own parser
andrewor14 May 16, 2014
b7ba0d8
Address a few comments (minor)
andrewor14 May 16, 2014
7eebda8
Merge branch 'master' of github.com:apache/spark into pyspark-submit
andrewor14 May 16, 2014
456d844
Guard against shlex hanging if PYSPARK_SUBMIT_ARGS is not set
andrewor14 May 16, 2014
1866f85
Windows is not cooperating
andrewor14 May 16, 2014
c8cb3bf
Handle perverse app names (with escaped quotes)
andrewor14 May 17, 2014
01066fa
bin/pyspark for Windows
andrewor14 May 17, 2014
bf37e36
Minor changes
andrewor14 May 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address a few comments (minor)
  • Loading branch information
@andrewor14
andrewor14 committed May 16, 2014
commit b7ba0d86b350de91628f5d3d28796e4cb41f79d0
6 changes: 3 additions & 3 deletions bin/pyspark
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ export SPARK_HOME="$FWDIR"

SCALA_VERSION=2.10

if [[ "$@" == *--help* ]]; then
if [[ "$@" = *--help ]] || [[ "$@" = *--h ]]; then
Copy link
Contributor

@pwendell pwendell May 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doesn't seem to match -h for me... not sure if that was the intention of the second condition.

Copy link
Contributor

@pwendell pwendell May 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think maybe you want this to say

if [[ "$@" = *--help ]] || [[ "$@" = *-h ]]; then

One thing is this will only detect if -h or --help is the _last_argument, but I think anything other than that is pretty tricky.

echo "Usage: ./bin/pyspark [options]"
./bin/spark-submit --help 2>&1 | grep -v Usage 1>&2
exit 0
Expand Down Expand Up @@ -79,8 +79,8 @@ export PYSPARK_SUBMIT_ARGS

# If a python file is provided, directly run spark-submit.
if [[ "$1" =~ \.py$ ]]; then
echo -e "\nWARNING: Running python applications through ./bin/pyspark is deprecated as of Spark 1.0."
echo -e "Use ./bin/spark-submit <python file>\n"
echo -e "\nWARNING: Running python applications through ./bin/pyspark is deprecated as of Spark 1.0." 1>&2
echo -e "Use ./bin/spark-submit <python file>\n" 1>&2
exec $FWDIR/bin/spark-submit "$@"
else
# Only use ipython if no command line arguments were provided [SPARK-1134]
Expand Down
2 changes: 1 addition & 1 deletion bin/spark-shell
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ esac
# Enter posix mode for bash
set -o posix

if [[ "$@" == *--help* ]]; then
if [[ "$@" = *--help ]] || [[ "$@" = *--h ]]; then
Copy link
Contributor

@pwendell pwendell May 17, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the second condition for? It doesn't seem to match -h. I noticed if I run this with -h then I seem to just get the help option from spark-submit.

patrick@patrick-t430s:~/Documents/spark$ ./bin/spark-shell -h
Usage: spark-submit [options] <app jar | python file> [app options]
Options:

echo "Usage: ./bin/spark-shell [options]"
./bin/spark-submit --help 2>&1 | grep -v Usage 1>&2
exit 0
Expand Down