Skip to content

[SPARK-1808] Route bin/pyspark through Spark submit #799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
andrewor14 wants to merge 15 commits into from
Closed

[SPARK-1808] Route bin/pyspark through Spark submit #799

Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
a371d26
Route bin/pyspark through Spark submit
andrewor14 May 16, 2014
f04aaa4
Merge branch 'master' of github.com:apache/spark into pyspark-submit
andrewor14 May 16, 2014
afe47bf
Fix spark shell
andrewor14 May 16, 2014
fe4c8a7
Update --help for bin/pyspark
andrewor14 May 16, 2014
6fba412
Deal with quotes + address various comments
andrewor14 May 16, 2014
a823661
Fix --die-on-broken-pipe not propagated properly
andrewor14 May 16, 2014
05879fa
Merge branch 'master' of github.com:apache/spark into pyspark-submit
andrewor14 May 16, 2014
06eb138
Use shlex instead of writing our own parser
andrewor14 May 16, 2014
b7ba0d8
Address a few comments (minor)
andrewor14 May 16, 2014
7eebda8
Merge branch 'master' of github.com:apache/spark into pyspark-submit
andrewor14 May 16, 2014
456d844
Guard against shlex hanging if PYSPARK_SUBMIT_ARGS is not set
andrewor14 May 16, 2014
1866f85
Windows is not cooperating
andrewor14 May 16, 2014
c8cb3bf
Handle perverse app names (with escaped quotes)
andrewor14 May 17, 2014
01066fa
bin/pyspark for Windows
andrewor14 May 17, 2014
bf37e36
Minor changes
andrewor14 May 17, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update --help for bin/pyspark
  • Loading branch information
@andrewor14
andrewor14 committed May 16, 2014
commit fe4c8a74e21db08ec9a0111cf7f82e7cc62e3ccb
6 changes: 6 additions & 0 deletions bin/pyspark
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,12 @@ export SPARK_HOME="$FWDIR"

SCALA_VERSION=2.10

if [[ "$@" == *--help* ]]; then
Copy link
Contributor

@mateiz mateiz May 16, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this recognize -h also

echo "Usage: ./bin/pyspark [python file] [options]"
Copy link
Contributor

@mateiz mateiz May 16, 2014

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we want to deprecate submitting a Python file with this, drop the [python file]

./bin/spark-submit --help 2>&1 | grep -v Usage 1>&2
exit 0
fi

# Exit if the user hasn't compiled Spark
if [ ! -f "$FWDIR/RELEASE" ]; then
# Exit if the user hasn't compiled Spark
Expand Down