[SPARK-4415] [PySpark] JVM should exit after Python exit

[davies]

When JVM is started in a Python process, it should exit once the stdin is closed.

test: add spark.driver.memory in conf/spark-defaults.conf

davies@dm:~/work/spark$ cat conf/spark-defaults.conf
spark.driver.memory       8g
davies@dm:~/work/spark$ bin/pyspark
>>> quit
davies@dm:~/work/spark$ jps
4931 Jps
286
davies@dm:~/work/spark$ python wc.py
943738
0.719928026199
davies@dm:~/work/spark$ jps
286
4990 Jps

[davies]

cc @andrewor14

[SparkQA]

Test build #23397 has started for PR 3274 at commit 050651f.

• This patch merges cleanly.

[vanzin]

So, if I understand correctly this handles the case where pyspark apps are not executed using the pyspark script, but with python directly?

It feels a little bit sketchy to support that, but the change looks good.

[andrewor14]

I would just call this PYSPARK, and rename the variable isPySpark

[andrewor14]

(before you do that, can you search the codebase to see if we already use the PYSPARK environment variable? It would be good to avoid clobbering it)

[andrewor14]

@vanzin Yes your understanding is correct. I think this is safe to support in case the user wants to use different versions of python. Otherwise this silently does not kill the outer process, which is unintuitive.

[vanzin]

in case the user wants to use different versions of python

Is there a way to define the python executable to use for the executors? Otherwise this will end up in tears, since pickle is not compatible across python versions...

[SparkQA]

Test build #23399 has started for PR 3274 at commit ce8599c.

• This patch merges cleanly.

[davies]

@vanzin The python used in exector could be defined by PYSPARK_PYTHON, so it's easy to run pyspark with different python, such as:

$ PYSPARK_PYTHON=pypy pypy wc.py

Or run python with any options

$ python -u -s -B wc.py

[vanzin]

Ah, cool. Thanks for clarifying.

[SparkQA]

Test build #23397 has finished for PR 3274 at commit 050651f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • case class InSet(value: Expression, hset: Set[Any])
  • case class In(attribute: String, values: Array[Any]) extends Filter

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23397/
Test PASSed.

[andrewor14]

Sorry, I missed this case. Actually not all pyspark applications should go through this path. I think we should rename this variable to IS_PYTHON_SUBPROCESS on second thought:

env["IS_PYTHON_SUBPROCESS"] = "1" # Tell JVM to exit after python exits

[andrewor14]

Hey @davies sorry I missed the case in which the python application is run through spark-submit, which doesn't actually go through this code path. I have provided suggestions for renaming the variables and rephrasing certain comments.

[SparkQA]

Test build #23399 timed out for PR 3274 at commit ce8599c after a configured wait of 120m.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23399/
Test FAILed.

[SparkQA]

Test build #23406 has started for PR 3274 at commit df0e524.

• This patch merges cleanly.

[SparkQA]

Test build #23406 has finished for PR 3274 at commit df0e524.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23406/
Test PASSed.

[andrewor14]

can run. I'll fix this when I merge it

[andrewor14]

Ok merging this master 1.2 thanks @davies