SPARK-3837. Warn when YARN kills containers for exceeding memory limits

[sryza]

I triggered the issue and verified the message gets printed on a pseudo-distributed cluster.

[AmplabJenkins]

Can one of the admins verify this patch?

[SparkQA]

QA tests have started for PR 2744 at commit af11185.

• This patch merges cleanly.

[SparkQA]

QA tests have finished for PR 2744 at commit af11185.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21564/Test PASSed.

[tgravescs]

Jenkins, test this please.

[tgravescs]

Any reason we don't print the diagnostics message here also?

[sryza]

Updated patch incorporates review comments

[tgravescs]

looks good. thanks @sryza!

[shivaram]

This patch seems to have broken the build ? See https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/821/console for an example

[sryza]

I'll look into it

On Oct 31, 2014, at 12:36 PM, Shivaram Venkataraman [email protected] wrote:

This patch seems to have broken the build ? See https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-with-YARN/HADOOP_PROFILE=hadoop-2.4,label=centos/821/console for an example

—
Reply to this email directly or view it on GitHub.

[shivaram]

I have a commit to just comment out the test at https://github.com/shivaram/spark-1/compare/fix-yarn-build?expand=1 -- But if you have a better fix, we can use that.

[jdanbrown]

@sryza I came across your PR while searching for ways to debug pyspark yarn containers running over yarn memory limits. This change will be very helpful; what are your thoughts on dumping the container process listing as well, like yarn MR tasks do? That's killer for figuring out which process in your container is out of control, like in the case of pyspark where you have a jvm and python processes to deal with.

Example:

• https://gist.githubusercontent.com/jdanbrown/cef65e9082b1f32d085e/raw/83347253348f246aa4f9989993e8fb6db14db313/gistfile1.txt

[sryza]

@jdanbrown that seems reasonable. Mind filing a JIRA for it?