[SPARK-5336][YARN]spark.executor.cores must not be less than spark.task.cpus #4123

WangTaoTheTonic · 2015-01-20T17:27:24Z

https://issues.apache.org/jira/browse/SPARK-5336

SparkQA · 2015-01-20T17:32:42Z

Test build #25826 has started for PR 4123 at commit 3112af9.

This patch merges cleanly.

SparkQA · 2015-01-20T18:40:35Z

Test build #25826 has finished for PR 4123 at commit 3112af9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-20T18:40:39Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25826/
Test PASSed.

sryza · 2015-01-21T08:16:12Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

Nit: import ordering should be alphabetical

sryza · 2015-01-21T08:18:44Z

We should warn about this in standalone mode and mesos coarse grained mode as well, right?

WangTaoTheTonic · 2015-01-21T08:41:15Z

@sryza
In standalone and mesos mode it is a bit different. JIRA is here:https://issues.apache.org/jira/browse/SPARK-5337
Seems like @CodingCat has created a PR: #4129, we can discuss the details there.

SparkQA · 2015-01-21T09:12:42Z

Test build #25883 has started for PR 4123 at commit d03d6fa.

This patch merges cleanly.

SparkQA · 2015-01-21T09:32:33Z

Test build #25887 has started for PR 4123 at commit 9632d3a.

This patch merges cleanly.

WangTaoTheTonic · 2015-01-21T09:34:11Z

Actually when cluster has not enough cores, the driver will also hang up. I made a trivial change here and hope it would not make much noise for this pr.

SparkQA · 2015-01-21T10:20:14Z

Test build #25883 has finished for PR 4123 at commit d03d6fa.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-21T10:20:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25883/
Test PASSed.

SparkQA · 2015-01-21T10:44:57Z

Test build #25887 has finished for PR 4123 at commit 9632d3a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-21T10:45:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25887/
Test PASSed.

tgravescs · 2015-01-21T14:52:27Z

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala

only nit here is that I might not have specified it via spark.executor.cores but rather via the spark-submit --executor-cores option. Perhaps we should just say "Executor cores must".

SparkQA · 2015-01-21T15:32:38Z

Test build #25890 has started for PR 4123 at commit 6c9676a.

This patch merges cleanly.

WangTaoTheTonic · 2015-01-21T15:32:43Z

@tgravescs Updated as you said. Perhaps we should do the same change in ExecutorAllocationManager, but I am not familiar with that feature, so let's leave it to guy who is in charge of that.

tgravescs · 2015-01-21T15:41:45Z

I agree, the text was that way in ExecutorAllocationManager so lets leave it for now.

This looks good. Thanks @WangTaoTheTonic

SparkQA · 2015-01-21T16:41:23Z

Test build #25890 has finished for PR 4123 at commit 6c9676a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-01-21T16:41:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25890/
Test PASSed.

…sk.cpus https://issues.apache.org/jira/browse/SPARK-5336 Author: WangTao <[email protected]> Author: WangTaoTheTonic <[email protected]> Closes apache#4123 from WangTaoTheTonic/SPARK-5336 and squashes the following commits: 6c9676a [WangTao] Update ClientArguments.scala 9632d3a [WangTaoTheTonic] minor comment fix d03d6fa [WangTaoTheTonic] import ordering should be alphabetical' 3112af9 [WangTao] spark.executor.cores must not be less than spark.task.cpus

spark.executor.cores must not be less than spark.task.cpus

3112af9

sryza reviewed Jan 21, 2015
View reviewed changes

yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala Outdated

Copy link

Contributor

sryza Jan 21, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: import ordering should be alphabetical

import ordering should be alphabetical'

d03d6fa

minor comment fix

9632d3a

CodingCat mentioned this pull request Jan 21, 2015

[SPARK-5337][Mesos][Standalone] respect spark.task.cpus when scheduling Applications #4129

Closed

tgravescs reviewed Jan 21, 2015
View reviewed changes

Update ClientArguments.scala

6c9676a

asfgit closed this in 8c06a5f Jan 21, 2015

CodingCat mentioned this pull request Sep 5, 2015

[SPARK-5337][Standalone] respect spark.task.cpus when scheduling Applications #8610

Closed

[SPARK-5336][YARN]spark.executor.cores must not be less than spark.task.cpus #4123

[SPARK-5336][YARN]spark.executor.cores must not be less than spark.task.cpus #4123

Uh oh!

Conversation

WangTaoTheTonic commented Jan 20, 2015

Uh oh!

SparkQA commented Jan 20, 2015

Uh oh!

SparkQA commented Jan 20, 2015

Uh oh!

AmplabJenkins commented Jan 20, 2015

Uh oh!

sryza Jan 21, 2015

Choose a reason for hiding this comment

Uh oh!

sryza commented Jan 21, 2015

Uh oh!

WangTaoTheTonic commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

WangTaoTheTonic commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

AmplabJenkins commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

AmplabJenkins commented Jan 21, 2015

Uh oh!

tgravescs Jan 21, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

WangTaoTheTonic commented Jan 21, 2015

Uh oh!

tgravescs commented Jan 21, 2015

Uh oh!

SparkQA commented Jan 21, 2015

Uh oh!

AmplabJenkins commented Jan 21, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants