[SPARK-1507][YARN]specify num of cores for AM

[XuTingjun]

I add some configurations below.
spark.yarn.am.cores/SPARK_MASTER_CORES/SPARK_DRIVER_CORES for yarn-client mode;
spark.driver.cores for yarn-cluster mode.

[AmplabJenkins]

Can one of the admins verify this patch?

[sryza]

SPARK_MASTER_CORES uses "master" incorrectly. The only reason we have a SPARK_MASTER_MEMORY was to preserve backwards capability.

This patch also still appears to use "--driver-cores" and SPARK_DRIVER_CORES to set the AM memory in yarn-client mode, which we talked about avoiding on the previous PR.

[XuTingjun]

@sryza, I am not agree with you. I only add the below code into cluster mode. So the "--driver-cores" will not work in client mode.
OptionAssigner(args.driverCores, YARN, CLUSTER, clOption = "--driver-cores"),

[XuTingjun]

@andrewor14

[sryza]

no need to add in a deprecated "--master-cores" property

[sryza]

@XuTingjun ah, that's correct. Looking more closely, my confusion was stemming from some existing weirdness, which is that setting the "spark.driver.memory" property will set the application master memory even in client mode. Also, that passing "--driver-memory" as part of the YARN code's ClientArguments (but not to spark-submit) will do this as well.

My opinion is that we should fix those, though I suppose it could be argued that it breaks backwards compatibility?

[XuTingjun]

@sryza, do you mean "spark.driver.memory" works in yarn-client and yarn-cluster mode, so we should use one configuration maybe named "spark.driver.cores" to set am cores in yarn-client and yarn-cluster mode?

[sryza]

I think the best thing would be to split spark.driver.memory into spark.driver.memory and spark.yarn.am.memory, and to have the latter only work for the yarn-client AM.

[XuTingjun]

Yearh, I agree with you. Later I will fix this. Thanks @sryza

[XuTingjun]

@sryza, I have splited spark.driver.memory into spark.driver.memory and spark.yarn.am.memory. Please have a look.

[WangTaoTheTonic]

Sorry but I have already filed a PR that splits spark.driver.memory in #3607.

Could you please check if anyone already did the same work first?

[andrewor14]

indent

[andrewor14]

Hi @XuTingjun there seems to be a certain degree of overlap with work in #3607. Also, my concern with this PR is that it conflates "driver" and "AM" in a few places. For instance, --driver-cores should never configure the number of cores used by the AM in client mode.