[SPARK-25073][Yarn] AM and Executor Memory validation message is not proper while submitting spark yarn application

[sujith71955]

## What changes were proposed in this pull request?
When the yarn.nodemanager.resource.memory-mb or yarn.scheduler.maximum-allocation-mb
memory assignment is insufficient, Spark always reports an error request to adjust
yarn.scheduler.maximum-allocation-mb even though in message it shows the memory value
of yarn.nodemanager.resource.memory-mb parameter,As the error Message is bit misleading to the user we can modify the same, We can keep the error message same as executor memory validation message.

Defintion of yarn.nodemanager.resource.memory-mb:
Amount of physical memory, in MB, that can be allocated for containers. It means the amount of memory YARN can utilize on this node and therefore this property should be lower then the total memory of that machine.
yarn.scheduler.maximum-allocation-mb:
It defines the maximum memory allocation available for a container in MB
it means RM can only allocate memory to containers in increments of "yarn.scheduler.minimum-allocation-mb" and not exceed "yarn.scheduler.maximum-allocation-mb" and It should not be more than total allocated memory of the Node.

## How was this patch tested?
Manually tested in hdfs-Yarn clustaer

[srowen]

What does this add?

[sujith71955]

As mentioned in the JIRA even though the memory defined in the yarn.nodemanager.resource.memory-mb parameter is less eg:
yarn.scheduler.maximum-allocation-mb =15g and yarn.nodemanager.resource.memory-mb =8g
Launch spark-shell --master yarn --conf spark.yarn.am.memory=10g
we are getting below error
java.lang.IllegalArgumentException: Required AM memory (10240+1024 MB) is above the max threshold (8096 MB) of this cluster! Please increase the value of 'yarn.scheduler.maximum-allocation-mb'.
This message is very confusing to a user since spark indicate to increase the arn.scheduler.maximum-allocation-mb which is actually more than 10G as per this scenario whereas the issue is with yarn.nodemanager.resource.memory-mb,

[srowen]

Both settings may have to change. But your text change doesn't mention anything new. Why does this help? I also don't think that the current behavior is wrong

[sujith71955]

Where as in the executor memory validation check we are displaying the proper message considering both yarn.nodemanager.resource.memory-mb and yarn.scheduler.maximum-allocation-mb

    if (executorMem > maxMem) {
      throw new IllegalArgumentException(s"Required executor memory ($executorMemory" +
        s"+$executorMemoryOverhead MB) is above the max threshold ($maxMem MB) of this cluster! " +
        "Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or " +
        "'yarn.nodemanager.resource.memory-mb and increase the memory appropriately.")
    }

so same message i think is required for am memory validation as well,

[sujith71955]

Please let me know for any suggestions. Thanks

[sujith71955]

Yes both settings has to be changed, but for the AM container memory allocation validation only yarn.scheduler.maximum-allocation-mb parameter is mentioned ,message does not indicate to increase yarn.nodemanager.resource.memory-mb, in my change i updated this message by including "yarn.nodemanager.resource.memory-mb" paramter

[srowen]

I understand the change below, that's probably fine. The change above doesn't seem to add anything. You want to tell the user to check both settings in both cases.

[sujith71955]

just trying to make the messages more informative by mentioning to increase the memory appropriately as per the business use-case .i shall remove the part where we are mentioning about 'increasing memory' if it doesn't make much difference to the information we are trying to convey.

[srowen]

Sure, though the solution may be to reduce the app memory usage rather than increase the YARN limits (which only admins can do). I think that's why the message more neutrally suggests that these are the limits that the app is hitting.

[sujith71955]

Got your point, i will update as per the suggestion. Thanks for the feedback Sean.

[jerryshao]

ok to test.

[SparkQA]

Test build #95196 has finished for PR 22199 at commit 3f4051c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.