[SPARK-23825][K8s] Requesting memory + memory overhead for pod memory #20943
Conversation
… drivers and executors
dvogelbacher
changed the title
dvogelbacher
changed the title
|
ok to test |
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Jenkins, ok to test |
|
Test build #88738 has finished for PR 20943 at commit
|
|
@felixcheung can you review this? |
felixcheung
left a comment
felixcheung
left a comment
There was a problem hiding this comment.
I think this is a reasonable point.
@foxish @mccheah @erikerlandson what do you think?
| val requests = resourceRequirements.getRequests.asScala | ||
| assert(requests("cpu").getAmount === "2") | ||
| assert(requests("memory").getAmount === "256Mi") | ||
| assert(requests("memory").getAmount === "456Mi") |
There was a problem hiding this comment.
I'm wondering - shouldn't this be 256M + 384M?
The documentation says - executorMemory * 0.10, with minimum of 384.
There was a problem hiding this comment.
nvm - @liyinan926 just pointed out that it's being set explicitly above which I totally missed.
| val requests = resourceRequirements.getRequests.asScala | ||
| assert(requests("cpu").getAmount === "2") | ||
| assert(requests("memory").getAmount === "256Mi") | ||
| assert(requests("memory").getAmount === "456Mi") |
There was a problem hiding this comment.
nvm - @liyinan926 just pointed out that it's being set explicitly above which I totally missed.
dvogelbacher
changed the title
## What changes were proposed in this pull request? Kubernetes driver and executor pods should request `memory + memoryOverhead` as their resources instead of just `memory`, see https://issues.apache.org/jira/browse/SPARK-23825 ## How was this patch tested? Existing unit tests were adapted. Author: David Vogelbacher <[email protected]> Closes apache#20943 from dvogelbacher/spark-23825.
## What changes were proposed in this pull request? Kubernetes driver and executor pods should request `memory + memoryOverhead` as their resources instead of just `memory`, see https://issues.apache.org/jira/browse/SPARK-23825 ## How was this patch tested? Existing unit tests were adapted. Author: David Vogelbacher <[email protected]> Closes apache#20943 from dvogelbacher/spark-23825.
5 participants
What changes were proposed in this pull request?
Kubernetes driver and executor pods should request
memory + memoryOverheadas their resources instead of justmemory, see https://issues.apache.org/jira/browse/SPARK-23825How was this patch tested?
Existing unit tests were adapted.