[SPARK-35200][CORE] Avoid to recompute the pending speculative tasks in the ExecutorAllocationManager and remove some unnecessary code. #32306
Conversation
…in the ExecutorAllocationManager and remove some unnecessary code.
|
Can one of the admins verify this patch? |
|
@cloud-fan @Ngone51 Kindly review, thanks. |
| val unschedulableTaskSets = listener.pendingUnschedulableTaskSetsPerResourceProfile(rpId) | ||
| val running = listener.totalRunningTasksPerResourceProfile(rpId) | ||
| val numRunningOrPendingTasks = pending + running | ||
| val numRunningOrPendingTasks = pendingTask + pendingSpeculative + running |
There was a problem hiding this comment.
previously pending is pendingTasksPerResourceProfile(rp) + pendingSpeculativeTasksPerResourceProfile(rp), now we use pendingTasksPerResourceProfile() + pendingSpeculative, and pendingSpeculative simply calls pendingSpeculativeTasksPerResourceProfile(...)
seems like a straightforward code clean up.
There was a problem hiding this comment.
previously the pendingSpeculativeTasksPerResourceProfile(rp) will be called not only by totalPendingTasksPerResourceProfile(rpId) but also by val pendingSpeculative = listener.pendingSpeculativeTasksPerResourceProfile(rpId), this PR we only call the pendingSpeculativeTasksPerResourceProfile(rp) once to get the pendingSpeculativeTasks , and use it for numRunningOrPendingTasks and pendingSpeculative.
|
cc @tgravescs |
|
Thank you, @weixiuli and all. |
5 participants
What changes were proposed in this pull request?
Avoid to recompute the pending speculative tasks in the ExecutorAllocationManager, and remove some unnecessary code.
Why are the changes needed?
The number of the pending speculative tasks is recomputed in the ExecutorAllocationManager to calculate the maximum number of executors required. While , it only needs to be computed once to improve performance.
Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests.