Skip to content

Commit 11f1c8d

Browse files
wuqingxinholdenk
authored andcommitted
[SPARK-26446][CORE] Add cachedExecutorIdleTimeout docs at ExecutorAllocationManager
## What changes were proposed in this pull request? Add docs to describe how remove policy act while considering the property `spark.dynamicAllocation.cachedExecutorIdleTimeout` in ExecutorAllocationManager ## How was this patch tested? comment-only PR. Closes apache#23386 from TopGunViper/SPARK-26446. Authored-by: wuqingxin <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent e6d6eaf commit 11f1c8d

File tree

1 file changed

+8
-2
lines changed

1 file changed

+8
-2
lines changed

core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,8 @@ import org.apache.spark.util.{Clock, SystemClock, ThreadUtils, Utils}
5757
* a long time to ramp up under heavy workloads.
5858
*
5959
* The remove policy is simpler: If an executor has been idle for K seconds, meaning it has not
60-
* been scheduled to run any tasks, then it is removed.
60+
* been scheduled to run any tasks, then it is removed. Note that an executor caching any data
61+
* blocks will be removed if it has been idle for more than L seconds.
6162
*
6263
* There is no retry logic in either case because we make the assumption that the cluster manager
6364
* will eventually fulfill all requests it receives asynchronously.
@@ -81,7 +82,12 @@ import org.apache.spark.util.{Clock, SystemClock, ThreadUtils, Utils}
8182
* This is used only after the initial backlog timeout is exceeded
8283
*
8384
* spark.dynamicAllocation.executorIdleTimeout (K) -
84-
* If an executor has been idle for this duration, remove it
85+
* If an executor without caching any data blocks has been idle for this duration, remove it
86+
*
87+
* spark.dynamicAllocation.cachedExecutorIdleTimeout (L) -
88+
* If an executor with caching data blocks has been idle for more than this duration,
89+
* the executor will be removed
90+
*
8591
*/
8692
private[spark] class ExecutorAllocationManager(
8793
client: ExecutorAllocationClient,

0 commit comments

Comments
 (0)