Skip to content
Closed
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
review feedback
  • Loading branch information
squito committed Feb 20, 2018
commit e69851b9b5a1a6bc1ecc1c14a656d4c08572b9d7
Original file line number Diff line number Diff line change
Expand Up @@ -336,7 +336,8 @@ private[spark] class ExecutorAllocationManager(

// If the new target has not changed, avoid sending a message to the cluster manager
if (numExecutorsTarget < oldNumExecutorsTarget) {
// We lower the target number of executors but don't actively kill any yet. We do this
// We lower the target number of executors but don't actively kill any yet. Killing is
// controlled separately by an idle timeout. Its still helpful to reduce the target number
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: it's

// in case an executor just happens to get lost (eg., bad hardware, or the cluster manager
// preempts it) -- in that case, there is no point in trying to immediately get a new
// executor, since we couldn't even use it yet.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/couldn't/wouldn't

Expand Down Expand Up @@ -462,9 +463,7 @@ private[spark] class ExecutorAllocationManager(
executorIdsToBeRemoved
} else {
// We don't want to change our target number of executors, because we already did that
// when the task backlog decreased. Normally there wouldn't be any tasks running on these
// executors, but maybe the scheduler *just* decided to run a task there -- in that case,
// we don't want to count those failures.
// when the task backlog decreased.
client.killExecutors(executorIdsToBeRemoved, adjustTargetNumExecutors = false,
countFailures = false, force = false)
}
Expand Down