[SPARK-6891] Fix the bug that ExecutorAllocationManager will request negative number executors

[ArcherShao]

In ExecutorAllocationManager, executor allocate schedule at a fix rate(100ms), it will call the method 'addOrCancelExecutorRequests' first, and then remove expired excutors.
Suppose at time T, no task is running or pending, and there a 5 executors runing, but all expired.

1. the method 'addOrCancelExecutorRequests' wiill be called, and the value of 'ExecutorAllocationManager.numExecutorsPending' will update to -5.
2. remove 5 expired excutors.
   Suppose still no task is running or pending at T+1, the method 'targetNumExecutors' will return -5, and method 'addExecutors' will be called,

private def addExecutors(maxNumExecutorsNeeded: Int): Int = {
val currentTarget = targetNumExecutors
....
val actualMaxNumExecutors = math.min(maxNumExecutors, maxNumExecutorsNeeded)
val newTotalExecutors = math.min(currentTarget + numExecutorsToAdd, actualMaxNumExecutors)
val addRequestAcknowledged = testing || client.requestTotalExecutors(newTotalExecutors)
....
}

newTotalExecutors will be a negative number, when client.requestTotalExecutors(newTotalExecutors) called, it will throw an exception.

Let method 'targetNumExecutors' return a value not less than minNumExecutors, then the newTotalExecutors will never be negative.

And targetNumExecutors not less than minNumExecutors is also make sense.

[SparkQA]

Test build #30908 has finished for PR 5676 at commit 1693b54.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[sryza]

This looks like a duplicate of SPARK-6954 (PR #5536)

[ArcherShao]

@sryza Should I close this PR?

[srowen]

@ArcherShao yes please, the JIRA was already marked as a duplicate. https://issues.apache.org/jira/browse/SPARK-6891. Do you mind closing this PR?