Skip to content

[SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend #20640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
IgorBerman wants to merge 14 commits into from
Closed

[SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend #20640

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
636959a
Removed hardcoded blacklist functionality, must be controled by Black…
antiout Apr 12, 2017
f09faf7
Removed hardcoded blacklist functionality, must be controled by Black…
antiout Apr 12, 2017
e2ddc1b
SPARK-19755 declining offers from blacklisted slave by BlacklistTracker
IgorBerman Feb 20, 2018
66ed5af
[SPARK-19755][Mesos] reverting logging on mesos slave task(executor) …
IgorBerman Mar 21, 2018
a7ff8cc
[SPARK-19755][Mesos] rewording log message and changing it to error l…
IgorBerman Mar 27, 2018
2c47271
[SPARK-19755][Mesos] adding comment regarding failures of mesos task …
IgorBerman Jun 15, 2018
5eda874
[SPARK-19755][Mesos] specifying that it's mesos task failing
IgorBerman Jun 15, 2018
104d44f
Merge branch 'master' into SPARK-19755-Blacklist-is-always-active-for…
IgorBerman Sep 25, 2019
95ca22c
SPARK-19755 fixing merge
IgorBerman Sep 25, 2019
cb8cb57
SPARK-19755 adding jira to testcase name
IgorBerman Sep 25, 2019
83cabff
SPARK-19755 removing duplicated line
IgorBerman Sep 25, 2019
2fc0288
alternative
dongjoon-hyun Sep 25, 2019
04f931d
Recover comments
dongjoon-hyun Sep 25, 2019
dbe7d18
Merge pull request #1 from dongjoon-hyun/PR-20640-2
IgorBerman Oct 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
SPARK-19755 declining offers from blacklisted slave by BlacklistTracker
  • Loading branch information
@IgorBerman
IgorBerman committed Feb 21, 2018
commit e2ddc1be19e2f978df4fe84073aff3f5b46afe45
3 changes: 2 additions & 1 deletion ...n/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,7 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
cpus + totalCoresAcquired <= maxCores &&
mem <= offerMem &&
numExecutors < executorLimit &&
!scheduler.nodeBlacklist().contains(offerHostname) &&
Copy link
Contributor

@squito squito Apr 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want to make really sure everybody understands the big change in behavior here -- nodeBlacklist() currently only gets updated based on failures in spark tasks. If a mesos task fails to even start -- that is, if a spark executor fails to launch on a node -- nodeBlacklist does not get updated. So you could have a node that is misconfigured somehow, and you might end up repeatedly trying to launch executors on it after this changed, with the executor even failing to start each time. That is even if you have blacklisting on.

This is SPARK-16630 for the non-mesos case. That is being actively worked on now -- however the work there will probably have to be yarn-specific, so there will still be followup work to get the same thing for mesos after that is in.

Copy link
Contributor

@skonto skonto May 19, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@squito sounds reasonable. In the mean time we have to deal with a limitation at the mesos side where the value is hardcoded. So we can move with this incrementally.

Copy link
Member

@felixcheung felixcheung May 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe comment on this in the code here and add a JIRA for tracking?

Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checking looks a little late. Can we decline more faster without calculating everything?

meetsPortRequirements &&
satisfiesLocality(offerHostname)
}
Expand Down Expand Up @@ -792,4 +793,4 @@ object IdHelper {
// Use atomic values since Spark contexts can be initialized in parallel
private[mesos] val nextSCNumber = new AtomicLong(0)
private[mesos] val startedBefore = new AtomicBoolean(false)
}
}
23 changes: 23 additions & 0 deletions ...la/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackendSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,28 @@ class MesosCoarseGrainedSchedulerBackendSuite extends SparkFunSuite
verifyTaskLaunched(driver, "o2")
}

test("mesos declines offers from blacklisted slave") {
setBackend()

// launches a task on a valid offer on slave s1
val minMem = backend.executorMemory(sc) + 1024
val minCpu = 4
val offer1 = Resources(minMem, minCpu)
offerResources(List(offer1))
verifyTaskLaunched(driver, "o1")

// for any reason executor(aka mesos task) failed on s1
val status = createTaskStatus("0", "s1", TaskState.TASK_FAILED)
backend.statusUpdate(driver, status)
when(taskScheduler.nodeBlacklist()).thenReturn(Set("hosts1"))
Copy link
Contributor

@squito squito Apr 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to re-iterate my point above -- in many cases, having an executor fail will not lead to taskScheduler.nodeBlacklist() changing as you're doing here.


val offer2 = Resources(minMem, minCpu)
// Offer resources from the same slave
offerResources(List(offer2))
// but since it's blacklisted the offer is declined
verifyDeclinedOffer(driver, createOfferId("o1"))
Copy link
Contributor

@squito squito Feb 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this actually pass? I thought it wouldn't b/c the filtering is done inside buildMesosTasks, which never calls declineOffer on offers that fail canLaunchTask. (a separate thing which needs fixing -- you could open another issue for that.)

Copy link
Contributor

@squito squito Feb 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah nevermind. I took another look at the code and now I see how this works

}

test("mesos supports spark.executor.cores") {
val executorCores = 4
setBackend(Map("spark.executor.cores" -> executorCores.toString))
Expand Down Expand Up @@ -790,6 +812,7 @@ class MesosCoarseGrainedSchedulerBackendSuite extends SparkFunSuite

taskScheduler = mock[TaskSchedulerImpl]
when(taskScheduler.sc).thenReturn(sc)
when(taskScheduler.nodeBlacklist()).thenReturn(Set[String]())

externalShuffleClient = mock[MesosExternalShuffleClient]

Expand Down