Skip to content

[SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend #20640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
IgorBerman wants to merge 14 commits into from
+36 −16
Closed

[SPARK-19755][Mesos] Blacklist is always active for MesosCoarseGrainedSchedulerBackend #20640

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
636959a
Removed hardcoded blacklist functionality, must be controled by Black…
antiout Apr 12, 2017
f09faf7
Removed hardcoded blacklist functionality, must be controled by Black…
antiout Apr 12, 2017
e2ddc1b
SPARK-19755 declining offers from blacklisted slave by BlacklistTracker
IgorBerman Feb 20, 2018
66ed5af
[SPARK-19755][Mesos] reverting logging on mesos slave task(executor) …
IgorBerman Mar 21, 2018
a7ff8cc
[SPARK-19755][Mesos] rewording log message and changing it to error l…
IgorBerman Mar 27, 2018
2c47271
[SPARK-19755][Mesos] adding comment regarding failures of mesos task …
IgorBerman Jun 15, 2018
5eda874
[SPARK-19755][Mesos] specifying that it's mesos task failing
IgorBerman Jun 15, 2018
104d44f
Merge branch 'master' into SPARK-19755-Blacklist-is-always-active-for…
IgorBerman Sep 25, 2019
95ca22c
SPARK-19755 fixing merge
IgorBerman Sep 25, 2019
cb8cb57
SPARK-19755 adding jira to testcase name
IgorBerman Sep 25, 2019
83cabff
SPARK-19755 removing duplicated line
IgorBerman Sep 25, 2019
2fc0288
alternative
dongjoon-hyun Sep 25, 2019
04f931d
Recover comments
dongjoon-hyun Sep 25, 2019
dbe7d18
Merge pull request #1 from dongjoon-hyun/PR-20640-2
IgorBerman Oct 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[SPARK-19755][Mesos] adding comment regarding failures of mesos task … 
…failures and linking to relevant jira
  • Loading branch information
@IgorBerman
IgorBerman committed Jun 15, 2018
commit 2c47271176b82e4859667ede9bb02b28b8fba50e
4 changes: 4 additions & 0 deletions ...n/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -568,6 +568,10 @@ private[spark] class MesosCoarseGrainedSchedulerBackend(
cpus + totalCoresAcquired <= maxCores &&
mem <= offerMem &&
numExecutors < executorLimit &&
// nodeBlacklist() currently only gets updated based on failures in spark tasks.
// If a mesos task fails to even start -- that is,
// if a spark executor fails to launch on a node -- nodeBlacklist does not get updated
// see SPARK-24567 for details
!scheduler.nodeBlacklist().contains(offerHostname) &&
Copy link
Contributor

@squito squito Apr 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want to make really sure everybody understands the big change in behavior here -- nodeBlacklist() currently only gets updated based on failures in spark tasks. If a mesos task fails to even start -- that is, if a spark executor fails to launch on a node -- nodeBlacklist does not get updated. So you could have a node that is misconfigured somehow, and you might end up repeatedly trying to launch executors on it after this changed, with the executor even failing to start each time. That is even if you have blacklisting on.

This is SPARK-16630 for the non-mesos case. That is being actively worked on now -- however the work there will probably have to be yarn-specific, so there will still be followup work to get the same thing for mesos after that is in.

Copy link
Contributor

@skonto skonto May 19, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@squito sounds reasonable. In the mean time we have to deal with a limitation at the mesos side where the value is hardcoded. So we can move with this incrementally.

Copy link
Member

@felixcheung felixcheung May 28, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe comment on this in the code here and add a JIRA for tracking?

Copy link
Member

@dongjoon-hyun dongjoon-hyun Sep 25, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checking looks a little late. Can we decline more faster without calculating everything?

meetsPortRequirements &&
satisfiesLocality(offerHostname)
Expand Down