Skip to content

Add a note about jobs running in FIFO order in the default pool #20881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Alexis-D wants to merge 2 commits into from
Closed

Add a note about jobs running in FIFO order in the default pool #20881

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d0701bc
Add a note about jobs running in FIFO order in the default pool
Alexis-D Mar 22, 2018
c6b7675
fix typos
Alexis-D Apr 3, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/job-scheduling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,9 @@ pool), but inside each pool, jobs run in FIFO order. For example, if you create
means that each user will get an equal share of the cluster, and that each user's queries will run in
order instead of later queries taking resources from that user's earlier ones.

If jobs are not explicitly set to use a given pool, they end up in the default pool. This means that even if
`spark.scheduler.mode` is set to `FAIR` those jobs will be run in `FIFO` order (within the default pool).

Copy link
Contributor

@markhamstra markhamstra Apr 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not actually correct. There is no reason why you can't define a default pool that uses FAIR scheduling.

Copy link
Author

@Alexis-D Alexis-D Apr 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume you mean that the second sentence is incorrect? I drew that conclusion based from empirical observations +

spark/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala

Lines 109 to 117 in 992447f

private def buildDefaultPool() {
if (rootPool.getSchedulableByName(DEFAULT_POOL_NAME) == null) {
val pool = new Pool(DEFAULT_POOL_NAME, DEFAULT_SCHEDULING_MODE,
DEFAULT_MINIMUM_SHARE, DEFAULT_WEIGHT)
rootPool.addSchedulable(pool)
logInfo("Created default pool: %s, schedulingMode: %s, minShare: %d, weight: %d".format(
DEFAULT_POOL_NAME, DEFAULT_SCHEDULING_MODE, DEFAULT_MINIMUM_SHARE, DEFAULT_WEIGHT))
}
}

However, I might very well be missing something?

Copy link
Contributor

@markhamstra markhamstra Apr 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You seem to be missing a few somethings: 1) You can define your own default pool that does FAIR scheduling within that pool, so blanket statements about "the" default pool are dangerous; 2) spark.scheduler.mode controls the setup of the rootPool, not the scheduling within any pool; 3) If you don't define your own pool with a name corresponding to the DEFAULT_POOL_NAME (i.e. "default"), then you are going to get a default construction of "default", which does use FIFO scheduling within that pool.

So, item 2) effectively means that spark.scheduler.mode controls whether fair scheduling is possible at all, and it also defines the kind of scheduling that is used among the shedulable entities contained in the root pool -- i.e. among the scheduling pools nested within rootPool. One of those nested pools will be DEFAULT_POOL_NAME/"default", which will use FIFO scheduling for schedulable entities within that pool if you haven't defined it to use fair scheduling.

If you just want one scheduling pool that does fair scheduling among its schedulable entities, then you need to set spark.scheduler.mode to "FAIR" in your SparkConf and also define in the pool configuration file a "default" pool to use schedulingMode FAIR. You could alternatively define such a fair-scheduling-inside pool named something other than "default" and then make sure that all of your jobs get assigned to that pool.

Copy link
Author

@Alexis-D Alexis-D Apr 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, thanks @markhamstra I think I grasp what's going on now. Some form of your comment would be a useful addition to the documentation; rationale being that there seems to be a (common?) misunderstanding about how to schedule jobs in a FAIR way, e.g. https://stackoverflow.com/a/37882686/2813687, or myself trying to do this leading to this very PR. After reading your comment, the current documentation makes sense, and obviously this PR is incorrect (at the very least it doesn't underscore all the caveats/config knobs at play here). I'll take another look at improving the doc such that the actual behavior is obvious to Spark users not familiar with Spark scheduling nitty gritty who merely want to run a few jobs concurrently.

## Configuring Pool Properties

Specific pools' properties can also be modified through a configuration file. Each pool supports three
Expand Down