Add a note about jobs running in FIFO order in the default pool

[Alexis-D]

What changes were proposed in this pull request?

Make it clear in the doc that setting spark.scheduler.mode to FAIR isn't enough to get jobs to run in a FAIR fashion if the default pool is used.

[AmplabJenkins]

Can one of the admins verify this patch?

[sujithjay]

Hi @Alexis-D , there are a few minor typos here;
'explicitely' -> 'explicitly'.
'ran' -> 'run'

[Alexis-D]

right my bad -- I updated the PR

[markhamstra]

This is not actually correct. There is no reason why you can't define a default pool that uses FAIR scheduling.

[Alexis-D]

I assume you mean that the second sentence is incorrect? I drew that conclusion based from empirical observations +

spark/core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala

Lines 109 to 117 in 992447f

	private def buildDefaultPool() {
	if (rootPool.getSchedulableByName(DEFAULT_POOL_NAME) == null) {
	val pool = new Pool(DEFAULT_POOL_NAME, DEFAULT_SCHEDULING_MODE,
	DEFAULT_MINIMUM_SHARE, DEFAULT_WEIGHT)
	rootPool.addSchedulable(pool)
	logInfo("Created default pool: %s, schedulingMode: %s, minShare: %d, weight: %d".format(
	DEFAULT_POOL_NAME, DEFAULT_SCHEDULING_MODE, DEFAULT_MINIMUM_SHARE, DEFAULT_WEIGHT))
	}
	}

However, I might very well be missing something?

[markhamstra]

You seem to be missing a few somethings: 1) You can define your own default pool that does FAIR scheduling within that pool, so blanket statements about "the" default pool are dangerous; 2) spark.scheduler.mode controls the setup of the rootPool, not the scheduling within any pool; 3) If you don't define your own pool with a name corresponding to the DEFAULT_POOL_NAME (i.e. "default"), then you are going to get a default construction of "default", which does use FIFO scheduling within that pool.

So, item 2) effectively means that spark.scheduler.mode controls whether fair scheduling is possible at all, and it also defines the kind of scheduling that is used among the shedulable entities contained in the root pool -- i.e. among the scheduling pools nested within rootPool. One of those nested pools will be DEFAULT_POOL_NAME/"default", which will use FIFO scheduling for schedulable entities within that pool if you haven't defined it to use fair scheduling.

If you just want one scheduling pool that does fair scheduling among its schedulable entities, then you need to set spark.scheduler.mode to "FAIR" in your SparkConf and also define in the pool configuration file a "default" pool to use schedulingMode FAIR. You could alternatively define such a fair-scheduling-inside pool named something other than "default" and then make sure that all of your jobs get assigned to that pool.

[Alexis-D]

Cool, thanks @markhamstra I think I grasp what's going on now. Some form of your comment would be a useful addition to the documentation; rationale being that there seems to be a (common?) misunderstanding about how to schedule jobs in a FAIR way, e.g. https://stackoverflow.com/a/37882686/2813687, or myself trying to do this leading to this very PR. After reading your comment, the current documentation makes sense, and obviously this PR is incorrect (at the very least it doesn't underscore all the caveats/config knobs at play here). I'll take another look at improving the doc such that the actual behavior is obvious to Spark users not familiar with Spark scheduling nitty gritty who merely want to run a few jobs concurrently.