Skip to content

Commit 5767598

Browse files
HeartSaVioRkai-chi
authored andcommitted
[SPARK-26082][MESOS][FOLLOWUP] Add UT on fetcher cache option on MesosClusterScheduler
## What changes were proposed in this pull request? This patch adds UT on testing SPARK-26082 to avoid regression. While apache#23743 reduces the possibility to make a similar mistake, the needed lines of code for adding tests are not that huge, so I guess it might be worth to add them. ## How was this patch tested? Newly added UTs. Test "supports setting fetcher cache" fails when apache#23743 is not applied and succeeds when apache#23743 is applied. Closes apache#23744 from HeartSaVioR/SPARK-26082-add-unit-test. Authored-by: Jungtaek Lim (HeartSaVioR) <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]> (cherry picked from commit b4e1d14) Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 2b07c1c commit 5767598

File tree

1 file changed

+48
-0
lines changed

1 file changed

+48
-0
lines changed

resource-managers/mesos/src/test/scala/org/apache/spark/scheduler/cluster/mesos/MesosClusterSchedulerSuite.scala

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -254,6 +254,54 @@ class MesosClusterSchedulerSuite extends SparkFunSuite with LocalSparkContext wi
254254
assert(networkInfos.get(0).getLabels.getLabels(1).getValue == "val2")
255255
}
256256

257+
test("supports setting fetcher cache") {
258+
setScheduler(Map("spark.mesos.fetcherCache.enable" -> "true"))
259+
260+
val mem = 1000
261+
val cpu = 1
262+
263+
val response = scheduler.submitDriver(
264+
new MesosDriverDescription("d1", "jar", mem, cpu, true,
265+
command,
266+
Map("spark.mesos.executor.home" -> "test",
267+
"spark.app.name" -> "test"),
268+
"s1",
269+
new Date()))
270+
271+
assert(response.success)
272+
273+
val offer = Utils.createOffer("o1", "s1", mem, cpu)
274+
scheduler.resourceOffers(driver, List(offer).asJava)
275+
276+
val launchedTasks = Utils.verifyTaskLaunched(driver, "o1")
277+
val uris = launchedTasks.head.getCommand.getUrisList
278+
assert(uris.stream().allMatch(_.getCache))
279+
}
280+
281+
test("supports disabling fetcher cache") {
282+
setScheduler(Map("spark.mesos.fetcherCache.enable" -> "false"))
283+
284+
val mem = 1000
285+
val cpu = 1
286+
287+
val response = scheduler.submitDriver(
288+
new MesosDriverDescription("d1", "jar", mem, cpu, true,
289+
command,
290+
Map("spark.mesos.executor.home" -> "test",
291+
"spark.app.name" -> "test"),
292+
"s1",
293+
new Date()))
294+
295+
assert(response.success)
296+
297+
val offer = Utils.createOffer("o1", "s1", mem, cpu)
298+
scheduler.resourceOffers(driver, List(offer).asJava)
299+
300+
val launchedTasks = Utils.verifyTaskLaunched(driver, "o1")
301+
val uris = launchedTasks.head.getCommand.getUrisList
302+
assert(uris.stream().allMatch(!_.getCache))
303+
}
304+
257305
test("accept/decline offers with driver constraints") {
258306
setScheduler()
259307

0 commit comments

Comments
 (0)