Skip to content

[WIP]SPARK-1706: Allow multiple executors per worker in Standalone mode #636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
CodingCat wants to merge 10 commits into from
Closed

[WIP]SPARK-1706: Allow multiple executors per worker in Standalone mode #636

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
deeaa89
make master support multiple executors per worker
CodingCat May 4, 2014
1d17949
doc update
CodingCat May 4, 2014
830ae11
bug fix
CodingCat May 6, 2014
874ec7a
application specific corePerExecutor
CodingCat May 7, 2014
f0852cc
bug fix
CodingCat May 8, 2014
1be60da
style fix
CodingCat May 8, 2014
072dd61
bug fix
CodingCat May 9, 2014
f936a42
leftCoreToAssign should be limited by mem also
CodingCat May 9, 2014
5b21f87
capture noEnoughMemWorker every iteration
CodingCat May 9, 2014
ca70401
limit leftCoreToAssign with Mem
CodingCat May 9, 2014
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
leftCoreToAssign should be limited by mem also
  • Loading branch information
@CodingCat
CodingCat committed May 9, 2014
commit f936a4202be42ac6dc8a971b00366db8c2399c85
4 changes: 3 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/master/Master.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -515,7 +515,9 @@ private[spark] class Master(
app.desc.memoryPerExecutor).sortBy(_.coresFree).reverse
val maxCoreNumPerExecutor = app.desc.maxCorePerExecutor.get
var mostFreeCoreWorkerPos = 0
var leftCoreToAssign = math.min(app.coresLeft, usableWorkers.map(_.coresFree).sum)
val maxAvailableSlots = usableWorkers.map(worker => math.min(worker.coresFree,
worker.memoryFree / app.desc.memoryPerExecutor)).sum
var leftCoreToAssign = math.min(app.coresLeft, maxAvailableSlots)
val numUsable = usableWorkers.length
// Number of cores of each executor assigned to each worker
val assigned = Array.fill[ListBuffer[Int]](numUsable)(new ListBuffer[Int])
Expand Down