Skip to content
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
b34ec0c
make master support multiple executors per worker
CodingCat May 4, 2014
a5d629a
java doc
CodingCat Jan 27, 2015
a26096d
stylistic fix
CodingCat Jan 27, 2015
e5efabb
more java docs and consolidate canUse function
CodingCat Jan 27, 2015
ec7d421
test commit
CodingCat Jan 27, 2015
5b81466
remove outdated comments
CodingCat Jan 27, 2015
19d3da7
address the comments
CodingCat Feb 22, 2015
0b64fea
fix compilation issue
CodingCat Feb 22, 2015
35c462c
address Andrew's comments
CodingCat Feb 22, 2015
387f4ec
bug fix
CodingCat Feb 23, 2015
f64a28d
typo fix
CodingCat Feb 23, 2015
878402c
change the launching executor code
CodingCat Feb 23, 2015
497ec2c
address andrew's comments
CodingCat Mar 27, 2015
2c2bcc5
fix wrong usage info
CodingCat Mar 27, 2015
ff011e2
start multiple executors on the worker by rewriting startExeuctor logic
CodingCat Apr 5, 2015
4cf61f1
improve the code and docs
CodingCat Apr 5, 2015
63b3df9
change the description of the parameter in the submit script
CodingCat Apr 5, 2015
f595bd6
recover some unintentional changes
CodingCat Apr 5, 2015
d9c1685
remove unused var
CodingCat Apr 5, 2015
f035423
stylistic fix
CodingCat Apr 5, 2015
12a1b32
change the semantic of coresPerExecutor to exact core number
CodingCat Apr 9, 2015
2eeff77
stylistic fixes
CodingCat Apr 10, 2015
45967b4
remove unused method
CodingCat Apr 10, 2015
b8ca561
revert a change
CodingCat Apr 10, 2015
940cb42
avoid unnecessary allocation
CodingCat Apr 10, 2015
fbeb7e5
address the comments
CodingCat Apr 14, 2015
6dee808
change filter predicate
CodingCat Apr 14, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address the comments
  • Loading branch information
CodingCat committed Apr 9, 2015
commit 19d3da7163784722685c603cb35caeabc1a95c32
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ private[spark] class ApplicationDescription(
name, maxCores, memoryPerSlave, command, appUiUrl, eventLogDir, eventLogCodec)

// only valid when spark.executor.multiPerWorker is set to true
var maxCorePerExecutor = maxCores
var maxCorePerExecutor = None

override def toString: String = "ApplicationDescription(" + name + ")"
}
Original file line number Diff line number Diff line change
Expand Up @@ -609,7 +609,9 @@ private[master] class Master(
var leftExecutorNumToAssign = usableWorkers.map(_.memoryFree / memoryPerExecutor).sum
var maxCoresLeft = app.coresLeft
val numUsable = usableWorkers.length
// 2D array to track the number of cores of each executor assigned to each worker
// A 2D array that tracks the number of cores used by each executor launched on
// each worker. The first index refers to the usable worker, and the second index
// refers to the executor launched on that worker.
val assigned = Array.fill[ListBuffer[Int]](numUsable)(new ListBuffer[Int])
val assignedSum = Array.fill[Int](numUsable)(0)
var pos = 0
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -83,14 +83,14 @@ private[ui] class ApplicationPage(parent: MasterWebUI) extends WebUIPage("app")
<li><strong>Name:</strong> {app.desc.name}</li>
<li><strong>User:</strong> {app.desc.user}</li>
<li><strong>Cores:</strong>
{
{
if (app.desc.maxCores.isEmpty) {
"Unlimited (%s granted)".format(app.coresGranted)
} else {
"%s (%s granted, %s left)".format(
app.desc.maxCores.get, app.coresGranted, app.coresLeft)
}
}
}
</li>
<li>
<strong>Executor Memory:</strong>
Expand Down
28 changes: 9 additions & 19 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ of the most common options to set are:
Number of cores to use for the driver process, only in cluster mode.
</td>
</tr>
<tr>
<td><code>spark.driver.maxResultSize</code></td>
<td>1g</td>
<td>
Expand All @@ -127,6 +128,14 @@ of the most common options to set are:
or in your default properties file.
</td>
</tr>
<tr>
<td>spark.executor.maxCoreNumPerExecutor</td>
<td>1</td>
<td>
set the max number of cores assigned to each executor; this property is only valid when
<code>spark.executor.multiPerWorker</code> is set to true.
</td>
</tr>
<tr>
<td><code>spark.executor.memory</code></td>
<td>512m</td>
Expand Down Expand Up @@ -1255,25 +1264,6 @@ Apart from these, the following properties are also available, and may be useful
<tr>
<td><code>spark.ui.view.acls</code></td>
<td>Empty</td>
</tr>
<tr>
<td>spark.executor.multiPerWorker</td>
<td>false</td>
<td>
enable user to run multiple executors in the same worker.
</td>
</tr>
<tr>
<td>spark.executor.maxCoreNumPerExecutor</td>
<td>1</td>
<td>
set the max number of cores assigned to each executor; this property is only valid when
<code>spark.executor.multiPerWorker</code> is set to true.
</td>
</tr>
<tr>
<td>spark.executor.extraClassPath</td>
<td>(none)</td>
<td>
Comma separated list of users that have view access to the Spark web ui. By default only the
user that started the Spark job has view access.
Expand Down