Skip to content

[SPARK-4666] Improve YarnAllocator's parsing of "memory overhead" param #3525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ryan-williams wants to merge 24 commits into from
Closed

[SPARK-4666] Improve YarnAllocator's parsing of "memory overhead" param #3525

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3e1cc29
rename: executorMemory -> executorMemoryMB
ryan-williams Feb 10, 2015
cba802c
rename: amMemory -> amMemoryMB
ryan-williams Feb 10, 2015
0a717d7
rename: driverMemory -> driverMemoryMB
ryan-williams Feb 10, 2015
6635945
rename: DEFAULT_MEMORY -> DEFAULT_MEMORY_MB
ryan-williams Feb 10, 2015
96dfec0
rename: amMemoryOverhead -> amMemoryOverheadMB
ryan-williams Feb 10, 2015
5a3e4b8
rename: executorMemoryOverhead -> executorMemoryOverheadMB
ryan-williams Feb 10, 2015
63086eb
rename: amMemoryOverhead -> amMemoryOverheadMB
ryan-williams Feb 10, 2015
f54b0ce
rename: executorMemoryOverhead -> executorMemoryOverheadMB
ryan-williams Feb 10, 2015
fa3d69f
rename: MEMORY_OVERHEAD_MIN -> MEMORY_OVERHEAD_MIN_MB
ryan-williams Feb 10, 2015
f265d15
fix deprecation warning
ryan-williams Feb 9, 2015
c29da1d
rename: executorMemory -> executorMemoryMB
ryan-williams Feb 10, 2015
23a77be
rename: executorMemory -> executorMemoryMB
ryan-williams Feb 10, 2015
5057bd3
rename: memoryOverhead -> memoryOverheadMB
ryan-williams Feb 10, 2015
14bd3d5
rename: memory -> memoryMB
ryan-williams Feb 10, 2015
6e69b08
rename: mem -> memMB
ryan-williams Feb 10, 2015
48c5115
memoryStringToMb can have default scale specified
ryan-williams Nov 13, 2014
dc03bf2
move getMaxResultSize from Utils to SparkConf
ryan-williams Feb 10, 2015
40ac6ce
privatize amMemoryOverheadConf
ryan-williams Feb 10, 2015
dd9be85
refactor memory-size order-of-magnitude constants
ryan-williams Feb 10, 2015
bb66b22
add memory-string-parsing helpers to Utils
ryan-williams Feb 10, 2015
960b525
add `getMemory`, `getMB` helpers to SparkConf
ryan-williams Feb 10, 2015
e038867
use SparkConf.getMB helper in Yarn memory parsing
ryan-williams Feb 10, 2015
2ebb55a
update docs about YARN memory overhead parameters
ryan-williams Feb 10, 2015
50f0f52
code review feedback
ryan-williams Feb 10, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rename: driverMemory -> driverMemoryMB
  • Loading branch information
@ryan-williams
ryan-williams committed Feb 10, 2015
commit 0a717d7d20d7339fe20dae0f85019b140597ece9
6 changes: 3 additions & 3 deletions yarn/src/main/scala/org/apache/spark/deploy/yarn/ClientArguments.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
var priority = 0
def isClusterMode: Boolean = userClass != null

private var driverMemory: Int = 512 // MB
private var driverMemoryMB: Int = 512
private var driverCores: Int = 1
private val driverMemOverheadKey = "spark.yarn.driver.memoryOverhead"
private val amMemKey = "spark.yarn.am.memory"
Expand Down Expand Up @@ -116,7 +116,7 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
println(s"$key is set but does not apply in cluster mode.")
}
}
amMemoryMB = driverMemory
amMemoryMB = driverMemoryMB
amCores = driverCores
} else {
for (key <- Seq(driverMemOverheadKey, driverCoresKey)) {
Expand Down Expand Up @@ -165,7 +165,7 @@ private[spark] class ClientArguments(args: Array[String], sparkConf: SparkConf)
if (args(0) == "--master-memory") {
println("--master-memory is deprecated. Use --driver-memory instead.")
}
driverMemory = value
driverMemoryMB = value
args = tail

case ("--driver-cores") :: IntParam(value) :: tail =>
Expand Down