Skip to content

[SPARK-4666] Improve YarnAllocator's parsing of "memory overhead" param #3525

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ryan-williams wants to merge 24 commits into from
Closed

[SPARK-4666] Improve YarnAllocator's parsing of "memory overhead" param #3525

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
3e1cc29
rename: executorMemory -> executorMemoryMB
ryan-williams Feb 10, 2015
cba802c
rename: amMemory -> amMemoryMB
ryan-williams Feb 10, 2015
0a717d7
rename: driverMemory -> driverMemoryMB
ryan-williams Feb 10, 2015
6635945
rename: DEFAULT_MEMORY -> DEFAULT_MEMORY_MB
ryan-williams Feb 10, 2015
96dfec0
rename: amMemoryOverhead -> amMemoryOverheadMB
ryan-williams Feb 10, 2015
5a3e4b8
rename: executorMemoryOverhead -> executorMemoryOverheadMB
ryan-williams Feb 10, 2015
63086eb
rename: amMemoryOverhead -> amMemoryOverheadMB
ryan-williams Feb 10, 2015
f54b0ce
rename: executorMemoryOverhead -> executorMemoryOverheadMB
ryan-williams Feb 10, 2015
fa3d69f
rename: MEMORY_OVERHEAD_MIN -> MEMORY_OVERHEAD_MIN_MB
ryan-williams Feb 10, 2015
f265d15
fix deprecation warning
ryan-williams Feb 9, 2015
c29da1d
rename: executorMemory -> executorMemoryMB
ryan-williams Feb 10, 2015
23a77be
rename: executorMemory -> executorMemoryMB
ryan-williams Feb 10, 2015
5057bd3
rename: memoryOverhead -> memoryOverheadMB
ryan-williams Feb 10, 2015
14bd3d5
rename: memory -> memoryMB
ryan-williams Feb 10, 2015
6e69b08
rename: mem -> memMB
ryan-williams Feb 10, 2015
48c5115
memoryStringToMb can have default scale specified
ryan-williams Nov 13, 2014
dc03bf2
move getMaxResultSize from Utils to SparkConf
ryan-williams Feb 10, 2015
40ac6ce
privatize amMemoryOverheadConf
ryan-williams Feb 10, 2015
dd9be85
refactor memory-size order-of-magnitude constants
ryan-williams Feb 10, 2015
bb66b22
add memory-string-parsing helpers to Utils
ryan-williams Feb 10, 2015
960b525
add `getMemory`, `getMB` helpers to SparkConf
ryan-williams Feb 10, 2015
e038867
use SparkConf.getMB helper in Yarn memory parsing
ryan-williams Feb 10, 2015
2ebb55a
update docs about YARN memory overhead parameters
ryan-williams Feb 10, 2015
50f0f52
code review feedback
ryan-williams Feb 10, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add getMemory, getMB helpers to SparkConf
  • Loading branch information
@ryan-williams
ryan-williams committed Feb 10, 2015
commit 960b525bddbf98ac7b12f41239ff6cd8ea8f2f6f
19 changes: 18 additions & 1 deletion core/src/main/scala/org/apache/spark/SparkConf.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,24 @@ class SparkConf(loadDefaults: Boolean) extends Cloneable with Logging {

// Limit of bytes for total size of results (default is 1GB)
def getMaxResultSize: Long = {
Utils.memoryStringToMb(get("spark.driver.maxResultSize", "1g")).toLong << 20
getMemory("spark.driver.maxResultSize", "1g", outputScale = 'b')
}

private def getMemory(
key: String,
defaultValue: String,
defaultInputScale: Char = 'b',
outputScale: Char = 'm'): Long = {
Utils.parseMemoryString(getOption(key).getOrElse(defaultValue), defaultInputScale, outputScale)
}

def getMB(
key: String,
defaultValue: Int): Int = {
getOption(key)
.map(Utils.memoryStringToMb(_, defaultInputScale = 'm'))
.map(_.toInt)
.getOrElse(defaultValue)
}

/** Get all executor environment variables set on this SparkConf */
Expand Down