Skip to content

[SPARK-25865][CORE] Add GC information to ExecutorMetrics #22874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LantaoJin wants to merge 14 commits into from
Closed

[SPARK-25865][CORE] Add GC information to ExecutorMetrics #22874

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
6ad48d6
[SPARK-25865][CORE] Add GC information to ExecutorMetrics
LantaoJin Dec 17, 2018
b8c3382
add UT
LantaoJin Jan 2, 2019
098a246
address comments
LantaoJin Jan 2, 2019
1dbc68a
address comments
LantaoJin Jan 2, 2019
94fd962
make configurable
LantaoJin Jan 2, 2019
e4d9acd
address comments
LantaoJin Jan 2, 2019
aa3cf3c
address comments
LantaoJin Jan 7, 2019
544749d
address comments
LantaoJin Jan 8, 2019
fe5a04c
address rat check
LantaoJin Jan 10, 2019
12fc5cc
address comments
LantaoJin Jan 11, 2019
6d5643c
remove the unclear gc metric configuration
LantaoJin Jan 20, 2019
4d1021c
fix ut
LantaoJin Feb 20, 2019
ba500fe
address the comments
LantaoJin Feb 21, 2019
958a728
make log clear
LantaoJin Feb 25, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments
  • Loading branch information
@LantaoJin
LantaoJin committed Feb 19, 2019
commit e4d9acd1cc69d30abcd8ee3dff41a04a42266ddf
8 changes: 4 additions & 4 deletions core/src/main/scala/org/apache/spark/internal/config/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,16 +119,16 @@ package object config {
.booleanConf
.createWithDefault(true)

private[spark] val ADDITIONAL_YOUNG_GENERATION_GARBAGE_COLLECTORS =
ConfigBuilder("spark.additionalYoungGenerationGarbageCollectors")
private[spark] val EVENT_LOG_ADDITIONAL_YOUNG_GENERATION_GARBAGE_COLLECTORS =
ConfigBuilder("spark.eventLog.additionalYoungGenerationGarbageCollectors")
.doc("Names of additional young generation garbage collector, " +
"usually is the return of GarbageCollectorMXBean.getName, e.g. ParNew.")
.stringConf
.toSequence
.createWithDefault(Nil)

private[spark] val ADDITIONAL_OLD_GENERATION_GARBAGE_COLLECTORS =
ConfigBuilder("spark.additionalOldGenerationGarbageCollectors")
private[spark] val EVENT_LOG_ADDITIONAL_OLD_GENERATION_GARBAGE_COLLECTORS =
ConfigBuilder("spark.eventLog.additionalOldGenerationGarbageCollectors")
.doc("Names of additional old generation garbage collector, " +
"usually is the return of GarbageCollectorMXBean.getName, e.g. ConcurrentMarkSweep.")
.stringConf
Expand Down
8 changes: 5 additions & 3 deletions core/src/main/scala/org/apache/spark/metrics/ExecutorMetricType.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,12 +121,12 @@ case object GarbageCollectionMetrics extends ExecutorMetricType with Logging {

private lazy val youngGenGarbageCollector: Seq[String] = {
Seq(`copy`, `psScavenge`, `parNew`, `g1Young`) ++ /* additional young gc we added */
SparkEnv.get.conf.get(config.ADDITIONAL_YOUNG_GENERATION_GARBAGE_COLLECTORS)
SparkEnv.get.conf.get(config.EVENT_LOG_ADDITIONAL_YOUNG_GENERATION_GARBAGE_COLLECTORS)
}

private lazy val oldGenGarbageCollector: Seq[String] = {
Seq(`markSweepCompact`, `psMarkSweep`, `cms`, `g1Old`) ++ /* additional old gc we added */
SparkEnv.get.conf.get(config.ADDITIONAL_OLD_GENERATION_GARBAGE_COLLECTORS)
SparkEnv.get.conf.get(config.EVENT_LOG_ADDITIONAL_OLD_GENERATION_GARBAGE_COLLECTORS)
}

override private[spark] def getMetricValues(memoryManager: MemoryManager): Array[Long] = {
Expand All @@ -140,7 +140,9 @@ case object GarbageCollectionMetrics extends ExecutorMetricType with Logging {
gcMetrics(2) = mxBean.getCollectionCount
gcMetrics(3) = mxBean.getCollectionTime
} else {
logDebug(s"${mxBean.getName} is an unsupported garbage collector.")
logDebug(s"${mxBean.getName} is an unsupported garbage collector." +
s"Add it to ${config.EVENT_LOG_ADDITIONAL_YOUNG_GENERATION_GARBAGE_COLLECTORS} " +
s"or ${config.EVENT_LOG_ADDITIONAL_OLD_GENERATION_GARBAGE_COLLECTORS} to enable.")
}
}
}
Expand Down