[SPARK-25228][CORE]Add executor CPU time metric. #22218
Add executor CPU Time metric (jvmCpuTime) using ProcessCpuTime method…
… in OperatingSystem MX Bean, if available.
- Loading branch information
commit 95d31f69cac02af4eee63f8a1a364c485190951c
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -82,9 +82,9 @@ class ExecutorSource(threadPool: ThreadPoolExecutor, executorId: String) extends | |
| // It will use proprietary extensions such as com.sun.management.OperatingSystemMXBean or | ||
| // com.ibm.lang.management.OperatingSystemMXBean, if available. | ||
| metricRegistry.register(MetricRegistry.name("jvmCpuTime"), new Gauge[Long] { | ||
| val mBean: MBeanServer = ManagementFactory.getPlatformMBeanServer | ||
| val name = new ObjectName("java.lang", "type", "OperatingSystem") | ||
| override def getValue: Long = { | ||
| try { | ||
| val attribute = mBean.getAttribute(name, "ProcessCpuTime") | ||
| if (attribute != null) { | ||
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this isn't exposed except through dropwizard... not plumbed through to the driver too like some of the metrics below? just checking that this is all that needs to happen, that the metric can be used by external users but is not otherwise touched by Spark.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed, this is exposed only through dropwizard metrics system and not used otherwise in the Spark code. Another point worth mentioning is that currently executorSource is not registered when running in local mode.
On a related topic (although maybe for a more general discussion than the scope of this PR) I was wondering if it would make sense to introduce a few SparkConf properties to switch on/off certain families of (dropwizard) metrics in the Spark, as the list of available metrics is mecoming long in recent versions.