Skip to content

[SPARK-25449][CORE] Heartbeat shouldn't include accumulators for zero metrics #22473

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mukulmurthy wants to merge 9 commits into from
Closed

[SPARK-25449][CORE] Heartbeat shouldn't include accumulators for zero metrics #22473

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3e0d953
Don't send zero accumulators for metrics in heartbeat
mukulmurthy Sep 17, 2018
3cf88a4
add tests
mukulmurthy Sep 19, 2018
fc98256
Refactor heartbeat configs and gate dropping zero metrics behind a co…
mukulmurthy Sep 19, 2018
6bb91b0
update tests
mukulmurthy Sep 19, 2018
7d0c39b
refactor code per Ryan's review
mukulmurthy Sep 26, 2018
db14bd1
minor refactor
mukulmurthy Sep 27, 2018
e22ac78
use PrivateMethodTester
mukulmurthy Sep 27, 2018
9cbe14c
Use TestMemoryManager for test to not have to mock object field
mukulmurthy Sep 28, 2018
f6fa337
Use getTimeAsSeconds
mukulmurthy Sep 28, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Don't send zero accumulators for metrics in heartbeat
  • Loading branch information
@mukulmurthy
mukulmurthy committed Sep 17, 2018
commit 3e0d9536512300d27201e1d5cc4d9b5755a47871
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/executor/Executor.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -799,7 +799,8 @@ private[spark] class Executor(
if (taskRunner.task != null) {
taskRunner.task.metrics.mergeShuffleReadMetrics()
taskRunner.task.metrics.setJvmGCTime(curGCTime - taskRunner.startGCTime)
accumUpdates += ((taskRunner.taskId, taskRunner.task.metrics.accumulators()))
accumUpdates +=
((taskRunner.taskId, taskRunner.task.metrics.accumulators().filterNot(_.isZero)))
Copy link
Member

@zsxwing zsxwing Sep 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a flag for this behavior change?

}
}

Expand Down