Skip to content

[SPARK-25170][DOC] Add list and short description of Spark Executor Task Metrics to the documentation. #22397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
LucaCanali wants to merge 2 commits into from
Closed

[SPARK-25170][DOC] Add list and short description of Spark Executor Task Metrics to the documentation. #22397

LucaCanali wants to merge 2 commits into from

Conversation

@LucaCanali
Copy link
Contributor

@LucaCanali LucaCanali commented Sep 11, 2018

What changes were proposed in this pull request?

Add description of Executor Task Metrics to the documentation.

@LucaCanali LucaCanali mentioned this pull request Sep 11, 2018
Closed
@kiszk
Copy link
Member

kiszk commented Sep 12, 2018

LGTM cc @jiangxb1987 @srowen

srowen
srowen approved these changes Sep 12, 2018
View reviewed changes
Copy link
Member

@srowen srowen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs looks OK. I guess these weren't documented before? I spot-checked the units on a few metrics and they're right. I guess it's a little unfortunate there are some ns and ms times in here, but not a problem.

@LucaCanali
Copy link
Contributor Author

LucaCanali commented Sep 12, 2018

Thanks @srowen for reviewing this. The metrics are commented in the source code of TaskMetrics class, I took most of the descriptions from there, adding some additional explanations where needed. Indeed I agree that the fact that time units are not uniform is a bit incovenient.

@SparkQA
Copy link

SparkQA commented Sep 13, 2018

Test build #4340 has finished for PR 22397 at commit 27130ba.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

asfgit pushed a commit that referenced this pull request Sep 13, 2018
@LucaCanali @srowen
[SPARK-25170][DOC] Add list and short description of Spark Executor T…
cc19f42 
…ask Metrics to the documentation.

## What changes were proposed in this pull request?

Add description of Executor Task Metrics to the documentation.

Closes #22397 from LucaCanali/docMonitoringTaskMetrics.

Authored-by: LucaCanali <[email protected]>
Signed-off-by: Sean Owen <[email protected]>
(cherry picked from commit 45c4ebc)
Signed-off-by: Sean Owen <[email protected]>
@srowen
Copy link
Member

srowen commented Sep 13, 2018

Merged to master/2.4

@asfgit asfgit closed this in 45c4ebc Sep 13, 2018
@kiszk
Copy link
Member

kiszk commented Sep 13, 2018

Yes, time unit looks confusing, but not a problem. In general, due to APIs, elapsed time is ms, and CPU time is ns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@srowen srowen srowen approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@LucaCanali @kiszk @SparkQA @srowen