Skip to content

Add semantic_models to tracked resource counts #8078

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
peterallenwebb merged 4 commits into from
Jul 12, 2023
Merged

Add semantic_models to tracked resource counts #8078

Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a052570
Add semantic_models to tracked resource counts
jtcohen6 Jul 12, 2023
f8041d2
Add changelog entry
jtcohen6 Jul 12, 2023
4fbf037
Simplify node statistic tabulation.
peterallenwebb Jul 12, 2023
2137a13
Remove review comment. Replace with explanation.
peterallenwebb Jul 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Simplify node statistic tabulation.
  • Loading branch information
@peterallenwebb
peterallenwebb committed Jul 12, 2023
commit 4fbf0373c073ea3796aea44c521cb4f320b1ed2d
27 changes: 9 additions & 18 deletions core/dbt/compilation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,26 +84,17 @@ def _generate_stats(manifest: Manifest):
if _node_enabled(node):
stats[node.resource_type] += 1

for source in manifest.sources.values():
stats[source.resource_type] += 1
for exposure in manifest.exposures.values():
stats[exposure.resource_type] += 1
for metric in manifest.metrics.values():
stats[metric.resource_type] += 1
for macro in manifest.macros.values():
stats[macro.resource_type] += 1
for group in manifest.groups.values():
stats[group.resource_type] += 1
# REVIEW: Why are these counted if disabled, when nodes in the general node
# collection are not?
Copy link
Contributor

@MichelleArk MichelleArk Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poked into this -- I believe it's because the disabled mapping on the manifest stores a broader set of node types (GraphMemberNode) than the general node collection. For example, disabled exposures would be omitted from manifest.exposures and instead find themselves in manifest.disabled and so would not get included in the count. It looks like the _node_enabled is really just handling NodeType.Test resources which presumably find themselves in manifest.nodes even if they are disabled.

Copy link
Contributor

@peterallenwebb peterallenwebb Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That does make sense. I'll remove the comment.

stats[NodeType.Source] += len(manifest.sources)
stats[NodeType.Exposure] += len(manifest.exposures)
stats[NodeType.Metric] += len(manifest.metrics)
stats[NodeType.Macro] += len(manifest.macros)
stats[NodeType.Group] += len(manifest.groups)
stats[NodeType.SemanticModel] += len(manifest.semantic_models)

# TODO: should we be counting dimensions + entities?
# dimensions = set()
# entities = set()
for semantic_model in manifest.semantic_models.values():
stats[semantic_model.resource_type] += 1
# for dimension in semantic_model.dimensions:
# dimensions.add(dimension.name)
# for entity in semantic_model.entities:
# entities.add(entity.name)

return stats


Expand Down