Skip to content

Query/Ruler: Add metric for number of endpoints resolved for each configured address.#1260

Merged
bwplotka merged 1 commit intomasterfrom
thanos-dns-addrs-metric
Jun 18, 2019
Merged

Query/Ruler: Add metric for number of endpoints resolved for each configured address.#1260
bwplotka merged 1 commit intomasterfrom
thanos-dns-addrs-metric

Conversation

@devnev
Copy link
Contributor

@devnev devnev commented Jun 18, 2019

Changes

Add thanos_querier_store_apis_dns_provider_results / thanos_ruler_query_apis_dns_provider_results metrics.

count(dns_provider_results) tells us how many addresses have been
configured through flags and file-SD, while sum(dns_provider_results)
gives us the number of endpoints actually produced from these addresses.

Verification

Ran thanos query --store=localhost:10901, navigated to localhost:10902/metrics and found the entry:

# HELP thanos_querier_store_apis_dns_provider_results The number of resolved endpoints for each configured address
# TYPE thanos_querier_store_apis_dns_provider_results gauge
thanos_querier_store_apis_dns_provider_results{addr="localhost:10901"} 1

`count(dns_provider_results)` tells us how many addresses have been
configured through flags and file-SD, while `sum(dns_provider_results)`
gives us the number of endpoints actually produced from these addresses.
p.resolverLookupsCount.Inc()
if err != nil {
// The DNS resolution failed. Continue without modifying the old records.
p.resolved[addr] = p.resolved[addr] // Ensure metrics capture the result even if empty.
Copy link
Member

@bwplotka bwplotka Jun 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really get this part is, it intentional? Isn't this noop?
It's tricky what should be there TBH ;p Should we set results for both query to empty and metrics to 0? should we only adjust metrics? Not sure yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ensures that the map contains an entry for the address. So when we iterate over the map keys for the metric, we report that there was an address configured regardless of whether we got some results here, got no results, or continue using previous results.

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, except one thing, commented below.

Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Thanks!

@bwplotka bwplotka merged commit c7da9f2 into master Jun 18, 2019
@brancz brancz deleted the thanos-dns-addrs-metric branch June 18, 2019 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants