query: invalid rate/irate with deduplication for most recent values

[creker]

Thanos, Prometheus and Golang version used:
bitnami/thanos:0.13.0
prom/prometheus:v2.19.0

Prometheus in HA configuration, 2 instances. Single Thanos querier instance.

Object Storage Provider:
On premise minio deployment

What happened:
Executing rate or irate with deduplication sometimes results in the most recent value being invalid. Either it shoots up to very high or very low value. Turning off deduplication produces correct results from both Prometheus instances.

With deduplication, good result

With deduplication, bad result. Executing the same query eventually gives incorrect result like this

Same query without deduplication always produces correct result

What you expected to happen:
Query with deduplication always giving correct result like this

How to reproduce it (as minimally and precisely as possible):
Prometheus in HA, any rate or irate query.

Full logs to relevant components:
Nothing that would correlate with queries.

[bwplotka]

Thanks for reporting. We thought we found all those kind of issues, but there might be more. The last one was supposed to be fixed in 0.13.0. Can you double check if you run Thanos with the fix from #2401 included? (:

[bwplotka]

Note that it's Querier version that matters for deduplication

[creker]

Here's querier version report

thanos, version 0.13.0 (branch: HEAD, revision: adf6facb8d6bf44097aae084ec091ac3febd9eb8)
  build user:       root@ee9c796b3048
  build date:       20200622-09:49:32
  go version:       go1.14.2

[creker]

I can try 0.14.0 if it contains any relevant fixes

[bwplotka]

Please update but I think it should have the fix. If 0.14 won't work, then what would be awesome is to have exact chunks for that problematic period. You can obtain those by using following script: https://github.com/thanos-io/thanos/blob/40526f52f54d4501737e5246c0e71e56dd7e0b2d/scripts/insecure_grpcurl_series.sh against Querier gRPC API directly (: This will give us exact input that deduplication logic is using.

I think it has to do with some edge values.. 🤔

[creker]

0.14 has the same problem. Had to use different metric to catch it. The problem persists for a few second so I tried to get correct and incorrect results.
bad.txt
good.txt

[csmarchbanks]

I was about to report this same issue!

What is happening in my system at least is that the initialPenalty is by default 5000ms, but I have scrape intervals in the 15s-30s range. The two HA Prometheus instances will commonly scrape the same target ~10s apart which means the deduplication logic will always switch series after the first sample causing extra data.

Here is a failing test that reproduces the behavior I see: master...csmarchbanks:unstable-queries

I pushed out a custom version of thanos that uses a 30s initial penalty and the problem has gone away for me. However, if someone had a 1m scrape interval a 30s initial penalty still would not be enough, and would be way too big for someone with a 5s scrape interval.

[creker]

We have similar setup. Two Prometheus instances scraping the same targets with 10s interval.

[bwplotka]

Awesome. Looks like we might want to adjust penalty based on dynamic interval. The edge case is when interval is reconfigured to be different (:

…

On Wed, 15 Jul 2020 at 17:17, Antonenko Artem ***@***.***> wrote: We have similar setup. Two Prometheus instances scraping the same targets with 10s interval. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2890 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3OYVAJUZEHTV6CWKTG3R3XI73ANCNFSM4OZKKE3A> .

[krasi-georgiev]

I opened a PR that uses the request resolution to handle cases like this one. I am still working on the tests, but so far it is looking good

#3010

[krasi-georgiev]

update: resolution based dedup doesn't work with promql functions so reverted the PR.

In our case adjusting the default look back delta solved the problem so it seems that the problem here is different and caused by scrape time shifting.

@csmarchbanks in your case I think the main problem is that the scrapping between the different replicas is shifted by 30sec. That seems a lot if you manage to align it better problem should be solved.