receive: Track replications

[kakkoyun]

This PR adds a new metric to track replication behaviour. Previously, thanos_receive_forward_requests_total was used. However, that metric is now misleading because we have recently changed how replication behaves (#2621).

• I added CHANGELOG entry for this change.

Changes

• Add a new metric to track replicated request thanos_receive_replications_total fd04971
• Add alerts and dashboards for the new metric 0aa495d
• Modify alerts for the thanos_forward_requests_total 0aa495d

Verification

• make test-local
• make examples
• make example-rules-lint

cc @krasi-georgiev

[bwplotka]

I think the problem is a bit different here.

When I think about it I think what is happening is that we don't cancel the context here, however, context is still chained(!).

I think what would fix this problem would be actually starting new context for forwarding and then cancel it only if timeout OR ((not yet success or failure) && parent context is cancelled). WDYT?

This would have major benefit of knowing best effort forwards status properly and replication status is visible by write error / success no? cc @squat

[bwplotka]

In other words what I mean is that http_requests_total{handler="receive"} is really the same as our replication metric you add here no?

[kakkoyun]

@bwplotka Yes, eventually that's the result. We can use that metric instead of adding a new one. I just wanted to be more explicit about the metric.

I think what would fix this problem would be actually starting new context for forwarding and then cancel it only if timeout OR ((not yet success or failure) && parent context is cancelled). WDYT?

For me, this is just a misleading metric issue. Rather than detaching the forward requests from the incoming request's context, I would suggest to catch this case and handle it different than an error.

[bwplotka]

You cannot catch cancellation in any other way than having separate context ):

[bwplotka]

It's not error handling, it's premature context cancel from parent IMO

[kakkoyun]

@bwplotka @squat Closing this one.

We have talked with @bwplotka offline and decided to fix it in a simpler manner. I'll send another PR.

[kakkoyun]

@squat has pointed out offline that we have multiple replication batch operations per request so http_requests_total{handler="receive"} is not enough to do the trick. Thus opened this again.

[brancz]

Really awesome! Nothing to complain :D

Will leave for one more review though as this is a rather large change.

[bwplotka]

Looks good! Just some small nits 🤗

Thanks!