Querier/Store v0.8.0 regression: Thanos now considers identically externally labeled Store and Sidecar instances as having duplicate label sets. #1632
Description
Thanos, Prometheus and Golang version used: 0.8.0, 2.13.0, 1.13.1
Object Storage Provider: Azure
What happened: 0.7.0 I used the "replica" label with values "a" and b". I used --query.replica-label=replica in my Thanos Query nodes. Now, if I put the Store and Sidecar as stores in the same Thanos Query instance, one or the other gets dropped for labels as not unique.
What you expected to happen: Uniqueness of labels only be enforced for the same Thanos type (store versus store, sidecar versus sidecar).
How to reproduce it (as minimally and precisely as possible):
- Apply replica = "a" external_label to one Prometheus instance
- Apply replica = "b" to another Prometheus instance
- Add sidecars to above nodes
- Tell a Thanos Query instance to point to those sidecars.
Full logs to relevant components:
{"address":"thanos-query-app-swarm-sidecars:10903","caller":"storeset.go:261","component":"storeset","duplicates":2,"extLset":"{monitor=\"apps-app-swarm\",region=\"useast2\",replica=\"a\"},{monitor=\"apps-app-swarm\",region=\"useast2\",replica=\"b\"},{monitor=\"infra-app-swarm\",region=\"useast2\",replica=\"a\"},{monitor=\"infra-app-swarm\",region=\"useast2\",replica=\"b\"},{monitor=\"services-app-swarm\",region=\"useast2\",replica=\"a\"},{monitor=\"services-app-swarm\",region=\"useast2\",replica=\"b\"}","level":"warn","msg":"dropping store, external labels are not unique","ts":"2019-10-11T06:33:29.563785804Z"}
Anything else we need to know: Again this worked in 0.7.0. I did notice in the UI that the label sets for my Stores used to be blank, and now they are not. If I had to guess, the "glitch" got "fixed" there and caused a collision, but I am not seeing the "new way" to ensure that my given Thanos Store for a given Thanos Sidecar "replica" remains uniquely labelled.