Skip to content

fix: query announced endpoints match relabel-config#8370

Merged
GiedriusS merged 8 commits intothanos-io:mainfrom
open-ch:fix/querier-relabel-config
Jul 15, 2025
Merged

fix: query announced endpoints match relabel-config#8370
GiedriusS merged 8 commits intothanos-io:mainfrom
open-ch:fix/querier-relabel-config

Conversation

@verejoel
Copy link
Contributor

@verejoel verejoel commented Jul 11, 2025

The --selector.relabel-config flag was only filtering stores during query execution, but not when announcing available label sets via the Info API or /stores endpoint. This caused a mismatch where:

  • Queries correctly returned filtered results
  • UI and inter-querier communication showed unfiltered label sets

This happens because ProxyStore's LabelSet() method ignored filtered results from TSDBSelector.MatchLabelSets() and always included all label sets from matching stores.

In this PR, label set announcements now match the actual queryable data, fixing confusion in multi-tier querier deployments where root queriers would see all tenant labels despite leaf queriers being configured to filter specific tenants.

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • Apply TSDBSelector filtering consistently in ProxyStore.LabelSet() and TimeRange()
  • Filter /stores REST endpoint using same TSDBSelector logic
  • Pass TSDBSelector to QueryAPI for UI consistency

Verification

I have this working in our DEV env. We have a leaf querier that's setup with a hashmod relabel config across our ingester fleet.

For simplicity, I show this with a keep relabel config as well (both work)

        - --endpoint=dnssrvnoa+_grpc._tcp.thanos-ingester.thanos.svc.cluster.local
        - |
          - action: keep
            source_labels: [tenant_id]
            regex: sdp

We also have a root querier plugged into all leaf querier shards.

        - --endpoint=thanos-ingester-querier-0.thanos.svc.cluster.local:10901
        - --endpoint=thanos-ingester-querier-1.thanos.svc.cluster.local:10901
        - --endpoint=thanos-ingester-querier-2.thanos.svc.cluster.local:10901

Without the patch, in both the leaf and root queriers, the /stores endpoint shows all external labels on every endpoint.

image image

With the patch, the result is more expected:

image image

@verejoel verejoel force-pushed the fix/querier-relabel-config branch from 98aab38 to e176cf2 Compare July 11, 2025 23:14
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
@verejoel verejoel force-pushed the fix/querier-relabel-config branch from e176cf2 to 8b3c29a Compare July 11, 2025 23:16
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
@verejoel verejoel marked this pull request as ready for review July 11, 2025 23:38
@verejoel verejoel changed the title Fix/querier relabel config fix: query announced endpoints match relabel-config Jul 11, 2025
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
@verejoel verejoel force-pushed the fix/querier-relabel-config branch from 5fed3b3 to 0bb3e73 Compare July 13, 2025 18:24
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
MichaHoffmann
MichaHoffmann previously approved these changes Jul 13, 2025
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
@verejoel verejoel force-pushed the fix/querier-relabel-config branch from dbed3a5 to 0972c43 Compare July 13, 2025 19:17
Joel Verezhak added 3 commits July 13, 2025 21:42
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
Signed-off-by: Joel Verezhak <jverezhak@open-systems.com>
@verejoel verejoel requested a review from MichaHoffmann July 13, 2025 23:04
@fpetkovski
Copy link
Contributor

Awesome work 👍

@GiedriusS GiedriusS merged commit 77f12e3 into thanos-io:main Jul 15, 2025
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants