Skip to content

Downgrade grpc-go to v1.76.0 and revert compression changes#8671

Merged
MichaHoffmann merged 1 commit intothanos-io:mainfrom
jdgeisler:patch-grpc-0.41.0
Feb 10, 2026
Merged

Downgrade grpc-go to v1.76.0 and revert compression changes#8671
MichaHoffmann merged 1 commit intothanos-io:mainfrom
jdgeisler:patch-grpc-0.41.0

Conversation

@jdgeisler
Copy link
Contributor

  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

Reverting part of f3d3972 from https://github.com/thanos-io/thanos/releases/tag/v0.41.0-rc.0

Upgrading to grpc-go v1.78.0 with grpc compression was causing OOMKills in our queriers. Below was our querier config with the issue:

Containers:
  query:
    Container ID:  containerd://9cce0755a156abdaa575b3b9d6052eb3e5b5cd6ffdac494d910facad047b2e91
    Image:         quay.io/thanos/thanos:v0.41.0-rc.0
    Image ID:      quay.io/thanos/thanos@sha256:e97ab74506ce7a697a56f82cfce24f94c51de24ee768ec2f750943c873de77fb
    Ports:         10901/TCP, 10902/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      query
      --auto-gomemlimit.ratio=0.8
      --enable-auto-gomemlimit
      --query.metadata.default-time-range=24h
      --query.max-concurrent-select=80
      --query.max-concurrent=80
      --store.limits.request-samples=0
      --store.limits.request-series=0
      --query.auto-downsampling
      --grpc-compression=snappy
      --query.replica-label=replica
      --query.replica-label=prometheus_replica
      --store.response-timeout=60s
      --endpoint=dnssrv+_grpc._tcp.receive-headless.thanos.svc.cluster.local:10901
      --endpoint-group-strict=receive-store-short-0.thanos.svc.cluster.local:10901
      --endpoint-group-strict=receive-store-mid-0.thanos.svc.cluster.local:10901
      --endpoint-group-strict=receive-store-long-0.thanos.svc.cluster.local:10901
      --tracing.config=type: OTLP
      config:
        insecure: true
        client_type: "http"
        service_name: receive-query
        endpoint: "opentelemetry-tracing-agent.istio-system.svc.cluster.local:4318"
        url_path: "/v1/traces"

Verification

After reverting these changes the OOMKills were resolved

Screenshot 2026-02-09 at 3 13 55 PM

Signed-off-by: James Geisler <geislerjamesd@gmail.com>
@MichaHoffmann MichaHoffmann merged commit 809715f into thanos-io:main Feb 10, 2026
45 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants