Thanos Query stop responding if not queried for some time

[sysblade]

Thanos, Prometheus and Golang version used

Docker tags:

thanos:
  version: v0.2.1
prometheus:
  version: v2.5.0

What happened

When thanos-query is not used for some time, it stops answering when we try to query again and only work after are restart.
We saw that when we shut our persistent grafana dashboard (shown on screens on the office), now every time we go to grafana or to the thanos-query interface, the queries hang up forever until we restart thanos-query.
CPU usage on all the thanos relarted processes are low so it doesn't look like it's doing anything.
All the stores are up in thanos-query dashboard but all queries are failing (even up).
After restarting thanos query everything works fine for a few hours (I don't know the exact time)

What you expected to happen
Query working fine

How to reproduce it (as minimally and precisely as possible):

Don't do queries for a few hours, then try to query

Full logs to relevant components

Nothing in the logs except context cancelled because the HTTP queries are timing out on Grafana

Anything else we need to know

Environment:

• OS: Docker 18.09.0 on Centos 7
• Kernel (e.g. uname -a): Linux gc-euw1-prometheus-central-1 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Others:

[bwplotka]

Thanks for report.
What is your configuration for connecting Querier to StoreAPIs? What can you see on Store UI in Thanos Querier when you have case for Thanos Querier not responding?

[sysblade]

There you have the related salt state to deploy query and store :

Make sure Thanos Query is running in Docker:
  docker_container.running:
    - name: thanos-query
    - image: improbable/thanos:{{ pillar["thanos"]["version"] }}
    - user: 4002
    - restart_policy: always
    - network_mode: host
    - command: >
        query
        --cluster.peers store.thanos.internal.xxx.net:10903
        --query.replica-label replica
        --cluster.address 0.0.0.0:10906
        --cluster.advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10906
        --grpc-address 0.0.0.0:10907
        --grpc-advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10907
        --http-address 0.0.0.0:10908
        --http-advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10908
    - binds:
      - "/var/data/thanos:/var/data/thanos:Z"
      - "/etc/monitoring/prometheus:/etc/prometheus:Z"
    - environment:
      - GOOGLE_APPLICATION_CREDENTIALS: /etc/prometheus/gcloud.json

Make sure Thanos Store is running in Docker:
  docker_container.running:
    - name: thanos-store
    - image: improbable/thanos:{{ pillar["thanos"]["version"] }}
    - user: 4002
    - restart_policy: always
    - network_mode: host
    - command: >
        store
        --data-dir /var/data/thanos/store
        --cluster.peers query.thanos.internal.xxx.net:10900
        --objstore.config-file /etc/thanos/bucket.yml
        --cluster.address 0.0.0.0:10903
        --cluster.advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10903
        --grpc-address 0.0.0.0:10904
        --grpc-advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10904
        --http-address 0.0.0.0:10905
    - binds:
      - "/var/data/thanos:/var/data/thanos:Z"
      - "/etc/monitoring/prometheus:/etc/prometheus:Z"
      - "/etc/monitoring/thanos:/etc/thanos:Z"
    - environment:
      - GOOGLE_APPLICATION_CREDENTIALS: /etc/prometheus/gcloud.json

## Then on remote nodes containing Prometheus instances
Make sure Thanos Sidecar is running in Docker:
  docker_container.running:
    - name: thanos-sidecar
    - image: improbable/thanos:{{ pillar["thanos"]["version"] }}
    - user: 4000
    - restart_policy: always
    - network_mode: host
    - command: >
        sidecar
        --prometheus.url http://localhost:9090
        --tsdb.path /prometheus
        --objstore.config-file /etc/thanos/bucket.yml
        --cluster.peers query.thanos.internal.xxx.net:10906
        --cluster.peers store.thanos.internal.xxx.net:10903
        --cluster.address 0.0.0.0:10900
        --cluster.advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10900
        --grpc-address 0.0.0.0:10901
        --grpc-advertise-address {{ grains["ip4_interfaces"]["eth0"][0] }}:10901
        --http-address 0.0.0.0:10902
    - environment:
      - GOOGLE_APPLICATION_CREDENTIALS: /etc/prometheus/gcloud.json
    - binds:
      - "/etc/monitoring/prometheus:/etc/prometheus:Z"
      - "/etc/monitoring/thanos:/etc/thanos:Z"
      - "/var/data/prometheus:/prometheus:Z"

It happened right now, on the stores panel everything is green, health checks are still responding correctly but I can't for example run a up query, hanging forever on it.

Still no errors in the logs

[bwplotka]

Intesting. No immdiate clue - we cannot repro it.

So what is really weird in your setup, is that you have 5 replica HA for Prometheus.. Does it mean all 5 instances scrape the same data? Because that's for query-replica flag in Thanos querier is for. It might work despite that just accidently (because all internal metric labels are unique), but I think you are doing something you did not intent to do.

Nevertheless what might happen now, is that one of those StoreAPI might hang for some reason? Potentially store gateway? One hanging leaf can cause Thanos query to timeout after some time.

I would suggest:

• checking logs on leafs (for sure for 10.1.0.2 which is your store gateway)
• Turning off one by one leafs from gossip, (it's easier when you use DNS than gossip) and try Thanos Querier against on of those leafs only to detect which causes problem. Again, I think you might get some problem with store gateway acessing object store or something?

[sysblade]

Each instance monitor their own system (site or project), they don't share metrics data (Except to monitor each other monitoring system (monitoring each other prometheus, alertmanager, sidecars, etc.)

Should I remove the replica global label from my configuration ?

I've checked the store logs but there is nothing.

What I see is that for example queries like up that need all the leafs are hanging forever but targetting some leafs like up{replica="central"} works fine and it looks like one (bcn) is not working.

When taking a look at this leaf log, there's nothing shown failing recently (Just the classic upload new block).

When restarting this leaf (And not the thanos query like before) it works, but I suppose this will happen again in a few hours (as it was restarted before this ticket as I updated the version from master-2018-11-12-9cec837 to v0.2.1 to try to solve the issue)

[bwplotka]

Yes, they are definitely not replicas then. They are shards (: I would not remove replica label as then you external labels won't be unique (they have to). Just add something that suits your functional shard like cluster or shard (:

Not sure if that is the reason for your weirdiness query timeouts, but for a try.

Have you try turning of some leaf then? when Querier hangs, maybe worth to turn off them one by one.

Also in case on hanging worth to check Thanos Querier metrics.
Maybe concurrent queries are full for some reason for you?

# HELP prometheus_engine_queries The current number of queries being executed or waiting.
# TYPE prometheus_engine_queries gauge
prometheus_engine_queries 4
# HELP prometheus_engine_queries_concurrent_max The max number of concurrent queries.
# TYPE prometheus_engine_queries_concurrent_max gauge
prometheus_engine_queries_concurrent_max 100

``

[sysblade]

When I shutdown the only one that had the issue (bcn) and wait for it to appear red in the store part, queries starts to work again, then when I start it again it keeps working (for a few hours I suppose)

The query limit is 20 everywhere but I can see that sometime the thanos-query reach more than 60 concurrent queries, I think this is when I try to open Grafana and that it gets hanging forever because of the issue.

If I change the label (I suppose it need to be changed everywhere at the same time), that won't affect the ability to read the previous metrics ? Any consequence ?

[bwplotka]

The query limit is 20 everywhere but I can see that sometime the thanos-query reach more than 60 concurrent queries, I think this is when I try to open Grafana and that it gets hanging forever because of the issue.

Doesn't this sounds like your issue here? Can you try to change max-concurrent to 100 e.g? BTW what is running on 10.1.0.2:10904?

Well, the result will have different labels, otherwise for your case it should be fine (:

[sysblade]

10.1.0.2:10904 is the Thanos store, I will try next week to change the limit to 100, but I don't think that as this would only affect Grafana and not single queries from the query UI ?

[sysblade]

I just pushed the change, I will let you know if it happens again, btw maybe it would be interesting to sync the flag to have the same name than on Prometheus ? query.max-concurrent vs query.max-concurrency

[bwplotka]

I think it's up to your deployment/ config management to sync those. From Querier standpoint we don't know what StoreAPI is. (:

[sysblade]

So far setting the limit to 100 everywhere (checked on the metric) didn't fix the issue, I just tried right now and I'm getting the same issue.

By syncing I was meaning to use the same flag name than Prometheus, not to sync the value :)

[bwplotka]

ah, I yes - fully agree. Have you checked resource consumption for components during the issue?

Also, where you able to repro this issue with just the store?

[sysblade]

So far it's not related to the store, I tried to restart the store, didn't fix the issue (And Query doesn't reconnect to it until query is restarted), it looks like it's bound to my bcn sidecar, when restarting this one (or query) it starts working again. (Query and Store are on the same host so I don't think it's network either issue either)

CPU usage is almost 0 when doing the query. No errors in any log, network looks ok as hearthbeat goes fine everywhere.

[bwplotka]

Wonder if it's similar to this: https://improbable-eng.slack.com/archives/CA4UWKEEN/p1546867840210100

[sysblade]

I could not get the link to work, it always give me the end of the #thanos channel 😓