Prometheus incorrectly handles blocks created by thanos-receiver

[radeklesniewski]

Hi @brancz ,

I find "thanos-receiver" idea really interesting, so I've started playing around with sending some jsons with metrics via remote-write interface to receiver. After running tests for some time I've decided to attach blocks created by receiver directly to data directory of Prometheus instance. The main problem I've spotted on Prometheus UI is that I am able to see list of metrics persisted by receiver, however when I query for any metric, Prometheus does not show any data point for particular metric.

My first idea why is it working like that was incompatibility of "prometheus/tsdb" modules. So I've updated "thanos" and "prometheus" dependencies to get it working (I've created PR with changes, not sure if I made it correctly), however the problem is still occurring. So my question is: Is it normal that blocks created by thanos-receiver won't be compatible for Prometheus? Maybe I'm missing something obvious here...

[brancz]

Sorry for not adding a readme, but this is just a proof of concept, this will be directly implemented in to the Thanos project itself. But as the patches are already ready, I'm happy to merge those :)

[brancz]

Closed by #1

[radeklesniewski]

I'm not sure if we understood each other correctly :) the patch with upgraded dependencies didn't resolve the issue I'm talking about. It sure is just proof of concept, however my question was whether possibility of incorrect format of blocks is something expected? Or maybe there is no reason why thanos-receiver blocks wouldn't be compatible with Prometheus?

[brancz]

Oh whops. Sorry, yes I misunderstood. Yes blocks produced by this should be valid Prometheus blocks, although most important is that it can read them itself :) (you may also need to make sure you're using the right Prometheus version).

[rafal-dudek]

We have doubled checked the scenario, the configuration glitch was on our site. Metrics are visible in both Prometheus and Thanos Web UIs. Our setup augments thanos-receiver with thanos-sidecar (with mocked /api/v1/status/config) to transfer blocks to S3. Blocks can be used (copy pasted) to Prometheus directly too. We can close the issue.

[brancz]

Nice :) I’m currently on vacation but as soon as I’m back we will work towards getting this into main line Thanos and complete the featureset.

[norbertwnuk]

Hi @brancz, any news on continuing the work? Would like to take a look / help even in case of early stages :)

[brancz]

The first pieces have been merged upstream thanos-io/thanos#811 🎉