bucket: thanos.blocks is null

[MacroPower]

Thanos bucket version used: v0.11 & master

Object Storage Provider: S3

What happened:

We had an issue with overlapping data in one of our blocks. Bucket web worked fine while this data existed for some time. However, after running compactor (and removing a few blocks that it complained about), Bucket web no longer loads. Edit, there was an issue with my object store config.

When trying to load bucket web, we see an error in the console that thanos.blocks is null. And therefore (of course) there's no way to .sort().

What you expected to happen:

• Bucket web should display a helpful error message

Handling this could be pretty simple (see example), but I have no idea what error message would be most apt.

# bucket.js

function draw() {
    if (thanos.refreshedAt == "0001-01-01T00:00:00Z") {
        thanos.err = "Synchronizing blocks from remote storage";
    }
    else if (!thanos.blocks)
    {
        thanos.err = "No blocks are loaded";
    }
    ...

Or some other more elegant solution, any suggestions?

How to reproduce it (as minimally and precisely as possible):

• Create Thanos instance
• Load bucket web interface using an empty bucket

[daixiang0]

Do not set any external labels, or duplicate the external labels

If not set it, sidecar would fail to run.

Do i misunderstand it :(

[MacroPower]

@daixiang0 You could also just have two instances with identical labels, e.g.

	Prometheus-0	Prometheus-1
prometheus_group	ex1	ex1
prometheus_replica	r1	r1

Just some way for you to force some overlapping blocks into storage.

[GiedriusS]

@MacroPower what do you see in the logs of bucket web?

[MacroPower]

@GiedriusS Nothing super interesting from what I can see.

level=info ts=2020-03-13T16:18:34.449771Z caller=main.go:153 msg="Tracing will be disabled"
level=info ts=2020-03-13T16:18:34.45242Z caller=intrumentation.go:48 msg="changing probe status" status=ready
level=info ts=2020-03-13T16:18:34.453029Z caller=intrumentation.go:60 msg="changing probe status" status=healthy
level=info ts=2020-03-13T16:18:34.45307Z caller=http.go:56 service=http/server component=bucket msg="listening for requests and metrics" address=0.0.0.0:10902
level=info ts=2020-03-13T16:18:34.453071Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2020-03-13T16:18:34.456062Z caller=bucket.go:468 msg="synchronizing block metadata"
level=info ts=2020-03-13T16:18:34.801354Z caller=bucket.go:479 msg="downloaded blocks meta.json" num=0

The console logs are:

Sorry, I should have included this in the original post.

[GiedriusS]

Ah, I see. I was wondering how could this happen. Then, it seems, like we need to tweak the download function in cmd/thanos/bucket.go to at least return an empty array in case no blocks are found or check for it in the callee. 😄 AFAICT this was never accounted for. @MacroPower wanna contribute such a PR? :p

[MacroPower]

@GiedriusS Great catch, I didn't even realize it was loading 0 blocks! It should be loading 40+, not sure why it's loading none.

I might look into that further and submit a PR depending on my schedule over the next few weeks.

[MacroPower]

@GiedriusS Still no idea why bucket isn't picking up blocks, but I put in a PR #2271 just to handle this specific case. Take a look and feel free to make any suggestions!