Correct course of action when the size of index headers exhausts disk

[sinapah]

Hello,

I have three Mimir backend pods, running the compactor and store-gateway roles, running with replication factor=3. Each of the pods has a persistent volume of size 1Gi. The problem I'm having is that I see logs like:

2026-05-25T20:14:27.088Z [mimir] ts=2026-05-25T20:14:27.019411838Z caller=lazy_binary_reader.go:182 level=info user=anonymous msg="could not download sparse index-header from bucket; will reconstruct when the block is queried" err="getting sparse index-header from bucket: The specified key does not exist."
2026-05-25T20:14:27.088Z [mimir] ts=2026-05-25T20:14:27.065057295Z caller=bucket.go:450 level=error user=anonymous msg="loading block failed" elapsed=90.539934ms id=01KS4NCRN82HNRMXMV8ZGR0QW3 err="create index header reader: write index header: new binary index header writer: mkdir /data/tsdb-sync/anonymous/01KS4NCRN82HNRMXMV8ZGR0QW3: no space left on device"

I probably could resize the persistent volume, but I wonder if there is a better course of action here or whether this is a symptom that the compactor is failing to offload the sparse index-headers to S3 storage. Could lazy loading help here? Is there another way of reducing the number of index-headers being stored on disk?

[mg1986jp]

The 1Gi PV is almost certainly too small for the index headers. The store-gateway downloads and caches index-header files for each block, and with replication factor 3, each gateway is responsible for a portion of all blocks. As your data grows, the index headers will outgrow 1Gi pretty quickly.

I'd bump the PV size to at least 10-20Gi as a starting point, though the right size depends on your total number of blocks and compaction level. You can check the actual disk usage of the index headers directory to see how much space they need right now.

If you want to reduce the disk footprint without increasing the PV, you can try setting store_gateway.sharding_ring.replication_factor lower (though that affects query availability), or enable lazy loading with blocks_storage.bucket_store.index_header.lazy_loading_enabled: true so it only loads headers for blocks that are actually queried.

[sinapah]

Thanks @mg1986jp! This is very helpful. If you don't mind, I have another question for my understanding:

• I have Grafana dashboards that rely on Mimir as the datasource. Some of these dashboards visualize metrics over, say, the past 3 months. Since I think blocks are based on time ranges, the number of blocks that are queried will continue to rise over time. In that case, would lazy loading just be delaying the problem? Should I use blocks_storage.bucket_store.index_header.lazy_loading_enabled: true and blocks_storage.bucket_store.index_header.lazy_loading_idle_timeout: <some value higher than 0> so that Mimir offloads unused queries continuously? The trade-off between query latency the first time an index header can probably be accepted if it reduces the need for increasing disk space.

[mg1986jp]

@sinapah Good thinking. Lazy loading alone (without an idle timeout) would indeed just delay the problem — once a dashboard queries a 3-month range, all the relevant block headers get loaded and stay in memory/disk.

The key is combining both settings:

• lazy_loading_enabled: true — headers are loaded on demand instead of eagerly at startup
• lazy_loading_idle_timeout: 30m (or similar) — headers that haven't been accessed for that duration get unloaded, freeing disk space

This way, only the blocks actively being queried occupy disk at any given time. When a dashboard is opened, headers are loaded on-the-fly, and after the timeout they're evicted again.

That said, the total number of blocks doesn't grow linearly over time. The compactor merges smaller blocks (2h) into progressively larger ones (up to 24h by default). So over 3 months you might have ~90 compacted blocks rather than thousands of raw ones. The disk pressure comes more from the total size of index headers per block (which depends on series cardinality) than from block count alone.

For your setup, I'd recommend:

1. Enable lazy loading with an idle timeout (e.g. 30m) as you described
2. Monitor the actual peak disk usage under this config — it will be much lower than eager loading
3. Size your PV to handle the peak (likely 3-5Gi instead of 10-20Gi with lazy loading)

The first-query latency hit is typically a few hundred milliseconds per block header, which is very reasonable for dashboard use.

[sinapah]

Understood, thanks @mg1986jp!