Limit size of blocks to X bytes on compaction.

[bwplotka]

This is to ensure the scalability of the compactor.

Tricky part: How to shard block, what label to put, or maybe vertical compaction?

[bwplotka]

On it now, proposal to be created.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[bwplotka]

Still in progres. Vertical block sharding has to be designed and implemented.

[bwplotka]

Proposal is half done: #3390

The plan is to team up with LFX mentee for knowledge sharing 🤗

[someshkoli]

Hey @bwplotka, I'd like to pick this up again and work on it.

Proposal is half done: #3390

From what I understand, we need to shard the block after certain size continuously so that it won't hit the size limit (64gb as for now). Also how are we planning to shard these blocks, I remember planning of breaking blocks into half of the compacted size until its below the threshold value, Any pointers on that ?