compactor: Rare race of sharded compactors / slow upload causing data loss.

[bwplotka]

As per discussion, #1901 sharding compactor is currently not fully safe. Please do not use it or use only for smaller setups.

We are working on this, but the reason is also partially explained in https://docs.google.com/document/d/1QvHt9NXRvmdzy51s4_G21UW00QwfL9BfqBDegcqNvg0/edit#

TL;DR:

• Compactor A for shard cluster = A is uploading the block X - upload takes more than 30m
• Compactor B is syncing all blocks. It see block X that is being uploaded for more than 30m, removing block.

[bwplotka]

The workaround here is to change --consistencyDelay on Compactor to larger value like 2h, given block upload time being for Compactor, for biggest block being quicker than this. This has side effect of compaction process being delayed by 2h which should be fine.

cc @mattrco

[bwplotka]

• Delay clean up of potential partial uploads to 2 days: Use block.MetaFetcher in Compactor. #1937
• Add ModifiedTime attribute per object for all object storages and base on that: Help wanted.