Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
docs: Document Thanos Sharding
Signed-off-by: Xiang Dai <764524258@qq.com>
  • Loading branch information
daixiang0 committed Jan 29, 2020
commit 6b74e62114795f43dc437a2084b66ee0a0c1e8cd
61 changes: 61 additions & 0 deletions docs/sharing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: Sharing
type: docs
menu: thanos
slug: /Sharing.md
---

# Store and Compactor Sharding

Store and Compactor Sharding is for Long Term Retention Storage.

## Background

Currently all components that read from object store assume that all the operations and functionality should be done based
on **all** the available blocks that are present in the certain bucket's root directory.

This is in most cases totally fine, however with time and allowance of storing blocks from multiple `Sources` into the same bucket,
the number of objects in a bucket can grow drastically.

This means that with time you might want to scale out certain components e.g:

* Compactor: Larger number of objects does not matter much, however compactor has to scale (CPU, network) with number of Sources pushing blocks to the object storage.

### Relabelling

Similar to [promtail](https://github.com/grafana/loki/blob/master/docs/promtail.md#scrape-configs) this config will follow native
[Prometheus relabel-config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config) syntax.

The relabel config defines filtering process done on **every** synchronization with object storage.

We will allow potentially manipulating with several of inputs:

* External labels:
* `<name>`

Output:

* If output is empty, drop block.

By default, on empty relabel-config, all external labels are assumed.

Example usages would be:

* Drop blocks which contains external labels cluster=A

```yaml
- action: drop
regex: "A"
source_labels:
- cluster
```

* Keep only blocks which contains external labels cluster=A
```yaml
- action: keep
regex: "A"
source_labels:
- cluster
```

By controling which labels should be included in the blocks, we can do sharing.