Refresh `thanosbench`

[bwplotka]

AC:

• ./benchmark should be a separate repo in thanos-io org. (TO decide)
• thanosbench should be up to date with newest Thanos.

[yeya24]

I' m interesting on this. But with some questions.

1. Rename to thanosbench or remain benchmark?
2. Do we have any plan for this part? Will it be something like prombench?
3. Don't have any ideas about this Remove gossip from Dashboards and benchmark tool #1180 (comment)

[bwplotka]

Valid questions!

So first of all. The plan is to remove the benchmark directory totally. It's super old and not maintained. Let's stop confusing users: #1423

1. I think thanosbench makes sense (:
2. Have some rough ideas, but TBH for recent benchmarks, I used this: https://github.com/thanos-io/thanos/tree/865d5ec710b5651ead08a7dc73864a6f5dbeeaa9/tutorials/kubernetes-demo/blockgen and some infrastracture definitions in Go: https://github.com/bwplotka/mimic/tree/master/examples/prometheus-remote-read-benchmark
3. Commented.

Overall I am not sure if separate project even makes sense.

Benefits of the same repo:

• It's easier to synchronize versions of deps, thanos vs benchmarks
• It is tied to Thanos so it make sense to have it in same repo.
• Reuse CI, tooling etc.

Issues:

• Same go module? In this case, we might add more deps then needed to the Thanos.
• Different module? Then same issue with versions sync (:

[krasi-georgiev]

here are a couple of tools that can help
https://github.com/open-fresh/avalanche
https://github.com/m3db/promremotebench

And here is a bench test setup that I started on the ingestion path.
https://github.com/krasi-georgiev/benchTSDB

[bwplotka]

Created https://github.com/thanos-io/thanosbench

The initial idea is to have a place that will hold:

• CLI / docker tools for benchmarking
  • Querying
  • Generating blocks of different size and compactions, downsampling structure - for both object storage as well as filesystem
• Kubernetes templates based on Go (mimic) that will generate deployment YAMLs for different cases.
• Tooling to build thanos-io/thanos certain commit into docker image for Kubernetes use

That's for start. Feedback welcome, what would you like to see more?

[bwplotka]

https://github.com/thanos-io/thanosbench exists. Let's create issues on that repo related to benchmarking.