Skip to content

Add memcached support to index cache #1881

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bwplotka merged 34 commits into from
Jan 3, 2020
Merged

Add memcached support to index cache #1881

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
cc101b8
Moved index cache key struct outside of the file containing the in-me…
pracucci Dec 5, 2019
ba6af3d
Added cacheKey.string() to make it memcached friendly
pracucci Dec 5, 2019
0f57c95
Added MemcachedIndexCache support
pracucci Dec 12, 2019
aaa15f4
Export Prometheus metrics from MemcachedIndexCache
pracucci Dec 12, 2019
60b23d4
Fixed linter issues
pracucci Dec 12, 2019
b114623
Simplified workers wait group in memcachedClient
pracucci Dec 13, 2019
4afe30a
Fixed memcached client GetMulti() results batches channel buffer
pracucci Dec 13, 2019
bcc5ceb
Wait for addrs resolution gorouting to complete on memcachedClient.St…
pracucci Dec 16, 2019
ebd9af0
Return struct from NewMemcachedClient()
pracucci Dec 16, 2019
4581111
Update pkg/cacheutil/memcached_client.go
pracucci Dec 16, 2019
c9d37f9
Simplified memcachedClient tests
pracucci Dec 16, 2019
77413ca
Cleaned up code based on feedback
pracucci Dec 16, 2019
16d368a
Removed error from GetMulti() return and introduced metrics and traci…
pracucci Dec 16, 2019
9178cdd
Fixed compilation errors in store E2E tests
pracucci Dec 16, 2019
8951227
Added leaktest check to all tests
pracucci Dec 16, 2019
390e93c
Introduced --index.cache-config support
pracucci Dec 16, 2019
702dc82
Fixed compilation errors in store E2E tests
pracucci Dec 16, 2019
bf5dd7c
Updated store flags doc
pracucci Dec 16, 2019
fb6d122
Updated index cache doc
pracucci Dec 16, 2019
a7c20f2
Updated changelog
pracucci Dec 16, 2019
e2c72a0
Increased default memcached client timeout from 100ms to 500ms
pracucci Dec 17, 2019
f8b318e
Disabled memcached client max batch size by default
pracucci Dec 17, 2019
491c5bb
Fixed index cache key max length
pracucci Dec 17, 2019
db9356b
Removed TODO from memcached client since looks the general consensus …
pracucci Dec 17, 2019
bb3eb88
Allow to configure in-memory index cache using byte units
pracucci Dec 18, 2019
1ea97b2
Refactored index cache config file doc
pracucci Dec 18, 2019
08f1bd3
Fixed nits in comments
pracucci Dec 18, 2019
f83b2d2
Replaced hardcoded 16 with ULID calculated length based on review com…
pracucci Dec 18, 2019
b8b8517
Do not expose jumpHash func
pracucci Dec 18, 2019
f62a0b4
Updated changelog
pracucci Dec 18, 2019
b2d822e
Switched MemcachedClient GetMulti() concurrency limit to a gate
pracucci Dec 18, 2019
9113f0c
Fixed flaky tests
pracucci Dec 18, 2019
da13b4b
Fixed typos in comments
pracucci Dec 19, 2019
f1482f2
Renamed memcached config option addrs to addresses
pracucci Dec 19, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Updated store flags doc 
Signed-off-by: Marco Pracucci <marco@pracucci.com>
  • Loading branch information
@pracucci
pracucci committed Jan 3, 2020
commit bf5dd7c83250fe3741fcedac3e7033143322a58f
4 changes: 2 additions & 2 deletions cmd/thanos/store.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ func registerStore(m map[string]setupFunc, app *kingpin.Application) {
dataDir := cmd.Flag("data-dir", "Data directory in which to cache remote blocks.").
Default("./data").String()

indexCacheSize := cmd.Flag("index-cache-size", "Maximum size of items held in the in-memory index cache. Ignored if --index-cache.config option is specified.").
indexCacheSize := cmd.Flag("index-cache-size", "Maximum size of items held in the in-memory index cache. Ignored if --index-cache.config or --index-cache.config-file option is specified.").
Default("250MB").Bytes()

indexCacheConfig := extflag.RegisterPathOrContent(cmd, "index-cache.config",
Copy link
Member

@GiedriusS GiedriusS Dec 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

"YAML file that contains index cache configuration. See format details: https://thanos.io/caching.md/",
"YAML file that contains index cache configuration. See format details: https://thanos.io/components/store.md/",
false)

chunkPoolSize := cmd.Flag("chunk-pool-size", "Maximum size of concurrently allocatable bytes for chunks.").
Expand Down
13 changes: 12 additions & 1 deletion docs/components/store.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,18 @@ Flags:
CA is specified, there is no client
verification on server side. (tls.NoClientCert)
--data-dir="./data" Data directory in which to cache remote blocks.
--index-cache-size=250MB Maximum size of items held in the index cache.
--index-cache-size=250MB Maximum size of items held in the in-memory
index cache. Ignored if --index-cache.config or
--index-cache.config-file option is specified.
--index-cache.config-file=<file-path>
Path to YAML file that contains index cache
configuration. See format details:
https://thanos.io/components/store.md/
--index-cache.config=<content>
Alternative to 'index-cache.config-file' flag
(lower priority). Content of YAML file that
contains index cache configuration. See format
details: https://thanos.io/components/store.md/
--chunk-pool-size=2GB Maximum size of concurrently allocatable bytes
for chunks.
--store.grpc.series-sample-limit=0
Expand Down