Skip to content

store: fix chunk pool memory leak#1512

Merged
GiedriusS merged 1 commit intothanos-io:masterfrom
abursavich:pool-leak
Sep 13, 2019
Merged

store: fix chunk pool memory leak#1512
GiedriusS merged 1 commit intothanos-io:masterfrom
abursavich:pool-leak

Conversation

@abursavich
Copy link
Contributor

I experienced a slow drip of "key does not exist" errors, presumably due to compactions, then nearly all get_range requests began to fail with "pool exhausted" errors.

caller err
bucket.go:871 rpc error: code = Aborted desc = fetch series for block 01DM8ACH11QQQESTY0VET5PXKS: preload chunks: read range for 1: get range reader: The specified key does not exist.
bucket.go:871 rpc error: code = Aborted desc = fetch series for block 01DMA1C7D151KM6H1DGY498WWQ: preload chunks: read range for 6: allocate chunk bytes: pool exhausted

Screen Shot 2019-09-12 at 3 41 44 PM

Changes

Return the byte slice to the pool on error paths.

Verification

N/A

Signed-off-by: Andy Bursavich <abursavich@gmail.com>
Copy link
Member

@GiedriusS GiedriusS left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this, we are definitely forgetting to put the contents back into the pool here! All of the other places seem OK to me.

@GiedriusS GiedriusS merged commit 42e9506 into thanos-io:master Sep 13, 2019
Copy link
Member

@bwplotka bwplotka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch!

wbh1 pushed a commit to wbh1/thanos that referenced this pull request Sep 17, 2019
Signed-off-by: Andy Bursavich <abursavich@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants