proposal: Add scalable rule storage proposal#2661
Merged
bwplotka merged 1 commit intothanos-io:masterfrom Jun 25, 2020
Merged
Conversation
Member
|
fyi as for test failure: #2663 |
pracucci
reviewed
May 29, 2020
Contributor
pracucci
left a comment
pracucci
left a comment
There was a problem hiding this comment.
From the architectural perspective makes sense to me. Cortex does the same (ruler writes through the remote write path).
bwplotka
approved these changes
May 29, 2020
Member
bwplotka
left a comment
bwplotka
left a comment
There was a problem hiding this comment.
Amazing! As discussed offline I love this idea.
Wonder what @gouthamve and @SuperQ would say (:
LGTM, just some comments.
|
|
||
| A few large rules can create a significant amount of resulting time-series, | ||
| which should not be limited to what we can scale to with a single TSDB embedded | ||
| in the rule component. |
Member
There was a problem hiding this comment.
Suggested change
| in the rule component. | |
| in the rule component. | |
| Additionally, scaling out Ruler in terms of rule evaluations causes further defragmentation of TSDB blocks - as many of rules we have, that many of TSDB "streams" we have. While doable with vertical compaction, it might cause some operational complexity and unnecessary load on the system. |
Member
Author
There was a problem hiding this comment.
Added some of this but rewrote it slightly, could you double check if that's what you were trying to say?
9 tasks
povilasv
reviewed
Jun 16, 2020
povilasv
reviewed
Jun 16, 2020
povilasv
reviewed
Jun 16, 2020
brancz
force-pushed
the
scalable-ruler
branch
2 times, most recently
from
61e828d to
efc8ef5
Compare
Member
Author
|
I think I addressed all comments :) |
bwplotka
approved these changes
Jun 24, 2020
Member
bwplotka
left a comment
bwplotka
left a comment
There was a problem hiding this comment.
LGTM, just last nits, plus approved status (:
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
bwplotka
approved these changes
Jun 25, 2020
paulfantom
added a commit
to paulfantom/thanos
that referenced
this pull request
Jul 9, 2020
* upstream/release-0.14: (46 commits) Cut release v0.14.0-rc.1 (thanos-io#2853) Query: correctly marshal errors to JSON and ignore if nil (thanos-io#2848) ci: Manually download promu in crossbuild stage (thanos-io#2828) Cut release v0.14.0-rc.0 (thanos-io#2826) Soft cut changelog on master to indicate v0.14.0 being in progress (thanos-io#2824) Update ThanosReceiveNoUpload to select sum == 0 (thanos-io#2819) receive: Added more observability, fixed leaktest, to actually check leaks ): (thanos-io#2817) Query: always return a string in the `lastError` field (thanos-io#2809) Added missing CHANGELOG entry for PR 2613 (thanos-io#2820) receive: Fixed small options race; Removed unused StartTime feature. (thanos-io#2816) go.mod: Bump Prometheus to current latest (thanos-io#2814) Implement CLI Flags page in React UI (thanos-io#2796) Improve ThanosReceiveNoUpload to only alert on current instances store: Preallocate output buffer when encoding postings. (thanos-io#2812) compact: introduce flag --block-viewer.global.sync-block-interval (thanos-io#2752) docs: compact: add blurb about how retention policy works (thanos-io#2808) Reduced memory allocations in readIndexRange() (thanos-io#2807) ui: Add Stores page to React UI (thanos-io#2754) Added Kemal to Maintainer Role; Kemal is volounteering to be next release shephard (thanos-io#2804) proposal: Add scalable rule storage proposal (thanos-io#2661) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Add proposal for scalable rule storage.
When I started writing this, I actually thought it would be much more intricate, but it still changes the architecture of the rule component pretty significantly in the long term, so I still think it's important to have as a proposal first, and if accepted, then for people to understand the changes that have happened over time in the future.
@thanos-io/thanos-maintainers