receive: Add liveness and readiness probe#1537
Merged
brancz merged 5 commits intothanos-io:masterfrom Sep 20, 2019
Merged
Conversation
kakkoyun
force-pushed
the
probes_receive
branch
from
37839f0 to
4d3d4ef
Compare
Member
Author
|
cc @FUSAKLA |
FUSAKLA
reviewed
Sep 18, 2019
cmd/thanos/receive.go
Outdated
| s := newStoreGRPCServer(logger, reg, tracer, tsdbStore, opts) | ||
|
|
||
| level.Info(logger).Log("msg", "listening for StoreAPI gRPC", "address", grpcBindAddr) | ||
| statusProber.SetReady() |
Member
There was a problem hiding this comment.
I think the receiver should probably not be ready until the TSDB is ready? Also not sure about the hashring and the receive interface is also not guarantied to be up at this point.
Maybe this will require some more complex condition for the ready state 🤔
Member
Author
There was a problem hiding this comment.
@FUSAKLA For the TSDB, it's ready at this stage if you check line 270. It runs after TSDB is open.
For receive interface, I thought if something goes south it'll change liveness state so, readiness won't be needed.
I guess I need to double-check the hashring readiness.
I'll have another look at it.
kakkoyun
force-pushed
the
probes_receive
branch
from
105037a to
43c5a8a
Compare
Member
Author
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
kakkoyun
force-pushed
the
probes_receive
branch
from
43c5a8a to
c4279bb
Compare
metalmatze
approved these changes
Sep 20, 2019
brancz
approved these changes
Sep 20, 2019
squat
reviewed
Sep 26, 2019
| s := newStoreGRPCServer(logger, reg, tracer, tsdbStore, opts) | ||
|
|
||
| // Wait hashring to be ready before start serving metrics | ||
| <-hashringReady |
Member
There was a problem hiding this comment.
Why are we waiting for the hashring to be ready before serving metrics from the store? These things are entirely independent IMO
ivan-kiselev
pushed a commit
to ivan-kiselev/thanos
that referenced
this pull request
Sep 26, 2019
* Add prober to receive Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entries Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update README Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Wait hashring to be ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com>
ivan-kiselev
pushed a commit
to ivan-kiselev/thanos
that referenced
this pull request
Sep 26, 2019
* Add prober to receive Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entries Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update README Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Wait hashring to be ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me>
brancz
pushed a commit
that referenced
this pull request
Sep 26, 2019
* Some updates to compact docs Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * some formatting Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Update docs/components/compact.md accept PR suggestions Co-Authored-By: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Add metalmatze to list of maintainers (#1547) Signed-off-by: Matthias Loibl <mail@matthiasloibl.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * resolve comments Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * resolve last comment Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * receive: Add liveness and readiness probe (#1537) * Add prober to receive Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entries Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update README Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Wait hashring to be ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * downsample: Add liveness and readiness probe (#1540) * Add readiness and liveness probes for downsampler Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entry Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Set ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update CHANGELOG Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Clean CHANGELOG Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Document the dnssrvnoa option (#1551) Signed-off-by: Antonio Santos <antonio@santosvelasco.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * feat store: added readiness and livenes prober (#1460) Signed-off-by: Martin Chodur <m.chodur@seznam.cz> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Add Hotstar to adopters. (#1553) It's the largest streaming service in India that does cricket and GoT for India. They have insane scale and are using Thanos to scale their Prometheus. Spoke to them offline about adding the logo and will get a signoff here too. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Fix hotstar logo in the adoptor's list (#1558) Signed-off-by: Karthik Vijayaraju <karthik@hotstar.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Fix typos, including 'fomrat' -> 'format' in tracing.config-file help text. (#1552) Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Compactor: Fix for #844 - Ignore object if it is the current directory (#1544) * Ignore object if it is the current directory Signed-off-by: Jamie Poole <jimbobby5@yahoo.com> * Add full-stop Signed-off-by: Jamie Poole <jimbobby5@yahoo.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Adding doc explaining the importance of groups for compactor (#1555) Signed-off-by: Leo Meira Vital <leo.vital@nubank.com.br> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Add blank line for list (#1566) The format of these files is wrong in the web. Signed-off-by: dongwenjuan <dong.wenjuan@zte.com.cn> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Refactor compactor constants, fix bucket column (#1561) * compact: unify different time constants Use downsample.* constants where possible. Move the downsampling time ranges into constants and use them as well. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * bucket: refactor column calculation into compact Fix the column's name and name it UNTIL-DOWN because that is what it actually shows - time until the next downsampling. Move out the calculation into a separate function into the compact package. Ideally we could use the retention policies in this calculation as well but the `bucket` subcommand knows nothing about them :-( Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * compact: fix issues with naming Reorder the constants and fix mistakes. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * remove duplicate Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me>
GiedriusS
pushed a commit
that referenced
this pull request
Oct 28, 2019
* Add prober to receive Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entries Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update README Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Wait hashring to be ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
GiedriusS
pushed a commit
that referenced
this pull request
Oct 28, 2019
* Some updates to compact docs Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * some formatting Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Update docs/components/compact.md accept PR suggestions Co-Authored-By: Bartlomiej Plotka <bwplotka@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Add metalmatze to list of maintainers (#1547) Signed-off-by: Matthias Loibl <mail@matthiasloibl.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * resolve comments Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * resolve last comment Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * receive: Add liveness and readiness probe (#1537) * Add prober to receive Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entries Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update README Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Wait hashring to be ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * downsample: Add liveness and readiness probe (#1540) * Add readiness and liveness probes for downsampler Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Add changelog entry Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Remove default Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Set ready Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Update CHANGELOG Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> * Clean CHANGELOG Signed-off-by: Kemal Akkoyun <kakkoyun@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Document the dnssrvnoa option (#1551) Signed-off-by: Antonio Santos <antonio@santosvelasco.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * feat store: added readiness and livenes prober (#1460) Signed-off-by: Martin Chodur <m.chodur@seznam.cz> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Add Hotstar to adopters. (#1553) It's the largest streaming service in India that does cricket and GoT for India. They have insane scale and are using Thanos to scale their Prometheus. Spoke to them offline about adding the logo and will get a signoff here too. Signed-off-by: Goutham Veeramachaneni <gouthamve@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Fix hotstar logo in the adoptor's list (#1558) Signed-off-by: Karthik Vijayaraju <karthik@hotstar.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Fix typos, including 'fomrat' -> 'format' in tracing.config-file help text. (#1552) Signed-off-by: Callum Styan <callumstyan@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Compactor: Fix for #844 - Ignore object if it is the current directory (#1544) * Ignore object if it is the current directory Signed-off-by: Jamie Poole <jimbobby5@yahoo.com> * Add full-stop Signed-off-by: Jamie Poole <jimbobby5@yahoo.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Adding doc explaining the importance of groups for compactor (#1555) Signed-off-by: Leo Meira Vital <leo.vital@nubank.com.br> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Add blank line for list (#1566) The format of these files is wrong in the web. Signed-off-by: dongwenjuan <dong.wenjuan@zte.com.cn> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * Refactor compactor constants, fix bucket column (#1561) * compact: unify different time constants Use downsample.* constants where possible. Move the downsampling time ranges into constants and use them as well. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * bucket: refactor column calculation into compact Fix the column's name and name it UNTIL-DOWN because that is what it actually shows - time until the next downsampling. Move out the calculation into a separate function into the compact package. Ideally we could use the retention policies in this calculation as well but the `bucket` subcommand knows nothing about them :-( Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> * compact: fix issues with naming Reorder the constants and fix mistakes. Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com> Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> * remove duplicate Signed-off-by: Ivan Kiselev <kiselev_ivan@pm.me> Signed-off-by: Giedrius Statkevičius <giedriuswork@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR,
/-/healthyendpoint for liveness checks./-/readyendpoint for readiness checks.Changes
/-/healthyendpoint for liveness checks./-/readyendpoint for readiness checks.prober.Proberfor readiness and liveness endpoints.Verification
make testStarted
thanos receiveand made a request to related endpoints.