store: Start metric and status probe HTTP server as earlier as possible

[kakkoyun]

As a result of a couple of issues that we had by running Thanos Store with liveness probes on Kubernetes, this PR attempts to fix the liveness probe issues by moving them earlier in the start-up sequence. Now, metrics and status probe HTTP server starts as earlier as possible and returns success from /-/healthy endpoint and sets its status ready when everything else properly bootstrapped.

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

A good read on the subject: https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html

Changes

• starts metric and status probe HTTP server for Store
• attempts to fix store gateway: Start metric server from the very start of the proces. #1653

Verification

• make local-test
• MINIO_ENABLED=1 ./scripts/quickstart.sh while debug log enabled, and observed start sequence.

Thanos Store start sequence logs:

level=info name=store ts=2019-10-17T12:52:19.814229Z caller=main.go:170 msg="Tracing will be disabled"
level=info name=store ts=2019-10-17T12:52:19.814515Z caller=factory.go:39 msg="loading bucket configuration"
level=info name=store ts=2019-10-17T12:52:19.814916Z caller=cache.go:172 msg="created index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=math.MaxInt64
level=info name=store ts=2019-10-17T12:52:19.815104Z caller=main.go:257 msg="disabled TLS, key and cert must be set to enable"
level=info name=store ts=2019-10-17T12:52:19.815301Z caller=store.go:252 msg="starting store node"
level=info name=store ts=2019-10-17T12:52:19.815336Z caller=store.go:198 msg="initializing bucket store"
level=info name=store ts=2019-10-17T12:52:19.815359Z caller=main.go:353 msg="listening for requests and metrics" component=store address=0.0.0.0:10906
level=info name=store ts=2019-10-17T12:52:19.815421Z caller=prober.go:143 msg="changing probe status" status=healthy
level=info name=store ts=2019-10-17T12:52:19.818189Z caller=store.go:203 msg="bucket store ready" init_duration=3.131623ms
level=info name=store ts=2019-10-17T12:52:19.818247Z caller=store.go:245 msg="listening for StoreAPI gRPC" address=0.0.0.0:10905
level=info name=store ts=2019-10-17T12:52:19.818287Z caller=prober.go:114 msg="changing probe status" status=ready

[kakkoyun]

cc @bwplotka @brancz @FUSAKLA @squat @metalmatze

[squat]

Looking at the code changes, it seems to me that the only thing that has changed is the lexicographic ordering of the function calls but that the server is still started at exactly the same time. Note that the HTTP servers are assigned to the run group and that they only actually start serving requests when the run group itself is started.

[kakkoyun]

@squat Thanks for pointing it. I somehow missed that. I'll have another hard look at it.

[bwplotka]

Suggestions (:

[kakkoyun]

@squat @bwplotka I have updated scheduleHTTPServer, now it spawns a goroutine independent from the rungroup to start Server as soon as it's scheduled, and then syncs it with the rungroup.

[bwplotka]

Wonder if we can also add more logs to the startup as well, since you are touching this part as per: #1655

[kakkoyun]

@bwplotka Of course, I can I'll check the related conversation.

[bwplotka]

I don't understand really why we move around all of this scheduleHTTP servers but happy with that if there is a reason (:

2 comments, otherwise LGTM! Thanks.

I think we really need to advertise now to NOT add readiness probe on metric. Users do that and now they will have false impression store that store is ready, but it actually still just loading the blocks...

Wonder how to communicate that ):

I will add more logging to initial sync in separare PR (:

[kakkoyun]

@bwplotka @squat I removed unnecessary changes, fixed emphasized issues, updated CHANGELOG and store documentation.

Documentation definitely needs some help, I'm open to suggestions.

I will add more logging to initial sync in separare PR (:

I couldn't get to this one yet, however it would probably better since you have more context on the component, thanks a lot.

[bwplotka]

2 more nits and LGTM!

[FUSAKLA]

👍 great, thanks for doing this. I remember discussing the speedup of HTTP server startup with store but it eventually did not happen. Not sure why exactly.

Agree with @bwplotka on those two comments otherwise LGTM!

[squat]

🌮

[bwplotka]

Nice! Finally we have metrics on Thanos startup!

Let's make sure we have proper logging and metrics on sync - happy to attack this on Monday.

[bwplotka]

Good work @kakkoyun thanks! ❤️