Using leader election with webhook manager

[SchSeba]

When we create a manager with leader election and a webhook only one manager start but looking into the service the manger creates for the webhook he add all the pod.

Proposed solution:

Use the label statefulset.kubernetes.io/pod-name= for the webhook service and when the election change update the service label(or recreate the service)

More general solution can be to have a general label like controller-leader and just update the leader pod with this label.

[DirectXMan12]

So, we shouldn't do that, since webhooks support active-active HA quite easily. It should be possible to start the webhook server without starting the controllers. For that we probably need the concept of things which are active-passive vs active-active.

@mengqiy FYI

[DirectXMan12]

/kind bug
/priority important-soon

This is something we intended to work properly, but doesn't, so I'm treating it as a bug here.

[pytimer]

@DirectXMan12 I have the problem similar to @SchSeba said, but i use Deployment to deploy manager in Kubernetes cluster. I add a readinessProbe in manager yaml, below is my yaml:

name: manager
readinessProbe:
  tcpSocket:
    port: 9876
  initialDelaySeconds: 5
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 3
  timeoutSeconds: 2

This's works ok for me, but i see manager have the http: TLS handshake error from 10.244.3.1:47680: EOF log every 10 seconds, so i want to use httpGet instead of tcpSocket, but i have no idea how to setting.

Can we add a health check api in webhook, and user can add readinessProbe in manager.yaml when use leader election?

[DirectXMan12]

that's a different issue, but yeah, we definitely want to support registering generic health checks, etc.

[pytimer]

@DirectXMan12 If use leader election with manager, the service will add all pods, but only one pod can works. So i add a health check to listen webhook port.

That's good If it support health checks, but before support health checks, i maybe needs above method, otherwise the service connection refused. If you know other ways to solve it, hope you can tell me, thanks.

Are there any related documents or links about health checks?

[lilic]

So, we shouldn't do that, since webhooks support active-active HA quite easily. It should be possible to start the webhook server without starting the controllers. For that we probably need the concept of things which are active-passive vs active-active.

@DirectXMan12 So we want to start the webhooks and managers everywhere, and not let the leader election block it. But let the non-leader ones be passive. IIUC?

[SchSeba]

maybe we can implement it in a different way.

if we want only the leader to be active for the webhook we can create the service with a special label and when we took the leader it will update the pod with the relevant label?

[DirectXMan12]

Basically:

Start the manager everywhere. For most (I'd hazard all, but you never know) you want the webhook to be active on every replica (webhooks are almost always stateless), but you want the controllers to only be active on the leader replica.

[mengqiy]

Closing by #424