Bug 1817075: operator/controller pods: faster leader elections

[yuqi-zhang]

As a temporary solution in lieu of graceful shutdowns, have the
operator and controller pods use faster defaults for lease/renew/
retry, which shouldn't otherwise notably affect operation other
than the new pod acquiring a new lease faster during upgrades,
saving some waiting time.

Example in CVO: https://github.com/openshift/cluster-version-operator/blob/706a0062cade643767e95da2e6142823e8399986/pkg/start/start.go#L51

[openshift-ci]

@yuqi-zhang: This pull request references Bugzilla bug 1817075, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target release (4.8.0) matches configured target release for branch (4.8.0)
• bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @mike-nguyen

Details

In response to this:

Bug 1817075: operator/controller pods: faster leader elections

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@yuqi-zhang: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/okd-e2e-aws	84aac80	link	/test okd-e2e-aws
ci/prow/e2e-aws-disruptive	84aac80	link	/test e2e-aws-disruptive

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[sinnykumari]

From MCC log of upgrade job from this PR, I see time needed to acquire leader lease for upgrade job is ~1 min which was earlier ~2 mins

I0608 07:30:55.581365       1 leaderelection.go:243] attempting to acquire leader lease openshift-machine-config-operator/machine-config-controller...
I0608 07:31:34.172079       1 leaderelection.go:253] successfully acquired lease openshift-machine-config-operator/machine-config-controller

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sinnykumari, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [sinnykumari,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@yuqi-zhang: All pull requests linked via external trackers have merged:

• openshift/machine-config-operator#2603

Bugzilla bug 1817075 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1817075: operator/controller pods: faster leader elections

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[deads2k]

This isn't the direction we want to go. In order to handle short-term network (or other) disruption, all clients (including those using leases) should be able to make it at least 60 seconds without the kube-apiserver available. To have rapid hand-off in normal cases, clients using leases need to make use of graceful release.

We found this general case (not this one specifically) during single-node, when a 60 second disruption led to mass failures.

Is the alternative to this PR, providing graceful lease release already open so we can assess the size?