Skip to content

Conversation

@LalatenduMohanty
Copy link
Member

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 29, 2021
@openshift-ci openshift-ci bot requested review from sdodson and vrutkovs July 29, 2021 19:52
@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2021
@LalatenduMohanty LalatenduMohanty changed the title [WIP] Bug: 1985802 Updating the lease, renew, retry duration for CVO [WIP] Bug 1985802 : Updating the lease, renew, retry duration for CVO Jul 29, 2021
@LalatenduMohanty
Copy link
Member Author

/bugzilla refresh

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 29, 2021

@LalatenduMohanty: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@LalatenduMohanty LalatenduMohanty changed the title [WIP] Bug 1985802 : Updating the lease, renew, retry duration for CVO [WIP] Bug 1985802: Updating the lease, renew, retry duration for CVO Jul 29, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 29, 2021

@LalatenduMohanty: This pull request references Bugzilla bug 1985802, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @jianlinliu

Details

In response to this:

[WIP] Bug 1985802: Updating the lease, renew, retry duration for CVO

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jul 29, 2021
@openshift-ci openshift-ci bot requested a review from jianlinliu July 29, 2021 20:08
@sdodson
Copy link
Member

sdodson commented Jul 29, 2021

/uncc

@openshift-ci openshift-ci bot removed the request for review from sdodson July 29, 2021 20:48
@pamoedom
Copy link

/lgtm
/label qe-approved
/bugzilla cc-qa

[Pre-Merge QA Testing]

  • Custom version that includes this PR:
$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         68m     Cluster version is 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest
  • Status of the cluster after fresh installation:
$ oc get nodes
NAME                                                     STATUS   ROLES           AGE   VERSION
master-00.pamoedo-snotest3.qe.devcluster.openshift.com   Ready    master,worker   86m   v1.21.1+38b3ecc

$ oc get co
NAME                                       VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      67m
baremetal                                  4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
cloud-controller-manager                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
cloud-credential                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      105m
cluster-autoscaler                         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
config-operator                            4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
console                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      69m
csi-snapshot-controller                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
dns                                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
etcd                                       4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
image-registry                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      72m
ingress                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      73m
insights                                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      73m
kube-apiserver                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      73m
kube-controller-manager                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
kube-scheduler                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
kube-storage-version-migrator              4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
machine-api                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
machine-approver                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
machine-config                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
marketplace                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
monitoring                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      69m
network                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      81m
node-tuning                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
openshift-apiserver                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      69m
openshift-controller-manager               4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
openshift-samples                          4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      74m
operator-lifecycle-manager                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
operator-lifecycle-manager-catalog         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
operator-lifecycle-manager-packageserver   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      75m
service-ca                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
storage                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
  • Forced a kubeapiserver redeploy:
$ oc patch kubeapiserver/cluster --type merge -p "{\"spec\":{\"forceRedeploymentReason\":\"Forcing new revision with random number $RANDOM to make message unique\"}}"
kubeapiserver.operator.openshift.io/cluster patched

$ oc describe kubeapiserver/cluster | grep Redeployment
        f:forceRedeploymentReason:
  Force Redeployment Reason:  Forcing new revision with random number 14640 to make message unique
  • After some minutes, the clusteroperators finished to progress and all of them are properly running as expected:
$ oc get nodes
NAME                                                     STATUS   ROLES           AGE    VERSION
master-00.pamoedo-snotest3.qe.devcluster.openshift.com   Ready    master,worker   124m   v1.21.1+38b3ecc

$ oc get co
NAME                                       VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      115m
baremetal                                  4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
cloud-controller-manager                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
cloud-credential                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      153m
cluster-autoscaler                         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
config-operator                            4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
console                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      117m
csi-snapshot-controller                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
dns                                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
etcd                                       4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
image-registry                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      120m
ingress                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      122m
insights                                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      122m
kube-apiserver                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      121m
kube-controller-manager                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
kube-scheduler                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
kube-storage-version-migrator              4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
machine-api                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
machine-approver                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
machine-config                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
marketplace                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
monitoring                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      117m
network                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      129m
node-tuning                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
openshift-apiserver                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      117m
openshift-controller-manager               4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
openshift-samples                          4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      123m
operator-lifecycle-manager                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
operator-lifecycle-manager-catalog         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
operator-lifecycle-manager-packageserver   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      123m
service-ca                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
storage                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m

Best Regards.

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Jul 30, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jul 30, 2021

@pamoedom: This pull request references Bugzilla bug 1985802, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.9.0) matches configured target release for branch (4.9.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @pamoedom

Details

In response to this:

/lgtm
/label qe-approved
/bugzilla cc-qa

[Pre-Merge QA Testing]

  • Custom version that includes this PR:
$ oc get clusterversion
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         68m     Cluster version is 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest
  • Status of the cluster after fresh installation:
$ oc get nodes
NAME                                                     STATUS   ROLES           AGE   VERSION
master-00.pamoedo-snotest3.qe.devcluster.openshift.com   Ready    master,worker   86m   v1.21.1+38b3ecc

$ oc get co
NAME                                       VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      67m
baremetal                                  4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
cloud-controller-manager                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
cloud-credential                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      105m
cluster-autoscaler                         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
config-operator                            4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
console                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      69m
csi-snapshot-controller                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
dns                                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
etcd                                       4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
image-registry                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      72m
ingress                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      73m
insights                                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      73m
kube-apiserver                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      73m
kube-controller-manager                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
kube-scheduler                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
kube-storage-version-migrator              4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
machine-api                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
machine-approver                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
machine-config                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
marketplace                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      78m
monitoring                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      69m
network                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      81m
node-tuning                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
openshift-apiserver                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      69m
openshift-controller-manager               4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      79m
openshift-samples                          4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      74m
operator-lifecycle-manager                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
operator-lifecycle-manager-catalog         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
operator-lifecycle-manager-packageserver   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      75m
service-ca                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
storage                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      80m
  • Forced a kubeapiserver redeploy:
$ oc patch kubeapiserver/cluster --type merge -p "{\"spec\":{\"forceRedeploymentReason\":\"Forcing new revision with random number $RANDOM to make message unique\"}}"
kubeapiserver.operator.openshift.io/cluster patched

$ oc describe kubeapiserver/cluster | grep Redeployment
       f:forceRedeploymentReason:
 Force Redeployment Reason:  Forcing new revision with random number 14640 to make message unique
  • After some minutes, the clusteroperators finished to progress and all of them are properly running as expected:
$ oc get nodes
NAME                                                     STATUS   ROLES           AGE    VERSION
master-00.pamoedo-snotest3.qe.devcluster.openshift.com   Ready    master,worker   124m   v1.21.1+38b3ecc

$ oc get co
NAME                                       VERSION                                                  AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      115m
baremetal                                  4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
cloud-controller-manager                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
cloud-credential                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      153m
cluster-autoscaler                         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
config-operator                            4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
console                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      117m
csi-snapshot-controller                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
dns                                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
etcd                                       4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
image-registry                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      120m
ingress                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      122m
insights                                   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      122m
kube-apiserver                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      121m
kube-controller-manager                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
kube-scheduler                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
kube-storage-version-migrator              4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
machine-api                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
machine-approver                           4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
machine-config                             4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
marketplace                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      126m
monitoring                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      117m
network                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      129m
node-tuning                                4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
openshift-apiserver                        4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      117m
openshift-controller-manager               4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      127m
openshift-samples                          4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      123m
operator-lifecycle-manager                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
operator-lifecycle-manager-catalog         4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
operator-lifecycle-manager-packageserver   4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      123m
service-ca                                 4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m
storage                                    4.8.0-0.ci.test-2021-07-30-095611-ci-ln-dyxsvsb-latest   True        False         False      128m

Best Regards.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested a review from pamoedom July 30, 2021 13:42
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jul 30, 2021
@LalatenduMohanty LalatenduMohanty changed the title [WIP] Bug 1985802: Updating the lease, renew, retry duration for CVO Bug 1985802: Updating the lease, renew, retry duration for CVO Aug 3, 2021
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 3, 2021
@LalatenduMohanty
Copy link
Member Author

/retest

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 3, 2021
@LalatenduMohanty LalatenduMohanty changed the title Bug 1985802: Updating the lease, renew, retry duration for CVO Bug 1985802: Updating the lease, renew, retry duration Aug 4, 2021
For both single node clusters (SNO) and non-SNO clusters leader election
need to tolerate 60s worth of disruption [2].

The default in library-go has been upgaded to handle this case in 4.9 [1].
In essence, the kube-apiserver downtime tolerance is floor(renewDeadline/retryPeriod)*retryPeriod-retryPeriod.
Recommended defaults are

LeaseDuration=137s
RenewDealine=107s
RetryPeriod=26s.

These are the configurable values in k8s.io/client-go based leases and controller-runtime exposes them. This gives us

clock skew tolerance == 30s
kube-apiserver downtime tolerance == 78s
worst non-graceful lease reacquisition == 163s
worst graceful lease reacquisition == 26s

For learning more about client side leader election config refer to [3]

[1] https://github.com/openshift/library-go/blob/4b9033d00d37b88393f837a88ff541a56fd13621/pkg/config/leaderelection/leaderelection.go#L84
[2] https://github.com/openshift/enhancements/blob/c91e5ae66ed590405fb205433131c6a75f05610d/CONVENTIONS.md#high-availability
[3] https://pkg.go.dev/k8s.io/client-go/tools/leaderelection#LeaderElectionConfig

Signed-off-by: Lalatendu Mohanty <[email protected]>
Copy link
Member

@wking wking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 4, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 4, 2021

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: LalatenduMohanty, pamoedom, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [LalatenduMohanty,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

7 similar comments
@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-bot
Copy link
Contributor

/retest-required

Please review the full test history for this PR and help us cut down flakes.

@openshift-ci openshift-ci bot merged commit 0ec39d9 into openshift:master Aug 5, 2021
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Aug 5, 2021

@LalatenduMohanty: All pull requests linked via external trackers have merged:

Bugzilla bug 1985802 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1985802: Updating the lease, renew, retry duration

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants