OCPBUGS-306: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes #826

openshift-cherrypick-robot · 2022-08-18T20:34:31Z

This is an automated cherry-pick of #818

/assign wking

…ns[name=operator] changes David and Stephen identified an uneccessary delay [1]: * 9:42:00, CVO gives up on Kube API server ClusterOperator [2] * 9:42:47, Kube API server operator achieves 4.12 [3] * 9:46:22, after a cool-off sleep, the CVO starts in on a new manifest graph-walk attempt [4] * 9:46:34, CVO notices that the Kube API server ClusterOperator is happy [5] The 3+ minute delay from 9:42:47 to 9:46:22 is not helpful, and we've probably had delays like this since my old e02d148 (pkg/cvo/internal/operatorstatus: Replace wait-for with single-shot "is it alive now?", 2021-05-13, openshift#560), which landed in 4.6. This commit introduces a "ClusterOperator bumped versions[name=operator]" trigger to break out of the cool-off sleep. There's plenty of room to be more precise here. For example, you could currently have a versions[name=operator] bump during the sync loop that the CVO did notice, and that queued notification will break from the sleep and trigger a possible useless reconciliation round while we wait on some other resource. You could drain the notification queue before the sleep to avoid that, but you wouldn't want to drain new-work notifications, and I haven't done the work required to be able to split those apart. I'm only looking at ClusterOperator at the moment, because of the many types the CVO manages, ClusterOperator is the one we most frequently wait on, as large cluster components take their time updating. It's possible but less likely that we'd want similar triggers for additional types in the future (Deployment, etc.), if/when those types develop more elaborate "is the in-cluster resource sufficient happy?" checks. The panic-backed type casting in clusterOperatorInterfaceVersionOrDie also feel like a hack, but I wasn't able to find a cleaner way to get at the structured information I want. Improvements welcome :) [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2117033#c1 [2]: From Loki: E0808 09:42:00.022500 1 task.go:117] error running apply for clusteroperator "kube-apiserver" (107 of 806): Cluster operator kube-apiserver is updating versions [3]: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.12-upgrade-from-stable-4.11-e2e-gcp-sdn-upgrade/1556564581915037696/artifacts/e2e-gcp-sdn-upgrade/openshift-e2e-test/build-log.txt | grep 'clusteroperator/kube-apiserver versions:' Aug 08 09:33:48.603 I clusteroperator/kube-apiserver versions: raw-internal 4.11.0-rc.7 -> 4.12.0-0.ci-2022-08-07-192220 Aug 08 09:42:47.917 I clusteroperator/kube-apiserver versions: operator 4.11.0-rc.7 -> 4.12.0-0.ci-2022-08-07-192220 [4]: From Loki: I0808 09:46:22.998344 1 sync_worker.go:850] apply: 4.12.0-0.ci-2022-08-07-192220 on generation 3 in state Updating at attempt 5 [5]: From Loki: I0808 09:46:34.556374 1 sync_worker.go:973] Done syncing for clusteroperator "kube-apiserver" (107 of 806)

openshift-ci · 2022-08-18T20:34:37Z

@openshift-cherrypick-robot: An error was encountered cloning bug for cherrypick for bug 2117033 on the Bugzilla server at https://bugzilla.redhat.com. No known errors were detected, please see the full error message for details.

Full error message.


response code 401 not 200

Please contact an administrator to resolve this issue, then request a bug refresh with /bugzilla refresh.

Details

In response to this:

[release-4.11] Bug 2117033: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-08-18T20:46:12Z

@openshift-cherrypick-robot: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

Details

In response to this:

OCPBUGS-306: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2022-08-18T20:46:14Z

@openshift-cherrypick-robot: This pull request references [Jira Issue OCPBUGS-306](https://issues.redhat.com//browse/OCPBUGS-306), which is invalid:

expected dependent Jira Issue OCPBUGS-305 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is New instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

This is an automated cherry-pick of #818

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2022-08-18T20:46:51Z

/jira refresh

openshift-ci-robot · 2022-08-18T20:46:57Z

@wking: This pull request references [Jira Issue OCPBUGS-306](https://issues.redhat.com//browse/OCPBUGS-306), which is valid. The bug has been moved to the POST state.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.11.z) matches configured target version for branch (4.11.z)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST)
dependent bug Jira Issue OCPBUGS-305 is in the state Verified (Done), which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-305 targets the "4.12.0" version, which is one of the valid target versions: 4.12.0
bug has dependents

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking

Clean pick:

/lgtm

openshift-ci · 2022-08-18T20:47:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: openshift-cherrypick-robot, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LalatenduMohanty · 2022-08-18T20:55:09Z

/label backport-risk-assessed

wking · 2022-08-19T05:08:06Z

Early KubeAggregatedAPI* errors are orthogonal.

/override ci/prow/e2e-agnostic

Watchdog blips while compute is rolling:

also orthogonal:

/override ci/prow/e2e-agnostic-upgrade

openshift-ci · 2022-08-19T05:08:41Z

@wking: Overrode contexts on behalf of wking: ci/prow/e2e-agnostic, ci/prow/e2e-agnostic-upgrade

Details

In response to this:

Early KubeAggregatedAPI* errors are orthogonal.

/override ci/prow/e2e-agnostic

Watchdog blips while compute is rolling:

also orthogonal:

/override ci/prow/e2e-agnostic-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jiajliu · 2022-08-24T01:32:36Z

/label cherry-pick-approved

openshift-ci-robot · 2022-08-24T05:16:32Z

/retest-required

Remaining retests: 2 against base HEAD 0520e51 and 8 for PR HEAD 205f3b9 in total

openshift-ci-robot · 2022-08-24T09:09:55Z

/retest-required

Remaining retests: 1 against base HEAD 0520e51 and 7 for PR HEAD 205f3b9 in total

openshift-ci-robot · 2022-08-24T12:19:18Z

/retest-required

Remaining retests: 0 against base HEAD 0520e51 and 6 for PR HEAD 205f3b9 in total

wking · 2022-08-24T17:06:27Z

Watchdog gaps are orthogonal.

/override ci/prow/e2e-agnostic-upgrade

openshift-ci · 2022-08-24T17:08:16Z

@wking: Overrode contexts on behalf of wking: ci/prow/e2e-agnostic-upgrade

Details

In response to this:

Watchdog gaps are orthogonal.

/override ci/prow/e2e-agnostic-upgrade

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-08-24T17:08:22Z

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agnostic	`205f3b9`	link	true	`/test e2e-agnostic`
ci/prow/e2e-agnostic-upgrade	`205f3b9`	link	true	`/test e2e-agnostic-upgrade`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-ci-robot · 2022-08-24T17:11:15Z

@openshift-cherrypick-robot: All pull requests linked via external trackers have merged:

openshift/cluster-version-operator#826

[Jira Issue OCPBUGS-306](https://issues.redhat.com//browse/OCPBUGS-306) has been moved to the MODIFIED state.

Details

In response to this:

This is an automated cherry-pick of #818

/assign wking

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-cherrypick-robot mentioned this pull request Aug 18, 2022

Bug 2117033: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes #818

Merged

openshift-ci bot assigned wking Aug 18, 2022

openshift-ci bot requested review from LalatenduMohanty and vrutkovs August 18, 2022 20:35

wking changed the title ~~[release-4.11] Bug 2117033: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes~~ OCPBUGS-306: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes Aug 18, 2022

openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 18, 2022

wking approved these changes Aug 18, 2022

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 18, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 18, 2022

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Aug 18, 2022

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Aug 24, 2022

openshift-merge-robot merged commit bd8aa51 into openshift:release-4.11 Aug 24, 2022

OCPBUGS-306: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes #826

OCPBUGS-306: pkg/cvo/sync_worker: Trigger new sync round on ClusterOperator versions[name=operator] changes #826

Uh oh!

Conversation

openshift-cherrypick-robot commented Aug 18, 2022

Uh oh!

openshift-ci bot commented Aug 18, 2022

Uh oh!

openshift-ci bot commented Aug 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 18, 2022 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking commented Aug 18, 2022

Uh oh!

openshift-ci-robot commented Aug 18, 2022 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wking left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Aug 18, 2022

Uh oh!

LalatenduMohanty commented Aug 18, 2022

Uh oh!

wking commented Aug 19, 2022

Uh oh!

openshift-ci bot commented Aug 19, 2022

Uh oh!

jiajliu commented Aug 24, 2022

Uh oh!

openshift-ci-robot commented Aug 24, 2022

Uh oh!

openshift-ci-robot commented Aug 24, 2022

Uh oh!

openshift-ci-robot commented Aug 24, 2022

Uh oh!

wking commented Aug 24, 2022

Uh oh!

openshift-ci bot commented Aug 24, 2022

Uh oh!

openshift-ci bot commented Aug 24, 2022

Uh oh!

openshift-ci-robot commented Aug 24, 2022 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

openshift-ci bot commented Aug 18, 2022 •

edited

Loading

openshift-ci-robot commented Aug 18, 2022 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 18, 2022 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Aug 24, 2022 •

edited by openshift-ci bot

Loading