OCPBUGS-14246: Adding runbook URL for ClusterVersionOperatorDown

[ssierrab]

No description provided.

[openshift-ci-robot]

@ssierrab: This pull request references Jira Issue OCPBUGS-14246, which is invalid:

• expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Hi @ssierrab. Thanks for your PR.

I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[DavidHurta]

/cc

[DavidHurta]

/ok-to-test

[DavidHurta]

/jira refresh

[openshift-ci-robot]

@DavidHurta: This pull request references Jira Issue OCPBUGS-14246, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.21.0) matches configured target version for branch (4.21.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @jiajliu

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: DavidHurta, ssierrab

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [DavidHurta]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[DavidHurta]

@jiajliu, currently assigned as the QA contact for the bug, there was a PR that partly addressed the OCPBUGS-14246 - #1036

There is a comment from Petr on that PR:

Although this partially addresses OCPBUGS-14246, this PR does not change any code. It can be tested when OCPBUGS-14246 is fully fixed.

I believe we can now verify the entire "All Critical Alert Rules must have runbbok_url annotation" bug with this PR.

This PR has flown in mid-sprint without notice in the planning. Its priority is Normal. Feel free to verify the PR based on your discretion given the priority, Shift Week, and so on. I believe it is not urgent, thus it can wait a bit, no worries.

[jiajliu]

cc @dis016 for ClusterVersionOperatorDown alert's runbook URL

[jiajliu]

There is a comment from Petr on that PR:

Although this partially addresses OCPBUGS-14246, this PR does not change any code. It can be tested when OCPBUGS-14246 is fully fixed.

We will check both of prs in this sprint, but maybe after shift week:)

[dis016]

Test Scenario: Verify runbook url is available for alert ClusterVersionOperatorDown

1. Install a 4.21 cluster with the PR.

launch 4.21.0-0.nightly-2025-10-30-060549,openshift/cluster-version-operator#1250 gcp

dinesh@Dineshs-MacBook-Pro ~ % oc get clusterversion 
NAME      VERSION                                                AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.21.0-0-2025-11-03-161951-test-ci-ln-44d537b-latest   True        False         4m20s   Cluster version is 4.21.0-0-2025-11-03-161951-test-ci-ln-44d537b-latest
dinesh@Dineshs-MacBook-Pro ~ % 

2. Scale Down the CVO to generate the alert.

dinesh@Dineshs-MacBook-Pro ~ % oc get pods -n openshift-cluster-version
NAME                                       READY   STATUS    RESTARTS   AGE
cluster-version-operator-cf9ffc987-mkxqr   1/1     Running   0          45m
dinesh@Dineshs-MacBook-Pro ~ % 


dinesh@Dineshs-MacBook-Pro ~ % oc scale deployment cluster-version-operator --replicas=0 -n openshift-cluster-version
deployment.apps/cluster-version-operator scaled
dinesh@Dineshs-MacBook-Pro ~ % 

dinesh@Dineshs-MacBook-Pro ~ % oc get pods -n openshift-cluster-version                                              
No resources found in openshift-cluster-version namespace.
dinesh@Dineshs-MacBook-Pro ~ %

3. Verify alert is generated and runbook url is available.

dinesh@Dineshs-MacBook-Pro ~ % token=$(oc -n openshift-monitoring create token prometheus-k8s)   
dinesh@Dineshs-MacBook-Pro ~ % route=`oc get route prometheus-k8s -n openshift-monitoring -ojsonpath='{.status.ingress[].host}'`  
dinesh@Dineshs-MacBook-Pro ~ % curl -s -k -H "Authorization: Bearer $token" https://$route/api/v1/alerts | jq -r '.data.alerts[]| select(.labels.alertname == "ClusterVersionOperatorDown")'   
{
  "labels": {
    "alertname": "ClusterVersionOperatorDown",
    "namespace": "openshift-cluster-version",
    "severity": "critical"
  },
  "annotations": {
    "description": "The operator may be down or disabled. The cluster will not be kept up to date and upgrades will not be possible. Inspect the openshift-cluster-version namespace for events or changes to the cluster-version-operator deployment or pods to diagnose and repair.  For more information refer to https://console-openshift-console.apps.ci-ln-44d537b-72292.gcp-2.ci.openshift.org/k8s/cluster/projects/openshift-cluster-version.",
    "runbook_url": "https://github.com/openshift/runbooks/blob/master/alerts/cluster-version-operator/ClusterVersionOperatorDown.md",
    "summary": "Cluster version operator has disappeared from Prometheus target discovery."
  },
  "state": "firing",
  "activeAt": "2025-11-03T17:36:03.011708277Z",
  "value": "1e+00"
}
dinesh@Dineshs-MacBook-Pro ~ % 

[dis016]

/verified by @dis016

[openshift-ci-robot]

@dis016: This PR has been marked as verified by @dis016.

In response to this:

/verified by @dis016

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@ssierrab: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@ssierrab: Jira Issue OCPBUGS-14246: All pull requests linked via external trackers have merged:

• openshift/cluster-version-operator#1036
• openshift/cluster-version-operator#1250

Jira Issue OCPBUGS-14246 has been moved to the MODIFIED state.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-robot]

Fix included in accepted release 4.21.0-0.nightly-2025-11-05-234508