OCPBUGS-63478: Check operators for pod deletion

[weliang1]

[sig-network] pods should successfully create sandboxes by adding pod to network - failed in two Prow jobs recently in https://issues.redhat.com/browse/OCPBUGS-63478

Prow Job1:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade/1980375618641989632

Test Log:

namespace/openshift-etcd node/ci-op-tz2f8kj9-fbbf2-j6brz-master-0 pod/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 hmsg/b1ad8ffd50 - 297.49 seconds after deletion - firstTimestamp/2025-10-20T22:22:31Z interesting/true lastTimestamp/2025-10-20T22:22:31Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0_openshift-etcd_4f678bb2-10f6-47f8-b080-36cbae9d50a4_0(09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0): error adding pod openshift-etcd_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0" Netns:"/var/run/netns/4ab6fd45-b6fa-4709-bb88-a4e7a81b8b3e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-etcd;K8S_POD_NAME=etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0;K8S_POD_INFRA_CONTAINER_ID=09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0;K8S_POD_UID=4f678bb2-10f6-47f8-b080-36cbae9d50a4" Path:"" ERRORED: error configuring pod [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0] networking: Multus: [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0/4f678bb2-10f6-47f8-b080-36cbae9d50a4]: error waiting for pod: Get "https://api-int.ci-op-tz2f8kj9-fbbf2.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443/api/v1/namespaces/openshift-etcd/pods/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0?timeout=1m0s": context deadline exceeded
': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}}

Prow job2:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.21-upgrade-from-stable-4.20-e2e-metal-ipi-ovn-upgrade/1979889012780830720

Test log:

namespace/openshift-kube-apiserver node/master-2 pod/revision-pruner-9-master-2 hmsg/cde531b9d8 - 145.44 seconds after deletion - firstTimestamp/2025-10-19T15:01:26Z interesting/true lastTimestamp/2025-10-19T15:01:26Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-9-master-2_openshift-kube-apiserver_a08d5e14-b027-4a1a-ac4e-1e58d9103c01_0(a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a): error adding pod openshift-kube-apiserver_revision-pruner-9-master-2 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a" Netns:"/var/run/netns/55eab7e4-f851-47ac-9317-af272dacca1b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-kube-apiserver;K8S_POD_NAME=revision-pruner-9-master-2;K8S_POD_INFRA_CONTAINER_ID=a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a;K8S_POD_UID=a08d5e14-b027-4a1a-ac4e-1e58d9103c01" Path:"" ERRORED: error configuring pod [openshift-kube-apiserver/revision-pruner-9-master-2] networking: Multus: [openshift-kube-apiserver/revision-pruner-9-master-2/a08d5e14-b027-4a1a-ac4e-1e58d9103c01]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod revision-pruner-9-master-2 in out of cluster comm: SetNetworkStatus: failed to update the pod revision-pruner-9-master-2 in out of cluster comm: status update failed for pod /: Get "https://api-int.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/revision-pruner-9-master-2?timeout=1m0s": dial tcp: lookup api-int.ostest.test.metalkube.org on 192.168.111.1:53: no such host
': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}

For deleted pods, the code never checks if the failure occurred during operator Progressing. It only checks the time difference, which results in a hard failure at > 5 seconds. According to the code's own logic, if the etcd/dns/network operator was Progressing, sandbox failures should be treated more leniently (as flakes), but this check is missing in the deleted pod branch.

The fix adds operator Progressing checks to the deleted pod branch

[openshift-ci-robot]

@weliang1: This pull request references Jira Issue OCPBUGS-63478, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.21.0) matches configured target version for branch (4.21.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

[sig-network] pods should successfully create sandboxes by adding pod to network - failed in two Prow jobs recently in https://issues.redhat.com/browse/OCPBUGS-63478

Prow Job1:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade/1980375618641989632

Test Log:

namespace/openshift-etcd node/ci-op-tz2f8kj9-fbbf2-j6brz-master-0 pod/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 hmsg/b1ad8ffd50 - 297.49 seconds after deletion - firstTimestamp/2025-10-20T22:22:31Z interesting/true lastTimestamp/2025-10-20T22:22:31Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0_openshift-etcd_4f678bb2-10f6-47f8-b080-36cbae9d50a4_0(09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0): error adding pod openshift-etcd_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0" Netns:"/var/run/netns/4ab6fd45-b6fa-4709-bb88-a4e7a81b8b3e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-etcd;K8S_POD_NAME=etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0;K8S_POD_INFRA_CONTAINER_ID=09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0;K8S_POD_UID=4f678bb2-10f6-47f8-b080-36cbae9d50a4" Path:"" ERRORED: error configuring pod [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0] networking: Multus: [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0/4f678bb2-10f6-47f8-b080-36cbae9d50a4]: error waiting for pod: Get "https://api-int.ci-op-tz2f8kj9-fbbf2.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443/api/v1/namespaces/openshift-etcd/pods/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0?timeout=1m0s": context deadline exceeded
': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}}

Prow job2:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.21-upgrade-from-stable-4.20-e2e-metal-ipi-ovn-upgrade/1979889012780830720

Test log:

namespace/openshift-kube-apiserver node/master-2 pod/revision-pruner-9-master-2 hmsg/cde531b9d8 - 145.44 seconds after deletion - firstTimestamp/2025-10-19T15:01:26Z interesting/true lastTimestamp/2025-10-19T15:01:26Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-9-master-2_openshift-kube-apiserver_a08d5e14-b027-4a1a-ac4e-1e58d9103c01_0(a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a): error adding pod openshift-kube-apiserver_revision-pruner-9-master-2 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a" Netns:"/var/run/netns/55eab7e4-f851-47ac-9317-af272dacca1b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-kube-apiserver;K8S_POD_NAME=revision-pruner-9-master-2;K8S_POD_INFRA_CONTAINER_ID=a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a;K8S_POD_UID=a08d5e14-b027-4a1a-ac4e-1e58d9103c01" Path:"" ERRORED: error configuring pod [openshift-kube-apiserver/revision-pruner-9-master-2] networking: Multus: [openshift-kube-apiserver/revision-pruner-9-master-2/a08d5e14-b027-4a1a-ac4e-1e58d9103c01]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod revision-pruner-9-master-2 in out of cluster comm: SetNetworkStatus: failed to update the pod revision-pruner-9-master-2 in out of cluster comm: status update failed for pod /: Get "https://api-int.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/revision-pruner-9-master-2?timeout=1m0s": dial tcp: lookup api-int.ostest.test.metalkube.org on 192.168.111.1:53: no such host
': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}

For deleted pods, the code never checks if the failure occurred during operator Progressing. It only checks the time difference, which results in a hard failure at > 5 seconds. According to the code's own logic, if the etcd/dns/network operator was Progressing, sandbox failures should be treated more leniently (as flakes), but this check is missing in the deleted pod branch.

The fix adds operator Progressing checks to the deleted pod branch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[weliang1]

/retest

[dgoodwin]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, weliang1

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dgoodwin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[weliang1]

/verified later @weliang

[openshift-ci-robot]

@weliang1: This PR has been marked to be verified later by @weliang.

In response to this:

/verified later @weliang

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 4f47dc4 and 2 for PR HEAD 5f9cd4d in total

[weliang1]

/test e2e-aws-ovn-serial-2of2

[openshift-ci]

@weliang1: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@weliang1: Jira Issue OCPBUGS-63478: All pull requests linked via external trackers have merged:

• openshift/origin#30457

This pull request has the verified-later tag and will need to be manually moved to VERIFIED after testing. Jira Issue OCPBUGS-63478 has been moved to the MODIFIED state.

In response to this:

[sig-network] pods should successfully create sandboxes by adding pod to network - failed in two Prow jobs recently in https://issues.redhat.com/browse/OCPBUGS-63478

Prow Job1:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.20-upgrade-from-stable-4.19-e2e-azure-ovn-upgrade/1980375618641989632

Test Log:

namespace/openshift-etcd node/ci-op-tz2f8kj9-fbbf2-j6brz-master-0 pod/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 hmsg/b1ad8ffd50 - 297.49 seconds after deletion - firstTimestamp/2025-10-20T22:22:31Z interesting/true lastTimestamp/2025-10-20T22:22:31Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0_openshift-etcd_4f678bb2-10f6-47f8-b080-36cbae9d50a4_0(09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0): error adding pod openshift-etcd_etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0" Netns:"/var/run/netns/4ab6fd45-b6fa-4709-bb88-a4e7a81b8b3e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-etcd;K8S_POD_NAME=etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0;K8S_POD_INFRA_CONTAINER_ID=09b24379f8fe8571b7a5eda9a668692a1f31d31d76772a7dabdb19c9778a43d0;K8S_POD_UID=4f678bb2-10f6-47f8-b080-36cbae9d50a4" Path:"" ERRORED: error configuring pod [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0] networking: Multus: [openshift-etcd/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0/4f678bb2-10f6-47f8-b080-36cbae9d50a4]: error waiting for pod: Get "https://api-int.ci-op-tz2f8kj9-fbbf2.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX:6443/api/v1/namespaces/openshift-etcd/pods/etcd-guard-ci-op-tz2f8kj9-fbbf2-j6brz-master-0?timeout=1m0s": context deadline exceeded
': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}}

Prow job2:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.21-upgrade-from-stable-4.20-e2e-metal-ipi-ovn-upgrade/1979889012780830720

Test log:

namespace/openshift-kube-apiserver node/master-2 pod/revision-pruner-9-master-2 hmsg/cde531b9d8 - 145.44 seconds after deletion - firstTimestamp/2025-10-19T15:01:26Z interesting/true lastTimestamp/2025-10-19T15:01:26Z reason/FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_revision-pruner-9-master-2_openshift-kube-apiserver_a08d5e14-b027-4a1a-ac4e-1e58d9103c01_0(a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a): error adding pod openshift-kube-apiserver_revision-pruner-9-master-2 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a" Netns:"/var/run/netns/55eab7e4-f851-47ac-9317-af272dacca1b" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=openshift-kube-apiserver;K8S_POD_NAME=revision-pruner-9-master-2;K8S_POD_INFRA_CONTAINER_ID=a6bb2c3a3e4af84920d82a41b695d764a677a86e4acc11d80669f9351228444a;K8S_POD_UID=a08d5e14-b027-4a1a-ac4e-1e58d9103c01" Path:"" ERRORED: error configuring pod [openshift-kube-apiserver/revision-pruner-9-master-2] networking: Multus: [openshift-kube-apiserver/revision-pruner-9-master-2/a08d5e14-b027-4a1a-ac4e-1e58d9103c01]: error setting the networks status: SetPodNetworkStatusAnnotation: failed to update the pod revision-pruner-9-master-2 in out of cluster comm: SetNetworkStatus: failed to update the pod revision-pruner-9-master-2 in out of cluster comm: status update failed for pod /: Get "https://api-int.ostest.test.metalkube.org:6443/api/v1/namespaces/openshift-kube-apiserver/pods/revision-pruner-9-master-2?timeout=1m0s": dial tcp: lookup api-int.ostest.test.metalkube.org on 192.168.111.1:53: no such host
': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}

For deleted pods, the code never checks if the failure occurred during operator Progressing. It only checks the time difference, which results in a hard failure at > 5 seconds. According to the code's own logic, if the etcd/dns/network operator was Progressing, sandbox failures should be treated more leniently (as flakes), but this check is missing in the deleted pod branch.

The fix adds operator Progressing checks to the deleted pod branch

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.