Skip to content

Conversation

@openshift-bot
Copy link
Contributor

@openshift-bot openshift-bot commented Sep 19, 2025

disruptions Information

There were (310) added jobs and (180) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 591 jobs increased and 1487 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
kube-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 72.66% 8m13.01s 41.46%
cache-oauth-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 72.66% 8m18.83s 43.18%
cache-kube-api-reused-connections 4.20 s390x ovn none ha 104 3m22.55s 72.24% 8m17.86s 42.90%
cache-openshift-api-reused-connections 4.20 s390x ovn none ha 104 3m21.85s 69.62% 8m13.01s 41.56%
oauth-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 67.52% 8m17.86s 42.85%
openshift-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 66.15% 8m13.98s 41.79%
metrics-api-reused-connections 4.20 s390x ovn none ha 104 3m16.55s 66.85% 8m35.29s 49.15%
ingress-to-console-reused-connections 4.20 s390x ovn none ha 104 3m27.85s 56.51% 8m40.17s 48.03%
ingress-to-oauth-server-reused-connections 4.20 s390x ovn none ha 104 3m28.7s 53.68% 8m40.17s 48.03%
metrics-api-new-connections 4.20 s390x ovn none ha 104 3m31.5s 36.98% 9m20.88s 60.78%
ingress-to-console-new-connections 4.20 s390x ovn none ha 104 3m44s 33.81% 9m31.58s 62.55%
cache-oauth-api-new-connections 4.20 s390x ovn none ha 104 3m31.8s 36.29% 9m21.88s 60.56%
cache-kube-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.56%
oauth-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.50%
openshift-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.56%
kube-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.50%
cache-openshift-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.56%
ingress-to-oauth-server-new-connections 4.20 s390x ovn none ha 104 3m44s 33.49% 9m31.58s 62.55%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 366 27s 2600.00% 54s 625.81%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws ha 398 12.15s 1115.00% 54s 351.88%
openshift-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 7s 250.00% 18.1s 158.57%
oauth-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 6s 200.00% 18.1s 201.67%
gcp-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 994 9s 80.00% 15m14.45s 121.28%
oauth-api-reused-connections 4.20 4.20 amd64 ovn azure ha 399 5s 150.00% 12.02s 128.95%
openshift-api-reused-connections 4.20 4.20 amd64 ovn azure ha 399 5s 150.00% 11.06s 84.33%
azure-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 994 3s 200.00% 21m38.71s 135.51%
cache-openshift-api-reused-connections 4.21 4.20 amd64 ovn azure ha 1035 2s 100.00% 2.66s 33.00%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1035 1s 400.00% 2m17s 62.02%
cache-kube-api-reused-connections 4.20 4.20 amd64 ovn azure ha 399 2s 60.00% 3s 50.00%
cache-kube-api-new-connections 4.20 amd64 ovn azure external 113 10.2s 0.00% 23.64s 41.13%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn aws single 312 0s 0.00% 4.89s 63.00%
openshift-api-new-connections 4.20 4.20 amd64 ovn aws ha 368 1s 0.00% 3.99s 299.00%
ingress-to-console-reused-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 2.04s 63.20%
azure-network-liveness-new-connections 4.20 ppc64le ovn none ha 188 0s 0.00% 2s 100.00%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn aws single 312 1s 0.00% 2.89s 44.50%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 2s 100.00%
oauth-api-new-connections 4.20 4.20 arm64 ovn aws ha 113 0s 0.00% 1s 525.00%
host-to-host-reused-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 1m56.16s 42.09%
gcp-network-liveness-new-connections 4.20 4.19 amd64 ovn azure ha 902 0s 0.00% 4m17.07s 61.72%
metrics-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 15.18s 2936.00%
azure-network-liveness-reused-connections 4.20 4.19 amd64 ovn aws ha 937 0s 0.00% 1.6s 60.00%
azure-network-liveness-new-connections 4.20 4.19 amd64 ovn azure ha 902 0s 0.00% 7m9.53s 105.38%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 994 0s 0.00% 11.35s 89.17%
service-load-balancer-with-pdb-new-connections 4.20 4.19 amd64 ovn gcp ha 525 1s 0.00% 2.76s 38.00%
openshift-api-http2-service-network-reused-connections 4.20 4.19 amd64 ovn azure ha 902 1s 0.00% 15s 87.50%
kube-api-reused-connections 4.20 4.19 amd64 ovn azure ha 902 1s 0.00% 3s 50.00%
openshift-api-reused-connections 4.20 4.20 arm64 ovn aws ha 113 1s 0.00% 2s 100.00%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn aws ha 368 1.65s 0.00% 2m14.33s 125.23%
kube-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 18.1s 1348.00%
azure-network-liveness-reused-connections 4.20 amd64 ovn azure external 113 0s 0.00% 3.76s 401.33%
service-load-balancer-with-pdb-new-connections 4.20 amd64 ovn azure external 113 0s 0.00% 2.88s 64.57%
image-registry-reused-connections 4.20 4.20 arm64 ovn aws ha 113 0s 0.00% 880ms 450.00%
kube-api-http1-service-network-reused-connections 4.20 4.20 amd64 ovn azure ha 399 1s 0.00% 2s 100.00%
oauth-api-reused-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 4.33s 333.00%
kube-api-reused-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2.33s 53.29%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2s 100.00%
aws-network-liveness-new-connections 4.20 4.20 arm64 ovn aws ha 113 0s 0.00% 880ms 175.00%
oauth-api-new-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2.66s 166.00%
pod-to-host-new-connections 4.20 4.20 amd64 ovn azure ha 399 5.2s 0.00% 4m0.52s 33.07%
pod-to-host-new-connections 4.20 4.20 amd64 ovn aws ha 368 2s 0.00% 17.33s 57.55%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn azure ha 399 1s 0.00% 17.22s 474.00%
cache-oauth-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 18.1s 7140.00%
kube-api-new-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2.99s 69.89%
cache-openshift-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 18.1s 1710.00%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 0s 0.00% 32.6s 3160.00%
metrics-api-reused-connections 4.21 4.20 amd64 ovn azure ha 1035 2s 0.00% 3s 50.00%
service-load-balancer-with-pdb-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 3.66s 50.00%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3s 50.00%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 366 2s 0.00% 38.7s 1835.00%
cache-oauth-api-new-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 1.79s 59.82%
azure-network-liveness-reused-connections 4.21 s390x ovn none ha 105 0s 0.00% 52.12s 644.57%
cache-kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 2.95s 47.50%
metrics-api-new-connections 4.21 ppc64le ovn none ha 130 1s 0.00% 1.71s 71.00%
cache-openshift-api-new-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 2.58s 158.00%
oauth-api-new-connections 4.21 4.21 arm64 ovn aws ha 130 0s 0.00% 1s 69.49%
gcp-network-liveness-new-connections 4.21 s390x ovn none ha 105 0s 0.00% 20.28s 1200.00%
kube-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 2.75s 37.50%
cache-openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 3s 50.00%
image-registry-new-connections 4.21 amd64 ovn aws ha 133 1s 0.00% 23s 467.90%
oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3s 50.00%
kube-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 1s 0.00% 9s 50.00%
kube-api-http2-internal-lb-reused-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 740ms 85.00%
azure-network-liveness-reused-connections 4.21 4.21 amd64 ovn aws single 366 0s 0.00% 54s 625.81%
cache-oauth-api-new-connections 4.21 ppc64le ovn none ha 130 550ms 0.00% 1.71s 113.75%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1055 0s 0.00% 2m10s 96.08%
ingress-to-oauth-server-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 6.6s 560.00%
gcp-network-liveness-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 30.5s 937.41%
azure-network-liveness-new-connections 4.21 amd64 ovn aws external 611 0s 0.00% 53s 31076.47%
ci-cluster-network-liveness-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 15.9s 52.88%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 7.66s 53.20%
host-to-host-new-connections 4.21 4.20 amd64 ovn gcp ha 626 0s 0.00% 1s 233.33%
oauth-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 3.95s 31.67%
ci-cluster-network-liveness-new-connections 4.21 amd64 ovn aws ha 133 3.4s 0.00% 14.4s 35.72%
service-load-balancer-with-pdb-new-connections 4.21 amd64 ovn aws ha 133 0s 0.00% 1s 63.93%
openshift-api-http2-internal-lb-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 3.7s 85.00%
host-to-host-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 6m4.88s 88.08%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3.75s 87.50%
azure-network-liveness-reused-connections 4.21 4.21 arm64 ovn aws ha 130 550ms 0.00% 21.94s 120.50%
cache-kube-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 6.66s 93.60%
openshift-api-http2-internal-lb-reused-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 740ms 85.00%
kube-api-http2-internal-lb-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 5.18s 85.00%
azure-network-liveness-new-connections 4.21 multi ovn aws ha 231 1s 0.00% 51.7s 664.79%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 8s 63.93%
openshift-api-new-connections 4.21 ppc64le ovn none ha 130 1s 0.00% 1.71s 71.00%
openshift-api-http2-service-network-reused-connections 4.21 4.20 amd64 ovn azure ha 1035 2s 0.00% 15s 33.93%
image-registry-reused-connections 4.21 4.21 arm64 ovn aws ha 130 550ms 0.00% 1.71s 189.83%
kube-api-http1-internal-lb-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 5.92s 85.00%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 790ms 558.33%
ingress-to-console-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 4.5s 350.00%
ingress-to-console-reused-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 6s 41.51%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1035 300ms 0.00% 1m12.32s 44.64%
openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3s 50.00%
pod-to-host-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 6m21.64s 87.81%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws ha 398 2.15s 0.00% 40s 1233.33%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 105 4.8s 0.00% 54.96s 680.68%
image-registry-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 3.8s 280.00%
kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 3s 40.19%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1055 0s 0.00% 1m16.92s 58.27%
host-to-service-new-connections 4.21 4.21 arm64 ovn aws ha 130 550ms 0.00% 1s 69.49%
cache-kube-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 4.5s 125.00%
ci-cluster-network-liveness-new-connections 4.21 4.21 arm64 ovn aws ha 130 0s 0.00% 1s 69.49%

alerts Information

There were (4770) added jobs and (2602) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 75 jobs increased and 350 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn gcp ha 154 3h4m10.4s 271.19% 6h37m4.36s 106.60%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 105 8m38.4s 458.02% 22m4.8s 128.82%
CannotRetrieveUpdates openshift-cluster-version Warning 4.20 amd64 ovn gcp ha 135 5m37s 185.59% 38m26.8s 72.83%
CannotRetrieveUpdates openshift-cluster-version Warning 4.20 amd64 ovn vsphere ha 157 11m58s 37.76% 39m44.4s 53.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.20 arm64 ovn aws ha 108 3m25.8s 126.15% 6m47.84s 129.12%
NodeSystemSaturation openshift-monitoring Warning 4.20 4.20 amd64 ovn azure ha 151 2m12s 186.96% 6m13s 32.65%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.20 amd64 ovn azure ha 249 2m4s 113.79% 13m58.16s 173.55%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 arm64 ovn aws ha 108 1m48.8s 141.24% 24m18.68s 157.30%
NodeSystemSaturation openshift-monitoring Warning 4.20 4.20 amd64 ovn aws ha 112 1m55s 30.68% 7m11.5s 62.83%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.20 4.19 amd64 ovn azure ha 200 0s 0.00% 1m12.16s 223.59%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.20 4.20 amd64 ovn gcp ha 222 0s 0.00% 1m13.8s 442.65%
NodeSystemSaturation openshift-monitoring Warning 4.20 amd64 ovn azure ha 121 0s 0.00% 58s 107.14%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 amd64 ovn vsphere ha 157 8m16.4s 0.00% 34m22s 258.52%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 amd64 ovn aws ha 220 0s 0.00% 17m24.7s 788.96%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.20 amd64 ovn gcp ha 160 34.7s 0.00% 4m22.6s 345.84%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 114 0s 0.00% 15m37.72s 3249.00%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn vsphere ha 164 25m38.8s 0.00% 6h12m44.96s 35.43%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.20 amd64 ovn azure ha 249 0s 0.00% 35.36s 90.72%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn azure ha 170 0s 0.00% 1m15.36s 653.60%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 170 6m58s 0.00% 7m56.62s 277.37%
/cc @dgoodwin, @neisw, @DennisPeriquet

@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

In response to this:

disruptions Information

There were (162) added jobs and (12) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 410 jobs increased and 1257 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
azure-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 1072 14.45s 1345.00% 12m48.18s 39.30%
gcp-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 1072 18.45s 269.00% 9m18.42s 35.13%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 269 14s 1300.00% 52.64s 607.53%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws ha 309 11s 1000.00% 48.44s 305.36%
openshift-api-new-connections 4.20 4.20 amd64 ovn azure ha 400 5s 150.00% 10.02s 43.14%
oauth-api-reused-connections 4.20 4.20 amd64 ovn azure ha 400 5s 150.00% 10.02s 90.86%
oauth-api-new-connections 4.20 4.20 amd64 ovn azure ha 400 4.05s 102.50% 10.05s 67.50%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1058 1s 400.00% 2m17s 62.02%
cache-kube-api-reused-connections 4.20 4.20 amd64 ovn azure ha 400 2s 60.00% 3s 50.00%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn azure ha 293 1.4s 40.00% 14.08s 79.59%
openshift-api-new-connections 4.20 4.20 amd64 ovn aws ha 443 0s 0.00% 2.58s 158.00%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn aws single 364 0s 0.00% 4s 33.33%
oauth-api-reused-connections 4.20 4.20 amd64 ovn aws ha 443 0s 0.00% 2.74s 174.00%
azure-network-liveness-reused-connections 4.20 amd64 ovn azure external 117 0s 0.00% 1.84s 145.33%
image-registry-reused-connections 4.20 4.20 arm64 ovn aws ha 111 0s 0.00% 900ms 462.50%
pod-to-host-new-connections 4.20 4.20 amd64 ovn aws ha 443 0s 0.00% 16.58s 50.73%
kube-api-new-connections 4.20 4.20 amd64 ovn azure ha 400 0s 0.00% 2s 60.00%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn azure ha 400 0s 0.00% 2s 100.00%
oauth-api-new-connections 4.20 4.20 arm64 ovn aws ha 111 0s 0.00% 1s 525.00%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn azure ha 400 0s 0.00% 11s 266.67%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn aws ha 443 900ms 0.00% 1m49.22s 83.13%
kube-api-reused-connections 4.20 4.20 amd64 ovn aws ha 443 0s 0.00% 2.58s 69.74%
host-to-host-new-connections 4.20 4.20 amd64 ovn azure ha 400 50ms 0.00% 7m3.25s 52.25%
kube-api-http1-service-network-reused-connections 4.20 4.20 amd64 ovn azure ha 400 1s 0.00% 2s 100.00%
host-to-host-reused-connections 4.20 4.20 amd64 ovn azure ha 400 0s 0.00% 1m56.08s 41.99%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn aws ha 443 0s 0.00% 1.58s 58.00%
aws-network-liveness-new-connections 4.20 4.20 arm64 ovn aws ha 111 0s 0.00% 900ms 181.25%
openshift-api-reused-connections 4.20 4.20 arm64 ovn aws ha 111 1s 0.00% 1.9s 90.00%
cache-oauth-api-new-connections 4.20 4.20 amd64 ovn azure ha 400 0s 0.00% 1s 300.00%
oauth-api-new-connections 4.20 4.20 amd64 ovn aws ha 443 0s 0.00% 2s 100.00%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 269 7.2s 0.00% 40.64s 1932.00%
metrics-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 4.6s 53.33%
image-registry-new-connections 4.21 multi ovn aws ha 181 0s 0.00% 6.2s 520.00%
gcp-network-liveness-new-connections 4.21 multi ovn aws ha 181 0s 0.00% 33.6s 1042.86%
oauth-api-new-connections 4.21 4.21 amd64 ovn azure ha 293 10s 0.00% 17.24s 42.24%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1151 0s 0.00% 2m7.5s 92.31%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 5.6s 180.00%
metrics-api-reused-connections 4.21 4.20 amd64 ovn azure ha 1058 2s 0.00% 3s 50.00%
ingress-to-console-new-connections 4.21 amd64 ovn aws external 448 0s 0.00% 8.59s 39.22%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn azure ha 293 31.2s 0.00% 1m4.16s 5628.57%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws ha 309 4s 0.00% 40s 1233.33%
ingress-to-console-new-connections 4.21 multi ovn aws ha 181 0s 0.00% 2.4s 140.00%
image-registry-reused-connections 4.21 4.21 arm64 ovn aws ha 106 750ms 0.00% 1.95s 230.51%
aws-network-liveness-new-connections 4.21 4.20 amd64 ovn gcp ha 521 0s 0.00% 2s 100.00%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn azure ha 293 25s 0.00% 43.4s 4240.00%
cache-kube-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 5.8s 190.00%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1151 0s 0.00% 1m10s 44.03%
openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 5.4s 170.00%
host-to-service-new-connections 4.21 4.21 arm64 ovn aws ha 106 0s 0.00% 1s 69.49%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 741 0s 0.00% 1m38.4s 9740.00%
oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 6.2s 210.00%
kube-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 5.8s 190.00%
cache-oauth-api-reused-connections 4.21 4.21 amd64 ovn azure ha 293 2s 0.00% 3s 41.51%
ci-cluster-network-liveness-new-connections 4.21 4.21 arm64 ovn aws ha 106 0s 0.00% 1s 69.49%
service-load-balancer-with-pdb-new-connections 4.21 4.20 amd64 ovn azure ha 1058 0s 0.00% 4s 63.93%
cache-oauth-api-new-connections 4.21 4.21 amd64 ovn azure ha 293 0s 0.00% 2.16s 92.86%
ingress-to-oauth-server-new-connections 4.21 multi ovn aws ha 181 0s 0.00% 2.6s 160.00%
host-to-host-new-connections 4.21 4.21 amd64 ovn aws single 269 0s 0.00% 320ms 45.45%
metrics-api-reused-connections 4.21 4.21 amd64 ovn azure ha 293 2s 0.00% 3s 41.51%
azure-network-liveness-new-connections 4.21 multi ovn aws ha 181 3s 0.00% 54.2s 701.78%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1058 150ms 0.00% 1m11.86s 43.72%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 521 1s 0.00% 5.4s 170.00%
oauth-api-new-connections 4.21 4.21 arm64 ovn aws ha 106 0s 0.00% 1s 69.49%
ingress-to-console-reused-connections 4.21 amd64 ovn aws external 448 0s 0.00% 8.59s 39.22%
host-to-host-new-connections 4.21 4.20 amd64 ovn gcp ha 521 0s 0.00% 1s 233.33%
azure-network-liveness-new-connections 4.21 amd64 ovn aws external 448 0s 0.00% 530ms 211.76%

alerts Information

There were (2600) added jobs and (2012) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 101 jobs increased and 264 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
KubePodNotReady openshift-e2e-loki Warning 4.20 arm64 ovn aws ha 122 18h43m15s 206.14% 22h53m43.08s 165.89%
KubePodNotReady openshift-e2e-loki Warning 4.21 arm64 ovn aws ha 134 16h33m12s 31.99% 22h9m18.84s 41.62%
TargetDown openshift-e2e-loki Warning 4.20 arm64 ovn aws ha 122 3h7m12.5s 206.14% 3h48m57.18s 162.21%
KubeDaemonSetRolloutStuck openshift-e2e-loki Warning 4.20 arm64 ovn aws ha 122 2h19m15.9s 197.02% 3h2m33.58s 147.03%
TargetDown openshift-e2e-loki Warning 4.21 arm64 ovn aws ha 134 2h46m14.8s 37.10% 3h43m11.48s 43.43%
KubeDaemonSetRolloutStuck openshift-e2e-loki Warning 4.21 arm64 ovn aws ha 134 1h57m56.7s 47.52% 2h53m30.7s 91.22%
KubeContainerWaiting openshift-e2e-loki Warning 4.20 arm64 ovn aws ha 122 25m52s 26205.08% 46m11.48s 125.93%
CannotRetrieveUpdates openshift-cluster-version Warning 4.20 arm64 ovn aws ha 122 15m15.3s 3729.71% 30m17.8s 112.14%
NodeSystemSaturation openshift-monitoring Warning 4.20 amd64 ovn aws ha 185 5m56s 87.37% 17m26.32s 108.93%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.20 amd64 ovn aws ha 185 5m8s 75.00% 11m51.2s 118.32%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn aws ha 201 1m58s 321.43% 14m28s 61.79%
NodeSystemSaturation openshift-monitoring Warning 4.20 4.20 amd64 ovn aws ha 102 2m57.9s 102.16% 9m25.42s 113.37%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.20 arm64 ovn aws ha 122 2m56s 93.41% 5m49.7s 96.46%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.20 amd64 ovn aws ha 185 4m18.8s 37.08% 9m40.4s 45.74%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 amd64 ovn aws ha 185 0s 0.00% 14m37.84s 646.97%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 amd64 ovn vsphere ha 117 5m45.6s 0.00% 34m1.6s 254.97%
KubeletPlegDurationHigh kube-system Warning 4.20 amd64 ovn metal ha 191 0s 0.00% 1m59.2s 522.78%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 arm64 ovn aws ha 122 0s 0.00% 12m31.06s 32.48%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.20 4.19 amd64 ovn azure ha 216 0s 0.00% 46.4s 108.07%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.20 4.19 amd64 ovn aws ha 230 1m58s 0.00% 5m47.3s 47.00%
CannotRetrieveUpdates openshift-cluster-version Warning 4.20 amd64 ovn vsphere ha 117 11m6.8s 0.00% 35m15.76s 36.53%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.20 4.20 amd64 ovn aws ha 102 4m54.6s 0.00% 7m52.82s 31.05%
CannotRetrieveUpdates openshift-cluster-version Warning 4.21 amd64 ovn gcp ha 170 28s 0.00% 32m45.28s 45.94%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 124 0s 0.00% 2m5.76s 349.14%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn vsphere ha 177 5m36.4s 0.00% 52m39.6s 54.83%
KubeContainerWaiting openshift-e2e-loki Warning 4.21 arm64 ovn aws ha 134 26m57.5s 0.00% 56m57.38s 41.52%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn azure ha 124 0s 0.00% 44.66s 49.87%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 187 28s 0.00% 5m40.6s 169.68%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 201 3m56s 0.00% 8m54s 52.96%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.20 amd64 ovn azure ha 255 0s 0.00% 58.92s 217.80%
/cc @dgoodwin, @neisw, @DennisPeriquet

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Sep 19, 2025
@neisw
Copy link
Contributor

neisw commented Sep 22, 2025

/lgtm

@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 22, 2025
@neisw
Copy link
Contributor

neisw commented Sep 22, 2025

/lgtm

@neisw
Copy link
Contributor

neisw commented Sep 22, 2025

/verify bypass

@neisw
Copy link
Contributor

neisw commented Sep 22, 2025

/verified bypass

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Sep 22, 2025
@openshift-ci-robot
Copy link

@neisw: The verified label has been added.

In response to this:

/verified bypass

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@neisw
Copy link
Contributor

neisw commented Sep 22, 2025

/retest-required

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 92f0f58 and 2 for PR HEAD 32473ff in total

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD a08d2f3 and 1 for PR HEAD 32473ff in total

@neisw
Copy link
Contributor

neisw commented Sep 24, 2025

/retest-required

@neisw
Copy link
Contributor

neisw commented Sep 24, 2025

/override ci/prow/e2e-aws-ovn-fips

e2e-aws-ovn-fips has passed previously on this commit and is currently impacted by resource issues. no point running again

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 24, 2025

@neisw: Overrode contexts on behalf of neisw: ci/prow/e2e-aws-ovn-fips

In response to this:

/override ci/prow/e2e-aws-ovn-fips

e2e-aws-ovn-fips has passed previously on this commit and is currently impacted by resource issues. no point running again

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 6d8c2d0 and 0 for PR HEAD 32473ff in total

@openshift-ci-robot
Copy link

/hold

Revision 32473ff was retested 3 times: holding

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 24, 2025
@openshift-trt
Copy link

openshift-trt bot commented Sep 24, 2025

Job Failure Risk Analysis for sha: 32473ff

Job Name Failure Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-fips IncompleteTests
Tests for this run (105) are below the historical average (2612): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift IncompleteTests
Tests for this run (22) are below the historical average (1517): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift-serial IncompleteTests
Tests for this run (22) are below the historical average (772): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

@neisw
Copy link
Contributor

neisw commented Sep 25, 2025

/retest-required

@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from 32473ff to e44fcaa Compare September 26, 2025 01:03
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Sep 26, 2025
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

In response to this:

disruptions Information

There were (310) added jobs and (180) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 591 jobs increased and 1487 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
kube-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 72.66% 8m13.01s 41.46%
cache-oauth-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 72.66% 8m18.83s 43.18%
cache-kube-api-reused-connections 4.20 s390x ovn none ha 104 3m22.55s 72.24% 8m17.86s 42.90%
cache-openshift-api-reused-connections 4.20 s390x ovn none ha 104 3m21.85s 69.62% 8m13.01s 41.56%
oauth-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 67.52% 8m17.86s 42.85%
openshift-api-reused-connections 4.20 s390x ovn none ha 104 3m22.7s 66.15% 8m13.98s 41.79%
metrics-api-reused-connections 4.20 s390x ovn none ha 104 3m16.55s 66.85% 8m35.29s 49.15%
ingress-to-console-reused-connections 4.20 s390x ovn none ha 104 3m27.85s 56.51% 8m40.17s 48.03%
ingress-to-oauth-server-reused-connections 4.20 s390x ovn none ha 104 3m28.7s 53.68% 8m40.17s 48.03%
metrics-api-new-connections 4.20 s390x ovn none ha 104 3m31.5s 36.98% 9m20.88s 60.78%
ingress-to-console-new-connections 4.20 s390x ovn none ha 104 3m44s 33.81% 9m31.58s 62.55%
cache-oauth-api-new-connections 4.20 s390x ovn none ha 104 3m31.8s 36.29% 9m21.88s 60.56%
cache-kube-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.56%
oauth-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.50%
openshift-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.56%
kube-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.50%
cache-openshift-api-new-connections 4.20 s390x ovn none ha 104 3m31.65s 36.20% 9m21.88s 60.56%
ingress-to-oauth-server-new-connections 4.20 s390x ovn none ha 104 3m44s 33.49% 9m31.58s 62.55%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 366 27s 2600.00% 54s 625.81%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws ha 398 12.15s 1115.00% 54s 351.88%
openshift-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 7s 250.00% 18.1s 158.57%
oauth-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 6s 200.00% 18.1s 201.67%
gcp-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 994 9s 80.00% 15m14.45s 121.28%
oauth-api-reused-connections 4.20 4.20 amd64 ovn azure ha 399 5s 150.00% 12.02s 128.95%
openshift-api-reused-connections 4.20 4.20 amd64 ovn azure ha 399 5s 150.00% 11.06s 84.33%
azure-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 994 3s 200.00% 21m38.71s 135.51%
cache-openshift-api-reused-connections 4.21 4.20 amd64 ovn azure ha 1035 2s 100.00% 2.66s 33.00%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1035 1s 400.00% 2m17s 62.02%
cache-kube-api-reused-connections 4.20 4.20 amd64 ovn azure ha 399 2s 60.00% 3s 50.00%
cache-kube-api-new-connections 4.20 amd64 ovn azure external 113 10.2s 0.00% 23.64s 41.13%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn aws single 312 0s 0.00% 4.89s 63.00%
openshift-api-new-connections 4.20 4.20 amd64 ovn aws ha 368 1s 0.00% 3.99s 299.00%
ingress-to-console-reused-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 2.04s 63.20%
azure-network-liveness-new-connections 4.20 ppc64le ovn none ha 188 0s 0.00% 2s 100.00%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn aws single 312 1s 0.00% 2.89s 44.50%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 2s 100.00%
oauth-api-new-connections 4.20 4.20 arm64 ovn aws ha 113 0s 0.00% 1s 525.00%
host-to-host-reused-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 1m56.16s 42.09%
gcp-network-liveness-new-connections 4.20 4.19 amd64 ovn azure ha 902 0s 0.00% 4m17.07s 61.72%
metrics-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 15.18s 2936.00%
azure-network-liveness-reused-connections 4.20 4.19 amd64 ovn aws ha 937 0s 0.00% 1.6s 60.00%
azure-network-liveness-new-connections 4.20 4.19 amd64 ovn azure ha 902 0s 0.00% 7m9.53s 105.38%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn gcp ha 994 0s 0.00% 11.35s 89.17%
service-load-balancer-with-pdb-new-connections 4.20 4.19 amd64 ovn gcp ha 525 1s 0.00% 2.76s 38.00%
openshift-api-http2-service-network-reused-connections 4.20 4.19 amd64 ovn azure ha 902 1s 0.00% 15s 87.50%
kube-api-reused-connections 4.20 4.19 amd64 ovn azure ha 902 1s 0.00% 3s 50.00%
openshift-api-reused-connections 4.20 4.20 arm64 ovn aws ha 113 1s 0.00% 2s 100.00%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn aws ha 368 1.65s 0.00% 2m14.33s 125.23%
kube-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 18.1s 1348.00%
azure-network-liveness-reused-connections 4.20 amd64 ovn azure external 113 0s 0.00% 3.76s 401.33%
service-load-balancer-with-pdb-new-connections 4.20 amd64 ovn azure external 113 0s 0.00% 2.88s 64.57%
image-registry-reused-connections 4.20 4.20 arm64 ovn aws ha 113 0s 0.00% 880ms 450.00%
kube-api-http1-service-network-reused-connections 4.20 4.20 amd64 ovn azure ha 399 1s 0.00% 2s 100.00%
oauth-api-reused-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 4.33s 333.00%
kube-api-reused-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2.33s 53.29%
azure-network-liveness-reused-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2s 100.00%
aws-network-liveness-new-connections 4.20 4.20 arm64 ovn aws ha 113 0s 0.00% 880ms 175.00%
oauth-api-new-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2.66s 166.00%
pod-to-host-new-connections 4.20 4.20 amd64 ovn azure ha 399 5.2s 0.00% 4m0.52s 33.07%
pod-to-host-new-connections 4.20 4.20 amd64 ovn aws ha 368 2s 0.00% 17.33s 57.55%
ci-cluster-network-liveness-new-connections 4.20 4.20 amd64 ovn azure ha 399 1s 0.00% 17.22s 474.00%
cache-oauth-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 18.1s 7140.00%
kube-api-new-connections 4.20 4.20 amd64 ovn aws ha 368 0s 0.00% 2.99s 69.89%
cache-openshift-api-new-connections 4.20 4.20 amd64 ovn azure ha 399 0s 0.00% 18.1s 1710.00%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 0s 0.00% 32.6s 3160.00%
metrics-api-reused-connections 4.21 4.20 amd64 ovn azure ha 1035 2s 0.00% 3s 50.00%
service-load-balancer-with-pdb-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 3.66s 50.00%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3s 50.00%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 366 2s 0.00% 38.7s 1835.00%
cache-oauth-api-new-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 1.79s 59.82%
azure-network-liveness-reused-connections 4.21 s390x ovn none ha 105 0s 0.00% 52.12s 644.57%
cache-kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 2.95s 47.50%
metrics-api-new-connections 4.21 ppc64le ovn none ha 130 1s 0.00% 1.71s 71.00%
cache-openshift-api-new-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 2.58s 158.00%
oauth-api-new-connections 4.21 4.21 arm64 ovn aws ha 130 0s 0.00% 1s 69.49%
gcp-network-liveness-new-connections 4.21 s390x ovn none ha 105 0s 0.00% 20.28s 1200.00%
kube-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 2.75s 37.50%
cache-openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 3s 50.00%
image-registry-new-connections 4.21 amd64 ovn aws ha 133 1s 0.00% 23s 467.90%
oauth-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3s 50.00%
kube-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 1s 0.00% 9s 50.00%
kube-api-http2-internal-lb-reused-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 740ms 85.00%
azure-network-liveness-reused-connections 4.21 4.21 amd64 ovn aws single 366 0s 0.00% 54s 625.81%
cache-oauth-api-new-connections 4.21 ppc64le ovn none ha 130 550ms 0.00% 1.71s 113.75%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1055 0s 0.00% 2m10s 96.08%
ingress-to-oauth-server-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 6.6s 560.00%
gcp-network-liveness-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 30.5s 937.41%
azure-network-liveness-new-connections 4.21 amd64 ovn aws external 611 0s 0.00% 53s 31076.47%
ci-cluster-network-liveness-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 15.9s 52.88%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 7.66s 53.20%
host-to-host-new-connections 4.21 4.20 amd64 ovn gcp ha 626 0s 0.00% 1s 233.33%
oauth-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 3.95s 31.67%
ci-cluster-network-liveness-new-connections 4.21 amd64 ovn aws ha 133 3.4s 0.00% 14.4s 35.72%
service-load-balancer-with-pdb-new-connections 4.21 amd64 ovn aws ha 133 0s 0.00% 1s 63.93%
openshift-api-http2-internal-lb-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 3.7s 85.00%
host-to-host-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 6m4.88s 88.08%
cache-openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3.75s 87.50%
azure-network-liveness-reused-connections 4.21 4.21 arm64 ovn aws ha 130 550ms 0.00% 21.94s 120.50%
cache-kube-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 6.66s 93.60%
openshift-api-http2-internal-lb-reused-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 740ms 85.00%
kube-api-http2-internal-lb-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 5.18s 85.00%
azure-network-liveness-new-connections 4.21 multi ovn aws ha 231 1s 0.00% 51.7s 664.79%
cache-oauth-api-new-connections 4.21 4.20 amd64 ovn azure ha 1035 0s 0.00% 8s 63.93%
openshift-api-new-connections 4.21 ppc64le ovn none ha 130 1s 0.00% 1.71s 71.00%
openshift-api-http2-service-network-reused-connections 4.21 4.20 amd64 ovn azure ha 1035 2s 0.00% 15s 33.93%
image-registry-reused-connections 4.21 4.21 arm64 ovn aws ha 130 550ms 0.00% 1.71s 189.83%
kube-api-http1-internal-lb-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 5.92s 85.00%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 790ms 558.33%
ingress-to-console-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 4.5s 350.00%
ingress-to-console-reused-connections 4.21 4.21 amd64 ovn azure ha 422 0s 0.00% 6s 41.51%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1035 300ms 0.00% 1m12.32s 44.64%
openshift-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 3s 50.00%
pod-to-host-new-connections 4.21 ppc64le ovn libvirt ha 127 0s 0.00% 6m21.64s 87.81%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws ha 398 2.15s 0.00% 40s 1233.33%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 105 4.8s 0.00% 54.96s 680.68%
image-registry-new-connections 4.21 multi ovn aws ha 231 0s 0.00% 3.8s 280.00%
kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1006 1s 0.00% 3s 40.19%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1055 0s 0.00% 1m16.92s 58.27%
host-to-service-new-connections 4.21 4.21 arm64 ovn aws ha 130 550ms 0.00% 1s 69.49%
cache-kube-api-new-connections 4.21 4.20 amd64 ovn gcp ha 626 1s 0.00% 4.5s 125.00%
ci-cluster-network-liveness-new-connections 4.21 4.21 arm64 ovn aws ha 130 0s 0.00% 1s 69.49%

alerts Information

There were (4770) added jobs and (2602) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 75 jobs increased and 350 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn gcp ha 154 3h4m10.4s 271.19% 6h37m4.36s 106.60%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 105 8m38.4s 458.02% 22m4.8s 128.82%
CannotRetrieveUpdates openshift-cluster-version Warning 4.20 amd64 ovn gcp ha 135 5m37s 185.59% 38m26.8s 72.83%
CannotRetrieveUpdates openshift-cluster-version Warning 4.20 amd64 ovn vsphere ha 157 11m58s 37.76% 39m44.4s 53.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.20 arm64 ovn aws ha 108 3m25.8s 126.15% 6m47.84s 129.12%
NodeSystemSaturation openshift-monitoring Warning 4.20 4.20 amd64 ovn azure ha 151 2m12s 186.96% 6m13s 32.65%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.20 amd64 ovn azure ha 249 2m4s 113.79% 13m58.16s 173.55%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 arm64 ovn aws ha 108 1m48.8s 141.24% 24m18.68s 157.30%
NodeSystemSaturation openshift-monitoring Warning 4.20 4.20 amd64 ovn aws ha 112 1m55s 30.68% 7m11.5s 62.83%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.20 4.19 amd64 ovn azure ha 200 0s 0.00% 1m12.16s 223.59%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.20 4.20 amd64 ovn gcp ha 222 0s 0.00% 1m13.8s 442.65%
NodeSystemSaturation openshift-monitoring Warning 4.20 amd64 ovn azure ha 121 0s 0.00% 58s 107.14%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 amd64 ovn vsphere ha 157 8m16.4s 0.00% 34m22s 258.52%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.20 amd64 ovn aws ha 220 0s 0.00% 17m24.7s 788.96%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.20 amd64 ovn gcp ha 160 34.7s 0.00% 4m22.6s 345.84%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 114 0s 0.00% 15m37.72s 3249.00%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn vsphere ha 164 25m38.8s 0.00% 6h12m44.96s 35.43%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.20 amd64 ovn azure ha 249 0s 0.00% 35.36s 90.72%
KubeDaemonSetMisScheduled openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn azure ha 170 0s 0.00% 1m15.36s 653.60%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 170 6m58s 0.00% 7m56.62s 277.37%
/cc @dgoodwin, @neisw, @DennisPeriquet

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 26, 2025
@neisw
Copy link
Contributor

neisw commented Sep 26, 2025

/lgtm
/hold cancel
/retest-required

@openshift-ci openshift-ci bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Sep 26, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 26, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw, openshift-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@neisw
Copy link
Contributor

neisw commented Sep 26, 2025

/verified bypass

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Sep 26, 2025
@openshift-ci-robot
Copy link

@neisw: The verified label has been added.

In response to this:

/verified bypass

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD da8a7f3 and 2 for PR HEAD e44fcaa in total

@openshift-trt
Copy link

openshift-trt bot commented Sep 26, 2025

Job Failure Risk Analysis for sha: e44fcaa

Job Name Failure Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-microshift-serial IncompleteTests
Tests for this run (22) are below the historical average (547): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 506d6c4 and 1 for PR HEAD e44fcaa in total

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 27, 2025

@openshift-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-metal-ipi-ovn-dualstack 32473ff link false /test e2e-metal-ipi-ovn-dualstack
ci/prow/e2e-metal-ipi-virtualmedia 32473ff link false /test e2e-metal-ipi-virtualmedia
ci/prow/e2e-aws-ovn-kube-apiserver-rollout 32473ff link false /test e2e-aws-ovn-kube-apiserver-rollout
ci/prow/e2e-gcp-ovn-techpreview 32473ff link false /test e2e-gcp-ovn-techpreview
ci/prow/e2e-aws-proxy 32473ff link false /test e2e-aws-proxy
ci/prow/e2e-metal-ipi-ovn-dualstack-local-gateway 32473ff link false /test e2e-metal-ipi-ovn-dualstack-local-gateway
ci/prow/e2e-aws-disruptive 32473ff link false /test e2e-aws-disruptive
ci/prow/e2e-hypershift-conformance 32473ff link false /test e2e-hypershift-conformance
ci/prow/e2e-aws-ovn-single-node e44fcaa link false /test e2e-aws-ovn-single-node
ci/prow/e2e-aws-ovn-single-node-serial e44fcaa link false /test e2e-aws-ovn-single-node-serial
ci/prow/e2e-openstack-ovn e44fcaa link false /test e2e-openstack-ovn
ci/prow/e2e-aws-ovn-single-node-upgrade e44fcaa link false /test e2e-aws-ovn-single-node-upgrade
ci/prow/okd-scos-e2e-aws-ovn e44fcaa link false /test okd-scos-e2e-aws-ovn
ci/prow/e2e-aws-ovn-edge-zones e44fcaa link false /test e2e-aws-ovn-edge-zones

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci-robot
Copy link

/retest-required

Remaining retests: 0 against base HEAD 44be851 and 0 for PR HEAD e44fcaa in total

@openshift-merge-bot openshift-merge-bot bot merged commit 241d056 into openshift:main Sep 27, 2025
24 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants