Skip to content

Conversation

@jubittajohn
Copy link
Contributor

@jubittajohn jubittajohn commented Oct 17, 2024

The following test covers a vertical scaling scenario when a member is unhealthy.This test validates that scale down happens before scale up if the deleted member is unhealthy.CPMS is disabled to observe that scale-down happens first in this case.

  1. If the CPMS is active, first disable it by deleting the CPMS custom resource.
  2. Remove the static pod manifest from a node and stop the kubelet on the node. This makes the member unhealthy.
  3. Delete the machine hosting the node in step 2.
  4. Verify the member removal and the total voting member count of 2 to ensure scale-down happens first when a member is unhealthy.
  5. Restore the initial cluster state by creating a new machine(scale-up) and re-enabling CPMS

@jubittajohn jubittajohn changed the title Add E2E test for scaling when an unhealthy member is present ETCD-674: Add E2E test for scaling when an unhealthy member is present Oct 17, 2024
@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 17, 2024

@jubittajohn: This pull request references ETCD-674 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 17, 2024
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 17, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jubittajohn
Once this PR has been reviewed and has the lgtm label, please assign hasbro17 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jubittajohn jubittajohn changed the title ETCD-674: Add E2E test for scaling when an unhealthy member is present ETCD-674: WIP: Add E2E test for scaling when an unhealthy member is present Oct 17, 2024
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from 97d7498 to fdbfada Compare October 21, 2024 16:44
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-ci-robot
Copy link

openshift-ci-robot commented Oct 21, 2024

@jubittajohn: This pull request references ETCD-674 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.18.0" version, but no target version was set.

In response to this:

The following test covers a vertical scaling scenario when a member is unhealthy.This test validates that scale down happens before scale up if the deleted member is unhealthy.CPMS is disabled to observe that scale-down happens first in this case.

  1. If the CPMS is active, first disable it by deleting the CPMS custom resource.
  2. Remove the static pod manifest from a node and stop the kubelet on the node. This makes the member unhealthy.
  3. Delete the machine hosting the node in step 2.
  4. Verify the member removal and the total voting member count of 2 to ensure scale-down happens first when a member is unhealthy.
  5. Restore the initial cluster state by creating a new machine(scale-up) and re-enabling CPMS

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: 34733b4

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout IncompleteTests
Tests for this run (101) are below the historical average (1155): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6 IncompleteTests
Tests for this run (101) are below the historical average (2074): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn IncompleteTests
Tests for this run (101) are below the historical average (2242): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from 0cd8930 to 5c2b0d3 Compare October 22, 2024 14:57
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: 5c2b0d3

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout IncompleteTests
Tests for this run (101) are below the historical average (1064): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6 IncompleteTests
Tests for this run (101) are below the historical average (1813): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn IncompleteTests
Tests for this run (101) are below the historical average (2078): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-kube-apiserver-rollout Low
[Conformance][Suite:openshift/kube-apiserver/rollout][Jira:"kube-apiserver"][sig-kube-apiserver] kube-apiserver should roll out new revisions without disruption [apigroup:config.openshift.io][apigroup:operator.openshift.io]
This test has passed 38.46% of 13 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-kube-apiserver-rollout'] in the last 14 days.

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from 7028ca9 to b5520ee Compare October 22, 2024 20:20
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: b5520ee

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-etcd-scaling High
[sig-etcd] etcd leader changes are not excessive [Late] [Suite:openshift/conformance/parallel]
This test has passed 100.00% of 7 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-gcp-ovn-etcd-scaling'] in the last 14 days.
pull-ci-openshift-origin-master-e2e-aws-ovn-etcd-scaling High
[sig-node] Managed cluster should verify that nodes have no unexpected reboots [Late] [Suite:openshift/conformance/parallel]
This test has passed 100.00% of 4 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-etcd-scaling'] in the last 14 days.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 23, 2024
@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from 28a3d90 to eee774f Compare October 23, 2024 16:24
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 23, 2024
@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from eee774f to 5d2c0c7 Compare October 23, 2024 16:26
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling


import (
"fmt"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you run make verify? Not sure what's up with this.

https://storage.googleapis.com/test-platform-results/pr-logs/pull/29203/pull-ci-openshift-origin-master-verify/1849125549352423424/build-log.txt

FAILURE after 1.054s: hack/verify-generated.sh:14: executing 'git diff --exit-code /go/src/github.com/openshift/origin/test/extended/util/annotate/generated/' expecting success: the command returned the wrong error code
Standard output from the command:
diff --git a/test/extended/util/annotate/generated/zz_generated.annotations.go b/test/extended/util/annotate/generated/zz_generated.annotations.go
index 3f99baec7a..dbd41569a8 100644
--- a/test/extended/util/annotate/generated/zz_generated.annotations.go
+++ b/test/extended/util/annotate/generated/zz_generated.annotations.go
@@ -2,7 +2,6 @@ package generated
 
 import (
 	"fmt"
-
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/ginkgo/v2/types"
 )

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: 5d2c0c7

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-vsphere-ovn-etcd-scaling High
[sig-api-machinery] disruption/cache-openshift-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
---
[sig-api-machinery] disruption/cache-oauth-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
---
[sig-api-machinery] disruption/kube-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
pull-ci-openshift-origin-master-e2e-aws-ovn-etcd-scaling High
[sig-architecture] platform pods in ns/openshift-etcd should not exit an excessive amount of times
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-etcd-scaling'] in the last 14 days.

Open Bugs
etcd platform pod exist test failing on etcd-scaling jobs
---
[bz-etcd][invariant] alert/etcdMembersDown should not be at or above info
This test has passed 100.00% of 4 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-etcd-scaling'] in the last 14 days.

Open Bugs
etcd-scaling jobs failing ~60% of the time

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from f4dae15 to da504c1 Compare October 24, 2024 19:07
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: da504c1

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-vsphere-ovn-etcd-scaling High
[sig-api-machinery] disruption/cache-kube-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
---
[sig-api-machinery] disruption/cache-oauth-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
---
[sig-api-machinery] disruption/kube-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
---
[sig-api-machinery] disruption/oauth-api connection/new should be available throughout the test
This test has passed 100.00% of 2 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-vsphere-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-vsphere-ovn-etcd-scaling'] in the last 14 days.
---
Showing 4 of 7 test results

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from 9ef1128 to a9cb8f4 Compare October 31, 2024 15:36
@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from a9cb8f4 to e240756 Compare October 31, 2024 15:39
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@jubittajohn
Copy link
Contributor Author

/test e2e-gcp-ovn-etcd-scaling

@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: e240756

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-etcd-scaling High
[bz-etcd][invariant] alert/etcdMembersDown should not be at or above info
This test has passed 100.00% of 4 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-etcd-scaling' 'periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-etcd-scaling'] in the last 14 days.

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch 2 times, most recently from 6462803 to c85351e Compare November 12, 2024 05:57
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 12, 2024
@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: c85351e

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-kube-apiserver-rollout Low
[Conformance][Suite:openshift/kube-apiserver/rollout][Jira:"kube-apiserver"][sig-kube-apiserver] kube-apiserver should roll out new revisions without disruption [apigroup:config.openshift.io][apigroup:operator.openshift.io]
This test has passed 69.23% of 13 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-kube-apiserver-rollout'] in the last 14 days.

@jubittajohn jubittajohn force-pushed the vertical-scaling-unhealthy-member branch from c85351e to 723212b Compare November 12, 2024 15:26
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 12, 2024
@jubittajohn
Copy link
Contributor Author

/test e2e-aws-ovn-etcd-scaling
/test e2e-gcp-ovn-etcd-scaling
/test e2e-azure-ovn-etcd-scaling
/test e2e-vsphere-ovn-etcd-scaling

@jubittajohn jubittajohn changed the title ETCD-674: WIP: Add E2E test for scaling when an unhealthy member is present ETCD-674: Add E2E test for scaling when an unhealthy member is present Nov 12, 2024
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Nov 12, 2024

@jubittajohn: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn 723212b link true /test e2e-gcp-ovn
ci/prow/e2e-openstack-ovn 723212b link false /test e2e-openstack-ovn
ci/prow/e2e-gcp-ovn-etcd-scaling 723212b link false /test e2e-gcp-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-etcd-scaling 723212b link false /test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-single-node-serial 723212b link false /test e2e-aws-ovn-single-node-serial
ci/prow/e2e-azure-ovn-etcd-scaling 723212b link false /test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-vsphere-ovn-etcd-scaling 723212b link false /test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-agnostic-ovn-cmd 723212b link false /test e2e-agnostic-ovn-cmd

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt-bot
Copy link

Job Failure Risk Analysis for sha: 723212b

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-etcd-scaling High

@jubittajohn
Copy link
Contributor Author

Added this test to the PR: #29236

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants