Skip to content

MCO-1984: Add Component Readiness tests for boot image skew enforcement #5547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
djoshy wants to merge 7 commits into
base: main
Choose a base branch
from
Open

MCO-1984: Add Component Readiness tests for boot image skew enforcement #5547

djoshy wants to merge 7 commits into from

Conversation

@djoshy
Copy link
Contributor

@djoshy djoshy commented Jan 12, 2026 •
edited
Loading

This PR adds 5 e2e tests for the boot image skew enforcement feature. It's based on #5428 and should only merge after that PR has landed. Only the last commit is relevant to the e2e. Tests added:

  1. Manual mode within skew limits — Sets manual mode with a compliant boot image and verifies the CO's Upgradeable condition is True.
  2. Manual mode outside skew limits — Sets manual mode with a non-compliant boot image and verifies the CO's Upgradeable condition is False.
  3. Automatic mode — Uses the default Automatic mode and verifies the CO's Upgradeable condition is True.
  4. Automatic mode with degraded controller — Uses Automatic mode but injects an older boot image with a non-existent user-data secret, causing the boot image controller to degrade. Verifies the CO's Upgradeable condition is False.
  5. None mode (opted out) — Sets None mode to opt out of skew enforcement and verifies the CO's Upgradeable condition is True.

Additional note: It also has some minor verify fixes that I was encountering locally. I kept the fixes in the same commit(they were just lack of "%s" specifiers), but I'm happy to break it into a separate commit if needed.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 12, 2026
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jan 12, 2026 •
edited by openshift-ci bot
Loading

@djoshy: This pull request references MCO-1984 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR adds 5 e2e tests for the boot image skew enforcement feature. It's based on #5428 and should only merge after that PR lands. Only the last commit is relevant to this PR; the remaining are from #5428. Tests added:

  1. Manual mode within skew limits — Sets manual mode with a compliant boot image and verifies the CO's Upgradeable condition is True.
  2. Manual mode outside skew limits — Sets manual mode with a non-compliant boot image and verifies the CO's Upgradeable condition is False.
  3. Automatic mode — Uses the default Automatic mode and verifies the CO's Upgradeable condition is True.
  4. Automatic mode with degraded controller — Uses Automatic mode but injects an older boot image with a non-existent user-data secret, causing the boot image controller to degrade. Verifies the CO's Upgradeable condition is False.
  5. None mode (opted out) — Sets None mode to opt out of skew enforcement and verifies the CO's Upgradeable condition is True.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 12, 2026
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 12, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 12, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 12, 2026
@djoshy djoshy force-pushed the implement-skew-enforcement-e2e branch from 0415a0f to 12f0f71 Compare January 12, 2026 20:34
@djoshy
Copy link
Contributor Author

djoshy commented Jan 12, 2026

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-2of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 12, 2026

@djoshy: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-1of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-2of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0d85c830-eff6-11f0-8cec-7bb3f17fd777-0

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jan 12, 2026 •
edited by openshift-ci bot
Loading

@djoshy: This pull request references MCO-1984 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR adds 5 e2e tests for the boot image skew enforcement feature. It's based on #5428 and should only merge after that PR lands. Only the last commit is relevant to this PR; the remaining are from #5428. Tests added:

  1. Manual mode within skew limits — Sets manual mode with a compliant boot image and verifies the CO's Upgradeable condition is True.
  2. Manual mode outside skew limits — Sets manual mode with a non-compliant boot image and verifies the CO's Upgradeable condition is False.
  3. Automatic mode — Uses the default Automatic mode and verifies the CO's Upgradeable condition is True.
  4. Automatic mode with degraded controller — Uses Automatic mode but injects an older boot image with a non-existent user-data secret, causing the boot image controller to degrade. Verifies the CO's Upgradeable condition is False.
  5. None mode (opted out) — Sets None mode to opt out of skew enforcement and verifies the CO's Upgradeable condition is True.

Additional note: It also has some minor verify fixes that was encountering locally that I kept in the same commit(mostly lack of "%s"), I'm happy to break it into a separate commit if needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jan 12, 2026 •
edited by openshift-ci bot
Loading

@djoshy: This pull request references MCO-1984 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR adds 5 e2e tests for the boot image skew enforcement feature. It's based on #5428 and should only merge after that PR lands. Only the last commit is relevant to this PR; the remaining are from #5428. Tests added:

  1. Manual mode within skew limits — Sets manual mode with a compliant boot image and verifies the CO's Upgradeable condition is True.
  2. Manual mode outside skew limits — Sets manual mode with a non-compliant boot image and verifies the CO's Upgradeable condition is False.
  3. Automatic mode — Uses the default Automatic mode and verifies the CO's Upgradeable condition is True.
  4. Automatic mode with degraded controller — Uses Automatic mode but injects an older boot image with a non-existent user-data secret, causing the boot image controller to degrade. Verifies the CO's Upgradeable condition is False.
  5. None mode (opted out) — Sets None mode to opt out of skew enforcement and verifies the CO's Upgradeable condition is True.

Additional note: It also has some minor verify fixes that I was encountering locally. I kept the fixes in the same commit(they were just lack of "%s" specifiers), but I'm happy to break it into a separate commit if needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@djoshy djoshy force-pushed the implement-skew-enforcement-e2e branch from 12f0f71 to 80fa5d5 Compare January 13, 2026 15:11
@djoshy
Copy link
Contributor Author

djoshy commented Jan 13, 2026

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 13, 2026

@djoshy: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1ca51e00-f092-11f0-88fd-2976531df82a-0

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jan 13, 2026 •
edited by openshift-ci bot
Loading

@djoshy: This pull request references MCO-1984 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR adds 5 e2e tests for the boot image skew enforcement feature. It's based on #5428 and #5539 and should only merge after those PRs have landed. Only the last commit is relevant to the e2es; the remaining are from #5428 and #5539. Tests added:

  1. Manual mode within skew limits — Sets manual mode with a compliant boot image and verifies the CO's Upgradeable condition is True.
  2. Manual mode outside skew limits — Sets manual mode with a non-compliant boot image and verifies the CO's Upgradeable condition is False.
  3. Automatic mode — Uses the default Automatic mode and verifies the CO's Upgradeable condition is True.
  4. Automatic mode with degraded controller — Uses Automatic mode but injects an older boot image with a non-existent user-data secret, causing the boot image controller to degrade. Verifies the CO's Upgradeable condition is False.
  5. None mode (opted out) — Sets None mode to opt out of skew enforcement and verifies the CO's Upgradeable condition is True.

Additional note: It also has some minor verify fixes that I was encountering locally. I kept the fixes in the same commit(they were just lack of "%s" specifiers), but I'm happy to break it into a separate commit if needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@djoshy djoshy force-pushed the implement-skew-enforcement-e2e branch 3 times, most recently from e62ac96 to b38b41c Compare January 13, 2026 16:55
@djoshy djoshy mentioned this pull request Jan 13, 2026
Open
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Jan 13, 2026 •
edited by openshift-ci bot
Loading

@djoshy: This pull request references MCO-1984 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

This PR adds 5 e2e tests for the boot image skew enforcement feature. It's based on #5428 and should only merge after that PR has landed. Only the last commit is relevant to the e2e. Tests added:

  1. Manual mode within skew limits — Sets manual mode with a compliant boot image and verifies the CO's Upgradeable condition is True.
  2. Manual mode outside skew limits — Sets manual mode with a non-compliant boot image and verifies the CO's Upgradeable condition is False.
  3. Automatic mode — Uses the default Automatic mode and verifies the CO's Upgradeable condition is True.
  4. Automatic mode with degraded controller — Uses Automatic mode but injects an older boot image with a non-existent user-data secret, causing the boot image controller to degrade. Verifies the CO's Upgradeable condition is False.
  5. None mode (opted out) — Sets None mode to opt out of skew enforcement and verifies the CO's Upgradeable condition is True.

Additional note: It also has some minor verify fixes that I was encountering locally. I kept the fixes in the same commit(they were just lack of "%s" specifiers), but I'm happy to break it into a separate commit if needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@djoshy
Copy link
Contributor Author

djoshy commented Jan 13, 2026

/payload-abort

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 13, 2026

@djoshy: aborted 2 active payload job(s) for pull request #5547

@djoshy
Copy link
Contributor Author

djoshy commented Jan 13, 2026

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-2of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 13, 2026

@djoshy: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-1of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-2of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9b814080-f0a3-11f0-8afd-59d511167709-0

djoshy added 6 commits January 16, 2026 08:51
@djoshy
operator: populate skew enforcement status
87789e4 
This change adds logic to populate the BootImageSkewEnforcementStatus
field in the MachineConfiguration status based on spec configuration,
platform support, and cluster version information.
@djoshy
operator: add skew enforcement status unit tests
b62a2c6 
This adds new unit tests for TestSyncMachineConfiguration to test the
BootImageSkewEnforcementStatus sync logic added in the previous commit.
@djoshy
bootimage: update skew enforcement status
77dc57f 
This commit updates the machine-set-boot-image controller to track and
update the BootImageSkewEnforcementStatus when in Automatic mode.
@djoshy
operator: block upgrades on skew violation
9d05958 
This commit implements upgrade blocking when boot image version skew
exceeds acceptable limits, via the ClusterOperator Upgradeable
condition.
@djoshy
operator: add upgrade block unit tests
7f1bb46 
This commit adds unit tests for the new Upgradeable guards added in the
previous commit.
@djoshy
operator: verify boot image controller state
e9d08c6 
This commit ensures that the boot image controller state is acceptable before checking the skew. This check is only done in Automatic mode.
@djoshy djoshy force-pushed the implement-skew-enforcement-e2e branch from b38b41c to 31cba48 Compare January 16, 2026 13:54
@djoshy
Copy link
Contributor Author

djoshy commented Jan 16, 2026

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

rebased to capture changes from #5539, and now that openshift/api#2648 has landed, these tests should begin to pass. Unsure if the CRD will show up in the test suites yet, but doing a single run to see

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 16, 2026

@djoshy: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-2of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7ac7ae50-f2e3-11f0-827a-c71b5cd03599-0

@djoshy djoshy marked this pull request as ready for review January 16, 2026 13:58
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 16, 2026
@djoshy
Copy link
Contributor Author

djoshy commented Jan 17, 2026

Looks good, running the other gcp shard and the azure variant

/payload-job periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-1of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-2of2 periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Jan 17, 2026

@djoshy: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-1of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-azure-mco-disruptive-techpreview-2of2
  • periodic-ci-openshift-machine-config-operator-release-4.22-periodics-e2e-gcp-mco-disruptive-techpreview-1of2

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/0a3bea50-f399-11f0-8cab-53afd62452e4-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cheesesashimi cheesesashimi Awaiting requested review from cheesesashimi

@pablintino pablintino Awaiting requested review from pablintino

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@djoshy @openshift-ci-robot