[release-4.21] MCO-2017: Backport OSImageStreams to 4.21

[pablintino]

- What I did
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

- How to verify it

TBD

- Description for the changelog
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

[openshift-ci-robot]

@pablintino: This pull request references MCO-2017 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

Details

In response to this:

- What I did
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

- How to verify it

TBD

- Description for the changelog
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pablintino

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [pablintino]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ptalgulk01]

Pre-merge verified:

Environment Setup
OpenShift Version: 4.21.0-0-2026-01-08-104425-test-ci-ln-zjmf0hk-latest
Platform: AWS, GCP, Azure, Vsphere, FIPS on TechPreview Cluster

Verification Steps

Checking Switch

• Enable the TechPreview and check OSStream enabled

status:
  featureGates:
           - name: OSStreams

• Default is rhel-9

$  oc get mcp worker -ojsonpath='{.status.osImageStream.name}'
 empty
$ oc debug node/ip-10-0-71-60.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-71-60us-east-2computeinternal-debug-qqxzm ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build05.ci.openshift.org/ci-ln-qz3x1kb/stable@sha256:707f4ec6193865bb7eda34e8e9bb5f1310b00eb4342fd666ae72a55c218811aa
                   Digest: sha256:707f4ec6193865bb7eda34e8e9bb5f1310b00eb4342fd666ae72a55c218811aa
                  Version: 9.6.20251219-0 (2025-12-25T12:49:21Z)

• Switch to rhel-10 (Tested on both Master and Worker pool)

 $ oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

$ oc debug node/ip-10-0-8-235.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-8-235us-east-2computeinternal-debug-tglv9 ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build05.ci.openshift.org/ci-ln-qz3x1kb/stable@sha256:8eab89184784583b739b785772d1697a73b8bd09f8dfb7b1f61dbb34d3a56bd0
                   Digest: sha256:8eab89184784583b739b785772d1697a73b8bd09f8dfb7b1f61dbb34d3a56bd0
                  Version: 10.1.20251222-0 (2025-12-26T23:56:41Z)

PIS Verification (No-issue)

Applied PIS on rhel-9 updated to rhel-10
Applied on rhel10 updated to rhel-9

Kernel Argument Verification (No-issue)

Applied the Kernel argument MC

$ oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker  
  name: 05-worker-kernelarg-selinuxpermissive  
spec:
  kernelArguments:
    - enforcing=0
EOF

Updated to rhel10.
Applied MC on rhel-10 -> switch

Custom Pool (No-issue)

• Created custom Pool and check the switch on the node

Scaling Node (No-issue)

• Able to Scale node with rhel-9 and rhel-10

Scaling node on Custom Machineset (No-issue)

• Created the Custom MS and scaled the node for it

$ oc get machinesets.machine.openshift.io -n openshift-machine-api ppt-0701a-6m9l2-worker-us-east-1d -o yaml > ms10.yaml

• Able to create custom MS and scale node for it for rhel-9 and rhel-10

Pause (No-issue)

• Pause the MCP with rhel-9 -> Switch to rhel10 -> Unpause -> Switched to rhel10
• Pause the MCP with rhel-10 -> Switch to rhel9-> Unpause -> Switched to rhel9

Max Unavailable (No-issue)

Kernel Argument (issue)

• Applied MC on rhel-9 cluster -> Switch to rhel10 -> Error
• Applied MC on rhel-10 -> Switch to rhel-9 -> MCP updated with error and modified to rhel10

oc create -f - << EOF                                                       
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker 
  name: worker-kernel-rt
spec:
  kernelType: realtime

EOF
machineconfig.machineconfiguration.openshift.io/worker-kernel-rt created

Error seen

  - lastTransitionTime: "2026-01-08T12:42:26Z"
    message: 'Node ip-10-0-62-222.us-east-2.compute.internal is reporting: "Node ip-10-0-62-222.us-east-2.compute.internal
      upgrade failure. error running rpm-ostree override remove kernel kernel-core
      kernel-modules kernel-modules-core kernel-modules-extra --install kernel-rt-core
      --install kernel-rt-modules --install kernel-rt-modules-extra --install kernel-rt-kvm:
      error: Packages not found: kernel-rt-kvm\n: exit status 1", Node ip-10-0-62-222.us-east-2.compute.internal
      is reporting: "error running rpm-ostree override remove kernel kernel-core kernel-modules
      kernel-modules-core kernel-modules-extra --install kernel-rt-core --install
      kernel-rt-modules --install kernel-rt-modules-extra --install kernel-rt-kvm:
      error: Packages not found: kernel-rt-kvm\n: exit status 1"'
    reason: 1 nodes are reporting degraded status on sync
    status: "True"
    type: NodeDegraded

Extension (No-issue)

• Applied MC on rhel-9 cluster -> Switch to rhel10 -> MCP got updated to rhel10
• Applied MC on rhel-10 -> Switch to rhel-9 -> MCP updated with error and modified to rhel10

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: master
  name: master-ext
spec:
  config:
    ignition:
      version: 3.1.0
  extensions:
    -  usbguard
EOF

SNO (no-issue)

FIPS based cluster (No-issue)

Verified the Day-1 installation (No-issue)

• Both for rhel10 and rhel-9

[pablintino]

/hold
Till https://issues.redhat.com/browse/OCPBUGS-71189 is resolved

[openshift-ci]

@pablintino: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/bootstrap-unit	d012c19	link	false	/test bootstrap-unit

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[ptalgulk01]

Pre-merge verification:
Environment Setup
OpenShift Version: 4.21.0-0-2026-01-15-073433-test-ci-ln-9gzhqyt-latest
Platform: AWS,proxy
Flexy Template: private-templates/functionality-testing/aos-4_21/ipi-on-gcp/versioned-installer_customer_vpc-http_proxy

• Tested on proxy based cluster with and without TechPreview enabled.

$ oc debug node/ip-10-0-70-104.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-70-104us-east-2computeinternal-debug-bsscz ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                   Digest: sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                  Version: 9.6.20260112-0 (2026-01-13T02:17:59Z)

Removing debug pod ...

# Patch the rhel-10
$ oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

# wait for mcp update
$ oc debug node/ip-10-0-48-232.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-48-232us-east-2computeinternal-debug-zc24c ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                   Digest: sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                  Version: 10.1.20260106-1 (2026-01-09T10:39:31Z)

Removing debug pod ...

For extension when we apply all extension MC the MCP gets degraded when we switch to rhel-10 which is reported
Tested using manual-installation too for rhel-10 installation.

/label qe-approved
/verified by @ptalgulk01

[openshift-ci-robot]

@pablintino: This pull request references MCO-2017 which is a valid jira issue.

Details

In response to this:

- What I did
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

- How to verify it

TBD

- Description for the changelog
This change is a manual cherry-pick of multiple PRs that backports the OSImageStreams to 4.21.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@ptalgulk01: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

Pre-merge verification:
Environment Setup
OpenShift Version: 4.21.0-0-2026-01-15-073433-test-ci-ln-9gzhqyt-latest
Platform: AWS,proxy
Flexy Template: private-templates/functionality-testing/aos-4_21/ipi-on-gcp/versioned-installer_customer_vpc-http_proxy

• Tested on proxy based cluster with and without TechPreview enabled.

$ oc debug node/ip-10-0-70-104.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-70-104us-east-2computeinternal-debug-bsscz ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                  Digest: sha256:905508f17491d092ce5c81283cac740bb7616d2dcf1947c883849a3a76cf9c4f
                 Version: 9.6.20260112-0 (2026-01-13T02:17:59Z)

Removing debug pod ...

# Patch the rhel-10
$ oc patch mcp worker --type merge -p '{"spec":{"osImageStream":{"name":"rhel-10"}}}'
machineconfigpool.machineconfiguration.openshift.io/worker patched

# wait for mcp update
$ oc debug node/ip-10-0-48-232.us-east-2.compute.internal -- chroot /host rpm-ostree status
Starting pod/ip-10-0-48-232us-east-2computeinternal-debug-zc24c ...
To use host binaries, run `chroot /host`
State: idle
Deployments:
* ostree-unverified-registry:registry.build10.ci.openshift.org/ci-ln-9gzhqyt/stable@sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                  Digest: sha256:c8bfbeb76f9ff28b68e396cbbdab65fc028460d641e0d8f53ed48bf7aaabb875
                 Version: 10.1.20260106-1 (2026-01-09T10:39:31Z)

Removing debug pod ...

For extension when we apply all extension MC the MCP gets degraded when we switch to rhel-10 which is reported
Tested using manual-installation too for rhel-10 installation.

/label qe-approved
/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.