WIP: fix(upsert): add last-applied-configuration annotation to detect manifest removals#7713
WIP: fix(upsert): add last-applied-configuration annotation to detect manifest removals#7713muraee wants to merge 2 commits intoopenshift:mainfrom
Conversation
…annotation Regenerate all testdata fixture files to include the new hypershift.openshift.io/last-applied-configuration annotation that will be stamped by ApplyManifest. Signed-off-by: Mulham Raee <mulham.raee@gmail.com> Commit-Message-Assisted-by: Claude (via Claude Code)
…fest removals DeepDerivative has fundamental flaws: it ignores trailing slice removals and treats zero-value fields as "don't care". This causes bugs like OCPBUGS-65879 where removing --image-overrides args goes undetected. Add a last-applied-configuration annotation pattern to ApplyManifest. After each successful apply, the desired manifest state is stored as JSON on the object. On subsequent reconciles, the new desired state is compared against this annotation to detect any change, including removals that DeepDerivative would miss. DeepDerivative is retained as a fallback for drift detection (external modifications to cluster objects) and migration (objects created before this change). On migration, the annotation is force-stamped on the first reconcile. Signed-off-by: Mulham Raee <mulham.raee@gmail.com> Commit-Message-Assisted-by: Claude (via Claude Code)
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
muraee
changed the title
WalkthroughThis pull request updates test fixture YAML manifests across the control-plane-operator testdata directory. Changes primarily add Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
openshift-ci-robot
added
the
jira/valid-reference
|
@muraee: This pull request explicitly references no jira issue. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
area/control-plane-operator
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: muraee The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
area/hypershift-operator
|
@muraee: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In
`@control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/AROSwift/zz_fixture_TestControlPlaneComponents_azure_cloud_config_secret.yaml`:
- Around line 6-7: The last-applied-configuration annotation on the Secret
(annotation key hypershift.openshift.io/last-applied-configuration for Secret
name azure-cloud-config) contains embedded sensitive Secret data; remove that
annotation or replace its value with a non-sensitive placeholder in the fixture
so secret material is not stored in metadata. Locate the Secret YAML containing
the hypershift.openshift.io/last-applied-configuration annotation and either
delete the entire annotation entry or replace the embedded "cloud.conf" blob
with a sanitized placeholder (e.g., "<redacted>" or an empty object) and ensure
ownerReferences/metadata remain intact; for future fixtures prefer using
Server-Side Apply/managedFields or explicit, non-sensitive test values instead
of embedding real secret contents.
| annotations: | ||
| hypershift.openshift.io/last-applied-configuration: '{"apiVersion":"v1","data":{"cloud.conf":"ewogICJjbG91ZCI6ICIiLAogICJ0ZW5hbnRJZCI6ICIiLAogICJ1c2VNYW5hZ2VkSWRlbnRpdHlFeHRlbnNpb24iOiBmYWxzZSwKICAidXNlRmVkZXJhdGVkV29ya2xvYWRJZGVudGl0eUV4dGVuc2lvbiI6IGZhbHNlLAogICJzdWJzY3JpcHRpb25JZCI6ICIiLAogICJhYWRDbGllbnRJZCI6ICIiLAogICJhYWRDbGllbnRTZWNyZXQiOiAiIiwKICAiYWFkQ2xpZW50Q2VydFBhdGgiOiAiIiwKICAiYWFkRmVkZXJhdGVkVG9rZW5GaWxlIjogIiIsCiAgImFhZE1TSURhdGFQbGFuZUlkZW50aXR5UGF0aCI6ICIvbW50L2NlcnRzL2Nsb3VkLXByb3ZpZGVyLWNyZWRzIiwKICAicmVzb3VyY2VHcm91cCI6ICIiLAogICJsb2NhdGlvbiI6ICIiLAogICJ2bmV0TmFtZSI6ICJteVZuZXROYW1lIiwKICAidm5ldFJlc291cmNlR3JvdXAiOiAibXlSZXNvdXJjZUdyb3VwTmFtZSIsCiAgInN1Ym5ldE5hbWUiOiAibXlTdWJuZXROYW1lIiwKICAic2VjdXJpdHlHcm91cE5hbWUiOiAibXlOU0dOYW1lIiwKICAic2VjdXJpdHlHcm91cFJlc291cmNlR3JvdXAiOiAibXlSZXNvdXJjZUdyb3VwTmFtZSIsCiAgInJvdXRlVGFibGVOYW1lIjogIiIsCiAgImNsb3VkUHJvdmlkZXJCYWNrb2ZmIjogdHJ1ZSwKICAiY2xvdWRQcm92aWRlckJhY2tvZmZEdXJhdGlvbiI6IDYsCiAgInVzZUluc3RhbmNlTWV0YWRhdGEiOiBmYWxzZSwKICAibG9hZEJhbGFuY2VyU2t1IjogInN0YW5kYXJkIiwKICAiZGlzYWJsZU91dGJvdW5kU05BVCI6IHRydWUsCiAgImxvYWRCYWxhbmNlck5hbWUiOiAiIiwKICAiY2x1c3RlclNlcnZpY2VMb2FkQmFsYW5jZXJIZWFsdGhQcm9iZU1vZGUiOiAic2hhcmVkIgp9"},"kind":"Secret","metadata":{"name":"azure-cloud-config","namespace":"hcp-namespace","ownerReferences":[{"apiVersion":"hypershift.openshift.io/v1beta1","blockOwnerDeletion":true,"controller":true,"kind":"HostedControlPlane","name":"hcp","uid":""}]}}' |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Find files that reference this annotation or handle last-applied-configuration
rg "last-applied-configuration" --type yaml --type go -A 2 -B 2 | head -100Repository: openshift/hypershift
Length of output: 253
🏁 Script executed:
# Search for annotation redaction or sanitization logic
rg "redact|sanitize" --type go -i | head -50Repository: openshift/hypershift
Length of output: 253
🏁 Script executed:
# Check if this fixture file has any special handling
fd "zz_fixture.*azure_cloud_config" --type fRepository: openshift/hypershift
Length of output: 253
🏁 Script executed:
# Look for test code that uses or validates this fixture
rg "azure_cloud_config_secret|AROSwift" --type go | head -20Repository: openshift/hypershift
Length of output: 253
🏁 Script executed:
# Check Kubernetes documentation about last-applied-configuration best practices
web_search
Kubernetes last-applied-configuration annotation security best practices sensitive dataRepository: openshift/hypershift
Length of output: 253
🌐 Web query:
Kubernetes last-applied-configuration annotation security sensitive data best practices
💡 Result:
Why the last-applied-configuration annotation is sensitive
kubectl client-side apply stores (a copy of) what you applied in the kubectl.kubernetes.io/last-applied-configuration annotation. If the manifest contains sensitive values (commonly Secrets, but also tokens in CRDs, ConfigMaps, etc.), those values can end up persisted in object metadata and become readable to anyone/tools that can read that object. This pattern has caused real-world leakage (for example, Argo CD cluster Secrets where the full secret body was present in the annotation). [3]
Best practices (security-focused)
-
Prefer Server-Side Apply (SSA) over client-side apply
Kubernetes SSA is designed as the replacement for legacy client-side apply and does not rely on the legacy “last-applied” annotation for its core mechanism (it uses field management /managedFieldsinstead). Use:kubectl apply --server-side
Also consider using a non-default field manager (example in docs) if you want to opt out of kubectl behaviors tied to the legacy annotation. [1]
-
Do not manage Secrets by “applying” Secret manifests that contain the secret material
Kubernetes explicitly warns against sharing/checking in Secret manifests (base64 is not encryption). If you put secret material in YAML and apply it, you’re increasing the number of places that secret can leak (git, CI logs, the annotation, etc.). [2] -
Remove the annotation from existing sensitive objects (and rotate if needed)
If a Secret (or any sensitive resource) already has the annotation containing sensitive content, delete it from the live object (and consider rotating the secret, depending on exposure). Advisories for real incidents note that existing Secrets may require manual annotation removal. [3] -
Lock down RBAC around Secrets and other sensitive resources
Minimize who canget/list/watchSecrets. Kubernetes notes that grantinglistaccess effectively enables retrieving Secret contents, and that broader permissions (like being able to create Pods in a namespace) can also lead to Secret exposure. [2] -
Use an external secret source when possible
Prefer integrating a dedicated secret store (for example via CSI driver / external secrets patterns) so you don’t ship secret values inside Kubernetes manifests at all. (Kubernetes docs recommend avoiding sharing Secret manifests and discuss provider-based approaches.) [2]
Practical rule of thumb
If a manifest contains credentials, keys, tokens, or private endpoints: assume last-applied-configuration would be a data leak if present, and use SSA / external secrets + strict RBAC.
Sources: Kubernetes Server-Side Apply docs [1], Kubernetes Secret security best practices [2], CVE/incident documentation showing full secret bodies stored in last-applied-configuration [3].
🌐 Web query:
Kubernetes Secret metadata annotation logging security exposure risks
💡 Result:
Why Secret metadata.annotations can increase logging-related exposure risk
-
Annotations are intended to be broadly retrievable metadata
- Kubernetes annotations are “arbitrary … metadata” that clients can retrieve from objects. If you put credentials, tokens, or private URLs in annotations, you’re treating a widely-consumed metadata channel as a secret store. [1]
-
Audit logging level can accidentally capture Secret bodies (including annotations)
- Kubernetes audit policies have levels:
Metadata: logs request metadata but not request/response bodies.Request/RequestResponse: include the request body (and possibly response body). [2]
- If Secrets are logged at
RequestorRequestResponse, the audit log can contain the full Secret object payload from the API request/response (which includesmetadata.annotations, and for Secrets also includesdata). Many example policies explicitly log Secrets only atMetadatato reduce sensitive-data leakage risk. [2][3]
- Kubernetes audit policies have levels:
-
“Secret data in logs” is a known class of risk
- Kubernetes Secret guidance explicitly warns applications to avoid logging secret data in clear text—because logs are a common accidental exfiltration path. If you store sensitive values in annotations, they’re more likely to end up in log pipelines that capture object YAML/JSON for debugging or auditing. [4]
Practical risk scenarios
- A cluster admin (or managed Kubernetes defaults) enables audit at
Requestfor core resources → Secret create/update payloads (including annotations) can land in central logging. [2][3] - Troubleshooting / GitOps / admission controllers / operators log or export object manifests → annotations get copied into tickets, chat, SIEM, or traces (often with broader access than Secrets should have). [1][4]
Mitigations / best practices
- Never put sensitive values in annotations (treat them as non-secret metadata). [1]
- Ensure audit policy logs Secrets at
Metadata, notRequest/RequestResponse(unless you have strong redaction controls). [2][3] - Follow Secret handling guidance: restrict access, encrypt at rest, and ensure apps/operators avoid logging sensitive material. [4][5]
Sources: Kubernetes Annotations [1], Kubernetes Auditing (levels + example policy logging Secrets at Metadata) [2], GKE audit policy levels + Secrets at Metadata example [3], Kubernetes Secret good practices (avoid logging secrets) [4], Kubernetes Secrets cautions (sensitivity/access implications) [5].
Remove or exclude sensitive data from the last-applied-configuration annotation.
The hypershift.openshift.io/last-applied-configuration annotation embeds Secret data, which persists metadata as non-secret and risks exposure if the object is logged, audited, or exported. Per Kubernetes best practices, prefer Server-Side Apply (which uses managedFields instead of the legacy annotation) or omit Secret material from the annotation. Even in test fixtures, this pattern should not be replicated in production deployments.
🤖 Prompt for AI Agents
In
`@control-plane-operator/controllers/hostedcontrolplane/testdata/azure-cloud-controller-manager/AROSwift/zz_fixture_TestControlPlaneComponents_azure_cloud_config_secret.yaml`
around lines 6 - 7, The last-applied-configuration annotation on the Secret
(annotation key hypershift.openshift.io/last-applied-configuration for Secret
name azure-cloud-config) contains embedded sensitive Secret data; remove that
annotation or replace its value with a non-sensitive placeholder in the fixture
so secret material is not stored in metadata. Locate the Secret YAML containing
the hypershift.openshift.io/last-applied-configuration annotation and either
delete the entire annotation entry or replace the embedded "cloud.conf" blob
with a sanitized placeholder (e.g., "<redacted>" or an empty object) and ensure
ownerReferences/metadata remain intact; for future fixtures prefer using
Server-Side Apply/managedFields or explicit, non-sensitive test values instead
of embedding real secret contents.
There was a problem hiding this comment.
This appears to be an extremely costly update to a large fleet that is already being stressed with etcd size explosion caused by HyperShift node pools. I don't think IBM Cloud can afford having hypershift.openshift.io/last-applied-configuration added. I would like to discuss with you all at our interlock meeting next week.
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
muraee
changed the title
openshift-ci
bot
added
the
do-not-merge/work-in-progress
openshift-merge-robot
added
the
needs-rebase
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
4 participants
Summary
hypershift.openshift.io/last-applied-configurationannotation toApplyManifestthat stores the JSON representation of the desired manifest state after each successful applyDeepDerivativecannot detect (trailing slice removals, zero-value field clearing)DeepDerivativeas a fallback for drift detection (external modifications) and migration (objects without the annotation)Details
DeepDerivativehas two fundamental flaws:len(desired)items, so removing items from the end (e.g.,--image-overridesargs) goes undetectedThe annotation-based approach stores the full desired state as JSON on each apply. On next reconcile, a simple string comparison of
desiredJSON != lastAppliedJSONcatches any change including removals. Objects exceeding 128KB skip the annotation and fall back to DeepDerivative-only comparison.For migration, objects created before this change get the annotation force-stamped on their first reconcile.
Test plan
toUnstructuredfield strippingmake lintpassesmake vetpassesmake testpasses with race detection🤖 Generated with Claude Code