OCPBUGS-25055: pkg/cvo/sync_worker: Verification-failure details for unforced updates too #1003
Conversation
openshift-ci
bot
added
the
approved
wking
force-pushed
the
understand-signature-errors
branch
from
8196661 to
9c2e6ca
Compare
|
Testing 9c2e6ca with $ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
Requested update to release image quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000
$ oc -n openshift-cluster-version get -o json events | jq -r '.items[] | select(.reason | contains("Payload")) | .reason + ": " + .message' | grep verified | sort | uniq -c
5 RetrievePayloadFailed: Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
$ oc adm upgrade
Cluster version is 4.15.0-0.test-2023-12-08-022856-ci-ln-8rvgdc2-latest
ReleaseAccepted=False
Reason: RetrievePayload
Message: Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
warning: Cannot display available updates:
Reason: NoChannel
Message: The update channel has not been configured.hrm, that doesn't seem to be working... |
wking
force-pushed
the
understand-signature-errors
branch
2 times, most recently
from
28212c2 to
588b59c
Compare
|
Testing 588b59c with launch 4.15,#1003 aws Cluster Bot cluster (logs): $ oc adm upgrade --allow-explicit-upgrade --to-image quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000
warning: The requested upgrade image is not one of the available updates. You have used --allow-explicit-upgrade for the update to proceed anyway
Requested update to release image quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000
$ oc -n openshift-cluster-version get -o json events | jq -r '.items[] | select(.reason | contains("Payload")) | .reason + ": " + .message' | grep verified | sort | uniq -c
4 BRetrievePayloadFailed: A Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
$ oc adm upgrade
Cluster version is 4.15.0-0.test-2023-12-08-165243-ci-ln-qbk5hm2-latest
ReleaseAccepted=False
Reason: RetrievePayload
Message: C Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
warning: Cannot display available updates:
Reason: NoChannel
Message: The update channel has not been configured.
$ oc -n openshift-cluster-version logs -l k8s-app=cluster-version-operator --tail -1 | grep 'debug unwrapping' | tail -3
I1208 18:53:08.723595 1 sync_worker.go:1301] debug unwrapping 0 The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
I1208 18:53:08.723599 1 sync_worker.go:1301] debug unwrapping 1 unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
I1208 18:53:08.723602 1 sync_worker.go:1301] debug unwrapping 2 [2023-12-08T18:53:08Z: prefix sha256-0000000000000000000000000000000000000000000000000000000000000000 in config map signatures-managed: no more signatures to check, 2023-12-08T18:53:08Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=0000000000000000000000000000000000000000000000000000000000000000/signature-1: no more signatures to check, 2023-12-08T18:53:08Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=0000000000000000000000000000000000000000000000000000000000000000/signature-1: no more signatures to check, 2023-12-08T18:53:08Z: parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2023-12-08T18:53:08Z: serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2023-12-08T18:53:08Z: serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check]So still not working, but at least we have the detailed error message there in the logs; I just need to figure out why I'm failing to get it up into the Event. |
wking
force-pushed
the
understand-signature-errors
branch
from
588b59c to
a032989
Compare
…s too We've including verification-failure details for forced updates since 5ad3c14 (pkg/cvo/updatepayload: Event when forcing through a sig-verification failure, 2022-04-07, 2022, openshift#763), but had not been including them in logs or other output in the "we aren't forcing, so this blocks the update's acceptance" case. This commit adds the detail to the Event, so it's available, but keeps only the high-level message in the RetrievePayload status output (which feeds the ReleaseAccepted condition in ClusterVersion), because while the low-level are useful for debugging, they're pretty chatty for condition consumers that are more interested in just knowing basically why the update request isn't being accepted. The newline-to-// replacement is because apparently Event messages truncate at the first newline. I have not tracked down docs or source to back that up, but confirmed it in pre-merge testing [1]. [1]: openshift#1003 (comment)
wking
force-pushed
the
understand-signature-errors
branch
from
c70e2ec to
c75d15b
Compare
|
The issue seems to have been Event messages truncating at the first newline. Should be fixed with c75d15b. |
|
After removing the debugging content, testing c75d15b with $ oc -n openshift-cluster-version get -o json events | jq -r '.items[] | select(.reason | contains("Payload")) | .reason + ": " + .message' | grep verified | sort | uniq -c
1 RetrievePayloadFailed: Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat // [2023-12-09T01:33:34Z: prefix sha256-0000000000000000000000000000000000000000000000000000000000000000 in config map signatures-managed: no more signatures to check, 2023-12-09T01:33:34Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=0000000000000000000000000000000000000000000000000000000000000000/signature-1: no more signatures to check, 2023-12-09T01:33:34Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=0000000000000000000000000000000000000000000000000000000000000000/signature-1: no more signatures to check, 2023-12-09T01:33:34Z: parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2023-12-09T01:33:34Z: serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check, 2023-12-09T01:33:34Z: serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check]
$ oc adm upgrade
Cluster version is 4.15.0-0.test-2023-12-09-005930-ci-ln-0ryhgq2-latest
ReleaseAccepted=False
Reason: RetrievePayload
Message: Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
warning: Cannot display available updates:
Reason: NoChannel
Message: The update channel has not been configured.That detailed verify failure message is still fairly intimidating, but it has a lot of information. Formatting: $ oc -n openshift-cluster-version get -o json events | jq -r '.items[] | select(.reason | contains("Payload")) | .metadata.creationTimestamp + " " + .reason + ": " + .message' | grep verified | tail -n1 | sed 's| // |\n|g;s/, 2023/,\n2023/g'
2023-12-09T01:35:29Z RetrievePayloadFailed: Retrieving payload failed version="" image="quay.io/openshift-release-dev/ocp-release@sha256:0000000000000000000000000000000000000000000000000000000000000000" failure=The update cannot be verified: unable to verify sha256:0000000000000000000000000000000000000000000000000000000000000000 against keyrings: verifier-public-key-redhat
[2023-12-09T01:35:29Z: prefix sha256-0000000000000000000000000000000000000000000000000000000000000000 in config map signatures-managed: no more signatures to check,
2023-12-09T01:35:29Z: unable to retrieve signature from https://storage.googleapis.com/openshift-release/official/signatures/openshift/release/sha256=0000000000000000000000000000000000000000000000000000000000000000/signature-1: no more signatures to check,
2023-12-09T01:35:29Z: unable to retrieve signature from https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release/sha256=0000000000000000000000000000000000000000000000000000000000000000/signature-1: no more signatures to check,
2023-12-09T01:35:29Z: parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check,
2023-12-09T01:35:29Z: serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check,
2023-12-09T01:35:29Z: serial signature store wrapping config maps in openshift-config-managed with label "release.openshift.io/verification-signatures", serial signature store wrapping ClusterVersion signatureStores unset, falling back to default stores, parallel signature store wrapping containers/image signature store under https://storage.googleapis.com/openshift-release/official/signatures/openshift/release, containers/image signature store under https://mirror.openshift.com/pub/openshift-v4/signatures/openshift/release: no more signatures to check]showing us walking the ConfigMap store, failing to find any signatures, and then walking both default sig-store stores, failing to find any signatures, and then failing out the wrapping stores |
wking
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@wking: This pull request references Jira Issue OCPBUGS-25055, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/jira refresh |
openshift-ci-robot
added
the
jira/valid-bug
|
@wking: This pull request references Jira Issue OCPBUGS-25055, which is valid. The bug has been moved to the POST state. 3 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
removed
the
jira/invalid-bug
|
/label qe-approved |
openshift-ci
bot
added
the
qe-approved
|
@wking: This pull request references Jira Issue OCPBUGS-25055, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: petr-muller, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest |
|
@wking: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
@wking: Jira Issue OCPBUGS-25055: All pull requests linked via external trackers have merged: Jira Issue OCPBUGS-25055 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[ART PR BUILD NOTIFIER] This PR has been included in build cluster-version-operator-container-v4.16.0-202312142332.p0.gf10051b.assembly.stream for distgit cluster-version-operator. |
|
/cherrypick release-4.15 |
|
@wking: new pull request created: #1007 DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
…s too We've including verification-failure details for forced updates since 5ad3c14 (pkg/cvo/updatepayload: Event when forcing through a sig-verification failure, 2022-04-07, 2022, openshift#763), but had not been including them in logs or other output in the "we aren't forcing, so this blocks the update's acceptance" case. This commit adds the detail to the Event, so it's available, but keeps only the high-level message in the RetrievePayload status output (which feeds the ReleaseAccepted condition in ClusterVersion), because while the low-level are useful for debugging, they're pretty chatty for condition consumers that are more interested in just knowing basically why the update request isn't being accepted. The newline-to-// replacement is because apparently Event messages truncate at the first newline. I have not tracked down docs or source to back that up, but confirmed it in pre-merge testing [1]. [1]: openshift#1003 (comment)
6 participants
We've including verification-failure details for forced updates since 5ad3c14 (#763), but had not been including them in logs or other output in the "we aren't forcing, so this blocks the update's acceptance" case. This commit adds the detail to the Event, so it's available, but keeps only the high-level message in the
RetrievePayloadstatus output (which feeds theReleaseAcceptedcondition in ClusterVersion), because while the low-level are useful for debugging, they're pretty chatty for condition consumers that are more interested in just knowing basically why the update request isn't being accepted.