OTA-1559: test/extended/cli/admin: Add 'oc adm upgrade recommend' smoke test #29831
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wking
changed the title
openshift-ci
bot
added
the
do-not-merge/work-in-progress
wking
commented
May 20, 2025
test/extended/cli/admin.go
Outdated
| g.It("runs successfully, even without upstream OpenShift Update Service customization", func() { | ||
| out, err := oc.Run("adm", "upgrade", "recommend").EnvironmentVariables("OC_ENABLE_CMD_UPGRADE_RECOMMEND=true").Output() | ||
| o.Expect(err).NotTo(o.HaveOccurred()) | ||
| o.Expect(out).To(o.MatchRegexp(`FIXME: not sure what to expect so fail`)) |
Member
Author
There was a problem hiding this comment.
We'll want something based on reality in the regexp here before we merge. I'm hoping presubmits will run the new test-case, fail, and give me example reality-based output I can use to craft a regexp.
Member
Author
There was a problem hiding this comment.
While waiting for feedback on openshift/api#2337, I just dropped the feature-gate portion of the string, and hooray, the test case runs :)
: [sig-cli] oc adm upgrade recommend runs successfully, even without upstream OpenShift Update Service customization [Suite:openshift/conformance/parallel]
Run #0: Failed 2s
{ fail [github.com/openshift/origin/test/extended/cli/admin.go:634]: Expected
<string>: warning: Cannot refresh available updates:
Reason: NoChannel
Message: The update channel has not been configured.
No updates available. You may still upgrade to a specific release image with --to-image or wait for new updates to be available.
to match regular expression
<string>: FIXME: not sure what to expect so fail
Ginkgo exit error 1: exit with code 1}
which is getting us into this code. I dunno if I want to assume that all clusters will have their channel unset, although most CI clusters will (openshift/release#40711). I guess we can start building out a case structure here where we try and predict the output based on different cluster configurations, but if we go too far down that road it ends up being "complicated code in the test-case predicts that similar complicated code in the product is generating similar results", and we can miss catching situations that break the test-case logic and the product logic in similar ways.
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
from
e685bb5 to
26183b7
Compare
Member
Author
|
e2e-aws-ovn died in prep work, before getting close to the test-case I'm trying to add: /retest-required |
|
Job Failure Risk Analysis for sha: 26183b7
|
Member
|
/cc |
wking
commented
May 21, 2025
|
|
||
| "[sig-cli][Feature:LegacyCommandTests][Disruptive][Serial] test-cmd: test/cmd/volumes.sh [apigroup:image.openshift.io]": "", | ||
|
|
||
| "[sig-cli][OCPFeatureGate:OCAdminUpgradeRecommend] oc adm upgrade recommend runs successfully, even without upstream OpenShift Update Service customization": " [Suite:openshift/conformance/parallel]", |
Member
Author
There was a problem hiding this comment.
Doesn't seem to be running in the stock parallel suite, looking at e2e-aws-ovn, with no sign of either OCAdminUpgradeRecommend or oc adm upgrade recommend:
$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/29831/pull-ci-openshift-origin-main-e2e-aws-ovn/1924979491902328832/artifacts/e2e-aws-ovn/openshift-e2e-test/build-log.txt | grep '^started:.*sig-cli.*oc adm [ru]'
started: 1/109/565 "[sig-cli] oc adm user-creation [apigroup:user.openshift.io] [Suite:openshift/conformance/parallel]"
started: 2/198/565 "[sig-cli] oc adm role-selectors [apigroup:template.openshift.io] [Suite:openshift/conformance/parallel]"
started: 2/212/565 "[sig-cli] oc adm ui-project-commands [apigroup:project.openshift.io][apigroup:authorization.openshift.io][apigroup:user.openshift.io] [Suite:openshift/conformance/parallel]"
started: 2/467/565 "[sig-cli] oc adm release extract image-references [Suite:openshift/conformance/parallel]"
started: 3/547/565 "[sig-cli] oc adm role-reapers [apigroup:authorization.openshift.io][apigroup:user.openshift.io] [Suite:openshift/conformance/parallel]"There also don't seem to be any tech-preview presubmits?
Member
Author
There was a problem hiding this comment.
I've opened openshift/api#2337, which will hopefully give the test suite a hint that a feature-gated test should run... somewhere...
Member
Author
There was a problem hiding this comment.
Testing against a default-feature set cluster without setting a feature-gate on the test-case works.
Member
|
There's plenty of optional techpreview presubmits /test |
Contributor
|
@stbenjam: The The following commands are available to trigger optional jobs: Use DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
from
26183b7 to
63d080c
Compare
|
Job Failure Risk Analysis for sha: 63d080c
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: 63d080c
New tests seen in this PR at sha: 63d080c
|
wking
changed the title
openshift-ci-robot
added
the
jira/valid-reference
|
@wking: This pull request references OTA-1559 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.20.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
from
63d080c to
8e0e528
Compare
|
Job Failure Risk Analysis for sha: 8e0e528
|
petr-muller
reviewed
Jul 11, 2025
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
from
8e0e528 to
340ba80
Compare
openshift-ci
bot
added
the
approved
|
Job Failure Risk Analysis for sha: 340ba80
Showing 20 of 43 jobs analysis |
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
from
340ba80 to
9194506
Compare
|
Job Failure Risk Analysis for sha: 9194506
Showing 20 of 49 jobs analysis |
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
2 times, most recently
from
cf4675f to
58c7356
Compare
openshift-ci
bot
removed
the
approved
Member
Author
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-serial 4 |
Contributor
|
@wking: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/590f90d0-725e-11f0-8421-07b559d8b3e4-0 |
|
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: bfed67f
New tests seen in this PR at sha: bfed67f
|
|
Risk analysis has seen new tests most likely introduced by this PR. New Test Risks for sha: bfed67f
New tests seen in this PR at sha: bfed67f
|
wking
force-pushed
the
smoke-test-oc-adm-upgrade-recommend
branch
from
bfed67f to
44cd78a
Compare
Member
Author
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.20-e2e-aws-ovn-serial 4 |
Contributor
|
@wking: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/00726410-728c-11f0-8be6-402aa3e41afb-0 |
The message I'd been expecting before worked for Upgradeable=True
clusters. But we have some clusters that are Upgradeable=False, like
our tech-preview jobs, and they'll have messages like [1]:
Failing=True:
Reason: ClusterOperatorDegraded
Message: Cluster operator network is degraded
Upstream update service: http://172.30.28.176:8000/graph
Channel: test-channel (available channels: other-channel, test-channel)
Updates to 4.21:
Version: 4.21.0
Image: example.com/test@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
Reason: MultipleReasons
Message: Cluster operator config-operator should not be upgraded between minor versions: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow updates
This is a test risk. https://example.com/testRiskA
Updates to 4.20:
VERSION ISSUES
4.20.999 no known issues relevant to this cluster
4.20.998 no known issues relevant to this cluster
and [2]:
Upstream update service: http://172.30.152.185:8000/graph
Channel: test-channel (available channels: other-channel, test-channel)
Update to 4.21.0 Recommended=False:
Image: example.com/test@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
Release URL: https://example.com/release/4.21.0
Reason: MultipleReasons
Message: Cluster operator config-operator should not be upgraded between minor versions: FeatureGatesUpgradeable: "TechPreviewNoUpgrade" does not allow update
This is a test risk. https://example.com/testRiskA
The opening Failing=True doesn't matter, because I'm not anchoring my
regexp with a ^. The new (TestRiskA|MultipleReasons) blocks cover
both "TestRiskA alone" and "TestRiskA and other stuff too". The
(?s:.*) in the message section also allows us to absorb additional
unrelated-issue wording, with the (?s:<re>) portion setting a flag so
'.' can match \n newlines [3].
[1]: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29831/pull-ci-openshift-origin-main-e2e-gcp-ovn-techpreview-serial-2of2/1952728832490344448
[2]: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29831/pull-ci-openshift-origin-main-e2e-gcp-ovn-techpreview-serial-1of2/1952716346328354816
[3]: https://github.com/google/re2/wiki/Syntax
|
Job Failure Risk Analysis for sha: 44cd78a
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: 44cd78a
|
|
Job Failure Risk Analysis for sha: 44cd78a
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: 44cd78a
|
Member
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hongkailiu, PratikMahajan, stbenjam, wking The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
Job Failure Risk Analysis for sha: 44cd78a
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: 44cd78a
|
Member
Author
|
Tide sounds like it wants this one ( /test e2e-vsphere-ovn-upi |
Member
Author
|
Retest for e2e-gcp-ovn doesn't seem to be starting. Explicitly ask for that too, in case it wiggles Prow loose: /test e2e-gcp-ovn |
|
Job Failure Risk Analysis for sha: 44cd78a
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: 44cd78a
|
1 similar comment
|
Job Failure Risk Analysis for sha: 44cd78a
Risk analysis has seen new tests most likely introduced by this PR. New tests seen in this PR at sha: 44cd78a
|
Member
Author
|
/retest-required |
Contributor
|
@wking: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
Job Failure Risk Analysis for sha: 44cd78a
|
openshift-merge-bot
bot
merged commit 7ccc307
into
openshift:main
37 of 47 checks passed
Contributor
|
[ART PR BUILD NOTIFIER] Distgit: openshift-enterprise-tests |
wking
added a commit
to wking/oc
that referenced
this pull request
Aug 13, 2025
Most OpenShift feature gates are tracked in openshift/api. This one
is just the local environment variable, but we still want to meet the
usual tech-preview-to-GA promotion criteria [1]:
* Tests must contain either [OCPFeatureGate:<FeatureGateName>] or the
standard upstream [FeatureGate:<FeatureGateName>].
This one isn't relevant to this feature, because the test-cases are
ungated to run the tech-preview functionality against production
clusters [2]. As docs say [3]:
Your cluster does not need to be a Technology Preview-enabled
cluster in order for you to use the 'oc adm upgrade recommend'
command.
* There must be at least five tests for each FeatureGate.
Sippy returns exactly five [4]:
[Serial][sig-cli] oc adm upgrade recommend runs successfully, even without upstream OpenShift Update Service customization [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend runs successfully with an empty channel [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend When the update service has conditional recommendations runs successfully when listing all updates [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend When the update service has conditional recommendations runs successfully with conditional recommendations to the --version target [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend When the update service has no recommendations runs successfully [Suite:openshift/conformance/serial]
* Every test must be run on every TechPreview platform we have jobs
for. (Ask for an exception if your feature doesn't support a
variant.)
* Every test must run at least 14 times on every platform/variant.
* Every test must pass at least 95% of the time on every
platform/variant.
Checking on one of the least-run test-cases ("... with conditional
recommendations to the --version target"), with 124 runs [5]:
$ curl -s 'https://sippy.dptools.openshift.org/api/tests?release=4.20&filter=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22%5BSerial%5D%5Bsig-cli%5D%20oc%20adm%20upgrade%20recommend%20When%20the%20update%20service%20has%20conditional%20recommendations%20runs%20successfully%20with%20conditional%20recommendations%20to%20the%20--version%20target%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&period=default&sortField=delta_from_passing_average&sort=asc&collapse=false' | jq -r '[.[] | select(.variants) | ([.variants[] | select(startswith("Platform:") or startswith("Architecture:") or startswith("Topology:") or startswith("NetworkStack:"))] | tostring) as $platform | {current_runs, current_successes, platform: $platform}] | group_by(.platform)[] | (map(.current_runs) | add | tostring) + " " + (map(.current_successes) | add | tostring) + " " + .[0].platform' | sort -n
1 1 ["Platform:none","Architecture:ppc64le","NetworkStack:ipv4","Topology:ha"]
1 1 ["Platform:none","Architecture:s390x","NetworkStack:ipv4","Topology:ha"]
5 5 ["Platform:openstack","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
6 6 ["Platform:aws","Architecture:multi","NetworkStack:ipv4","Topology:ha"]
6 6 ["Platform:metal","Architecture:amd64","NetworkStack:ipv6","Topology:ha"]
7 7 ["Platform:metal","Architecture:amd64","NetworkStack:dual","Topology:ha"]
10 10 ["Platform:aws","Architecture:arm64","NetworkStack:ipv4","Topology:ha"]
11 11 ["Platform:gcp","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
12 12 ["Platform:azure","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
13 13 ["Platform:metal","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
14 14 ["Platform:aws","Architecture:amd64","NetworkStack:ipv4","Topology:single"]
17 17 ["Platform:aws","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
21 21 ["Platform:vsphere","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
As a platform-agnostic feature, it seems like we need these variants [5]:
{"Cloud":"aws","Architecture":"amd64","Topology":"ha"} (and we have 17)
{"Cloud":"azure","Architecture":"amd64","Topology":"ha"} (and we have 12, a bit short of 14)
{"Cloud":"gcp","Architecture":"amd64","Topology":"ha"} (and we have 11, a bit short of 14)
{"Cloud":"vsphere","Architecture":"amd64","Topology":"ha"} (and we have 21)
{"Cloud":"metal","Architecture":"amd64","Topology":"ha","NetworkStack":"ipv4"} (and we have 13, one short of 14)
{"Cloud":"metal","Architecture":"amd64","Topology":"ha","NetworkStack":"ipv6"} (and we have 6, well short of 14)
{"Cloud":"metal","Architecture":"amd64","Topology":"ha","NetworkStack":"dual"} (and we have 7, well short of 14)
{"Cloud":"aws","Architecture":"amd64","Topology":"single"} (and we have 14)
* Test results are taken from the last 7 days if the test was run at
least 14 times during that period. Otherwise, data from the last 14
days is used.
The tests are still less than a week old, so this isn't relevant.
* Test flakes (even if the test eventually passes on a retry) are
considered failures and negatively impact the pass rate.
Every test-case has a 100% pass rate [4], so this isn't relevant.
So Azure, GCP, and metal are all short. But given this is
platform-agnostic code, and the success rate so far is 100% across all
variants, I'm confident enough to drop the guard now. Waiting another
week would not be wrong either.
[1]: https://github.com/openshift/api/blob/88b2b21555f3c12755740b197cbd5b9b4ca11e19/README.md#defining-featuregate-e2e-tests
[2]: openshift/origin#29831
[3]: https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/updating_clusters/performing-a-cluster-update#update-upgrading-oc-adm-upgrade-recommend_updating-cluster-cli
[4]: https://sippy.dptools.openshift.org/sippy-ng/tests/4.20?filters=%257B%2522items%2522%253A%255B%257B%2522columnField%2522%253A%2522current_runs%2522%252C%2522operatorValue%2522%253A%2522%253E%253D%2522%252C%2522value%2522%253A%25227%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522never-stable%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aggregated%2522%257D%252C%257B%2522columnField%2522%253A%2522current_flake_percentage%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522%253D%2522%252C%2522value%2522%253A%2522100%2522%257D%252C%257B%2522id%2522%253A99%252C%2522columnField%2522%253A%2522name%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522oc%2520adm%2520upgrade%2520recommend%2522%257D%255D%252C%2522linkOperator%2522%253A%2522and%2522%257D&sort=asc&sortField=net_improvement
[5]: https://sippy.dptools.openshift.org/sippy-ng/tests/4.20/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22%5BSerial%5D%5Bsig-cli%5D%20oc%20adm%20upgrade%20recommend%20When%20the%20update%20service%20has%20conditional%20recommendations%20runs%20successfully%20with%20conditional%20recommendations%20to%20the%20--version%20target%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&pageSize=50&test=%5BSerial%5D%5Bsig-cli%5D%20oc%20adm%20upgrade%20recommend%20When%20the%20update%20service%20has%20conditional%20recommendations%20runs%20successfully%20with%20conditional%20recommendations%20to%20the%20--version%20target%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D
[6]: https://github.com/openshift/api/blob/88b2b21555f3c12755740b197cbd5b9b4ca11e19/tools/codegen/cmd/featuregate-test-analyzer.go#L332-L375
wking
added a commit
to wking/oc
that referenced
this pull request
Aug 13, 2025
Most OpenShift feature gates are tracked in openshift/api. This one
is just the local environment variable, but we still want to meet the
usual tech-preview-to-GA promotion criteria [1]:
* Tests must contain either [OCPFeatureGate:<FeatureGateName>] or the
standard upstream [FeatureGate:<FeatureGateName>].
This one isn't relevant to this feature, because the test-cases are
ungated to run the tech-preview functionality against production
clusters [2]. As docs say [3]:
Your cluster does not need to be a Technology Preview-enabled
cluster in order for you to use the 'oc adm upgrade recommend'
command.
* There must be at least five tests for each FeatureGate.
Sippy returns exactly five [4]:
[Serial][sig-cli] oc adm upgrade recommend runs successfully, even without upstream OpenShift Update Service customization [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend runs successfully with an empty channel [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend When the update service has conditional recommendations runs successfully when listing all updates [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend When the update service has conditional recommendations runs successfully with conditional recommendations to the --version target [Suite:openshift/conformance/serial]
[Serial][sig-cli] oc adm upgrade recommend When the update service has no recommendations runs successfully [Suite:openshift/conformance/serial]
* Every test must be run on every TechPreview platform we have jobs
for. (Ask for an exception if your feature doesn't support a
variant.)
* Every test must run at least 14 times on every platform/variant.
* Every test must pass at least 95% of the time on every
platform/variant.
Checking on one of the least-run test-cases ("... with conditional
recommendations to the --version target"), with 124 runs [5]:
$ curl -s 'https://sippy.dptools.openshift.org/api/tests?release=4.20&filter=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22%5BSerial%5D%5Bsig-cli%5D%20oc%20adm%20upgrade%20recommend%20When%20the%20update%20service%20has%20conditional%20recommendations%20runs%20successfully%20with%20conditional%20recommendations%20to%20the%20--version%20target%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&period=default&sortField=delta_from_passing_average&sort=asc&collapse=false' | jq -r '[.[] | select(.variants) | ([.variants[] | select(startswith("Platform:") or startswith("Architecture:") or startswith("Topology:") or startswith("NetworkStack:"))] | tostring) as $platform | {current_runs, current_successes, platform: $platform}] | group_by(.platform)[] | (map(.current_runs) | add | tostring) + " " + (map(.current_successes) | add | tostring) + " " + .[0].platform' | sort -n
1 1 ["Platform:none","Architecture:ppc64le","NetworkStack:ipv4","Topology:ha"]
1 1 ["Platform:none","Architecture:s390x","NetworkStack:ipv4","Topology:ha"]
5 5 ["Platform:openstack","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
6 6 ["Platform:aws","Architecture:multi","NetworkStack:ipv4","Topology:ha"]
6 6 ["Platform:metal","Architecture:amd64","NetworkStack:ipv6","Topology:ha"]
7 7 ["Platform:metal","Architecture:amd64","NetworkStack:dual","Topology:ha"]
10 10 ["Platform:aws","Architecture:arm64","NetworkStack:ipv4","Topology:ha"]
11 11 ["Platform:gcp","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
12 12 ["Platform:azure","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
13 13 ["Platform:metal","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
14 14 ["Platform:aws","Architecture:amd64","NetworkStack:ipv4","Topology:single"]
17 17 ["Platform:aws","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
21 21 ["Platform:vsphere","Architecture:amd64","NetworkStack:ipv4","Topology:ha"]
As a platform-agnostic feature, it seems like we need these variants [6]:
{"Cloud":"aws","Architecture":"amd64","Topology":"ha"} (and we have 17)
{"Cloud":"azure","Architecture":"amd64","Topology":"ha"} (and we have 12, a bit short of 14)
{"Cloud":"gcp","Architecture":"amd64","Topology":"ha"} (and we have 11, a bit short of 14)
{"Cloud":"vsphere","Architecture":"amd64","Topology":"ha"} (and we have 21)
{"Cloud":"metal","Architecture":"amd64","Topology":"ha","NetworkStack":"ipv4"} (and we have 13, one short of 14)
{"Cloud":"metal","Architecture":"amd64","Topology":"ha","NetworkStack":"ipv6"} (and we have 6, well short of 14)
{"Cloud":"metal","Architecture":"amd64","Topology":"ha","NetworkStack":"dual"} (and we have 7, well short of 14)
{"Cloud":"aws","Architecture":"amd64","Topology":"single"} (and we have 14)
* Test results are taken from the last 7 days if the test was run at
least 14 times during that period. Otherwise, data from the last 14
days is used.
The tests are still less than a week old, so this isn't relevant.
* Test flakes (even if the test eventually passes on a retry) are
considered failures and negatively impact the pass rate.
Every test-case has a 100% pass rate [4], so this isn't relevant.
So Azure, GCP, and metal are all short. But given this is
platform-agnostic code, and the success rate so far is 100% across all
variants, I'm confident enough to drop the guard now. Waiting another
week would not be wrong either.
[1]: https://github.com/openshift/api/blob/88b2b21555f3c12755740b197cbd5b9b4ca11e19/README.md#defining-featuregate-e2e-tests
[2]: openshift/origin#29831
[3]: https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/updating_clusters/performing-a-cluster-update#update-upgrading-oc-adm-upgrade-recommend_updating-cluster-cli
[4]: https://sippy.dptools.openshift.org/sippy-ng/tests/4.20?filters=%257B%2522items%2522%253A%255B%257B%2522columnField%2522%253A%2522current_runs%2522%252C%2522operatorValue%2522%253A%2522%253E%253D%2522%252C%2522value%2522%253A%25227%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522never-stable%2522%257D%252C%257B%2522columnField%2522%253A%2522variants%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522aggregated%2522%257D%252C%257B%2522columnField%2522%253A%2522current_flake_percentage%2522%252C%2522not%2522%253Atrue%252C%2522operatorValue%2522%253A%2522%253D%2522%252C%2522value%2522%253A%2522100%2522%257D%252C%257B%2522id%2522%253A99%252C%2522columnField%2522%253A%2522name%2522%252C%2522operatorValue%2522%253A%2522contains%2522%252C%2522value%2522%253A%2522oc%2520adm%2520upgrade%2520recommend%2522%257D%255D%252C%2522linkOperator%2522%253A%2522and%2522%257D&sort=asc&sortField=net_improvement
[5]: https://sippy.dptools.openshift.org/sippy-ng/tests/4.20/analysis?filters=%7B%22items%22%3A%5B%7B%22columnField%22%3A%22name%22%2C%22operatorValue%22%3A%22equals%22%2C%22value%22%3A%22%5BSerial%5D%5Bsig-cli%5D%20oc%20adm%20upgrade%20recommend%20When%20the%20update%20service%20has%20conditional%20recommendations%20runs%20successfully%20with%20conditional%20recommendations%20to%20the%20--version%20target%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22never-stable%22%7D%2C%7B%22columnField%22%3A%22variants%22%2C%22not%22%3Atrue%2C%22operatorValue%22%3A%22contains%22%2C%22value%22%3A%22aggregated%22%7D%5D%2C%22linkOperator%22%3A%22and%22%7D&pageSize=50&test=%5BSerial%5D%5Bsig-cli%5D%20oc%20adm%20upgrade%20recommend%20When%20the%20update%20service%20has%20conditional%20recommendations%20runs%20successfully%20with%20conditional%20recommendations%20to%20the%20--version%20target%20%5BSuite%3Aopenshift%2Fconformance%2Fserial%5D
[6]: https://github.com/openshift/api/blob/88b2b21555f3c12755740b197cbd5b9b4ca11e19/tools/codegen/cmd/featuregate-test-analyzer.go#L332-L375
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hear that folks want something similar to the OpenShift API conditions:
before promoting command-line features to GA. Or maybe that is a condition before shifting them from alpha to beta, like this upstream change moving a commend-line gate from
IsEnabledto!IsDisabled? Anyhow, I like testing, and while in the origin suite we don't really know what, if anything, we'll find in the ClusterVersion status properties therecommendcommand is reporting on, this commit at least runs the command to confirm it doesn't fail to execute.