Skip to content

Conversation

@openshift-bot
Copy link
Contributor

    This is release candidate PR open atomatically by elliott

@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Mar 5, 2020
@openshift-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: openshift-bot
To complete the pull request process, please assign eparis
You can assign the PR to them by writing /assign @eparis in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@shiywang shiywang closed this Mar 5, 2020
@openshift-ci-robot
Copy link

@openshift-bot: The following test failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/prow/publish 7adc35b link /test publish

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

joepvd pushed a commit to joepvd/cincinnati-graph-data that referenced this pull request Mar 6, 2020
Baked in edges:

  $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.3.0-rc.0-x86_64 | grep Upgrades
    Upgrades: 4.2.13
  $ oc adm release info quay.io/openshift-release-dev/ocp-release:4.3.0-rc.3-x86_64 | grep Upgrades
    Upgrades: 4.2.16, 4.3.0-rc.0, 4.3.0-rc.1, 4.3.0-rc.2

The wide 'from' regexp was appropriate for 4.3.0-rc.0, which had no
4.3 update sources.  But rc.3 does have update sources, and we want to
allow 4.3.0-rc.0 -> 4.3.0-rc.3, because it is not impacted by the
4.2->4.3 GCP update bug.  The overly-strict regexp was from 6d3db09
(Blocking edges to candidate 4.3.0-rc.3, 2020-01-23, openshift#34).

Also expand the referenced bugs to for the blocked 4.2 -> 4.3 edges:

* Update hangs with [1]:

    Working towards 4.3.0...: 13% complete

  and machine-config going Degraded=True with RequiredPoolsFailed:

    Unable to apply 4.3.0-...: timed out waiting for the condition
    during syncRequiredMachineConfigPools: pool master has not
    progressed to latest configuration: controller version mismatch
    for rendered-master-6c22... expected 23a6... has d780... retrying

  Fixed in 4.2 with MCO 31fed93 [2] and in 4.2 with MCO 25bb6ae [3].

    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.2.14 | grep machine-config
      machine-config-operator                       https://github.com/openshift/machine-config-operator                       d780d197a9c5848ba786982c0c4aaa7487297046
    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.2.16 | grep machine-config
      machine-config-operator                       https://github.com/openshift/machine-config-operator                       31fed93186c9f84708f5cdfd0227ffe4f79b31cd

  So the 4.2 fix was in 4.2.16.

    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-rc.0 | grep machine-config
      machine-config-operator                       https://github.com/openshift/machine-config-operator                       23a6e6fb37e73501bc3216183ef5e6ebb15efc7a
    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-rc.3 | grep machine-config
      machine-config-operator                       https://github.com/openshift/machine-config-operator                       25bb6aeb58135c38a667e849edf5244871be4992

  So the 4.3 fix was new in rc.3.

* Updates hang with FailedCreatePodSandBox events in the
  openshift-ingress namespace like [4]:

    pod/router-default-...: Failed create pod sandbox: rpc error: code
    = Unknown desc = failed to create pod network sandbox
    k8s_router-default-..._openshift-ingress_...(...): Multus: error
    adding pod to network "openshift-sdn": delegateAdd: error invoking
    DelegateAdd - "openshift-sdn": error in getting result from
    AddNetwork: CNI request failed with status 400: 'failed to run
    IPAM for ...: failed to run CNI IPAM ADD: failed to allocate for
    range 0: no IP addresses available in range set: <ip1>-<ip2>

  Fixed in 4.2 with MCO 9366460 [5] and in 4.3 with MCO 311a01e [6].

    $ git --no-pager log --first-parent --oneline -4 origin/release-4.2
    6e0df82c (origin/release-4.2) Merge pull request openshift#1347 from openshift-cherrypick-robot/cherry-pick-1285-to-release-4.2
    93664600 Merge pull request openshift#1362 from rphillips/fixes/1787581_4.2
    bd358bb7 Merge pull request openshift#1323 from openshift-cherrypick-robot/cherry-pick-1320-to-release-4.2
    31fed931 Merge pull request openshift#1358 from runcom/osimageurl-race-42

  so the 4.2 fix was after 4.2.16's 31fed93186.

    $ git --no-pager log --first-parent --oneline -8 origin/release-4.3
    3ad3a836 (origin/release-4.3) Merge pull request openshift#1399 from celebdor/haproxy-v4v6
    25503eee Merge pull request openshift#1353 from russellb/1211-4.3-backport
    67ab306b Merge pull request openshift#1426 from mandre/ssc43
    d74f56fe Merge pull request openshift#1410 from retroflexer/manual-cherry-pick-from-master
    207cc171 Merge pull request openshift#1406 from openshift-cherrypick-robot/cherry-pick-1396-to-release-4.3
    25bb6aeb Merge pull request openshift#1359 from runcom/osimageurl-race-43
    311a01e8 Merge pull request openshift#1361 from rphillips/fixes/1787581_4.3
    23a6e6fb Merge pull request openshift#1348 from openshift-cherrypick-robot/cherry-pick-1285-to-release-4.3

  So the 4.3 fix was between rc.0's 23a6e6fb37 and rc.3's 25bb6aeb58
  (see 'release info' calls in the previous list entry for those
  commit hashes).

* Update CI fails with [7,8]:

    Could not reach HTTP service through <ip>:80 after 2m0s

  and authentication going Degraded=True with RouteHealthDegradedFailedGet:

    RouteHealthDegraded: failed to GET route: dial tcp <ip>:443:
    connect: connection refused

  Fixed in 4.2 with SDN 677b3a8 [9] and in 4.3 with SDN 74a8aee [10].

    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.2.16 | grep ' node '
      node                                          https://github.com/openshift/sdn                                           770cb7bf922a721bc6c62af5490439d6174036fe
    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.2.14 | grep ' node '
      node                                          https://github.com/openshift/sdn                                           770cb7bf922a721bc6c62af5490439d6174036fe
    $ git --no-pager log --first-parent --oneline -4 origin/release-4.2
    098a6410 (origin/release-4.2) Merge pull request openshift#95 from danwinship/fork-k8s-client-go-4.2
    9955a65b Merge pull request openshift#72 from juanluisvaladas/too_many_dns_queries_42
    677b3a80 Merge pull request openshift#90 from openshift-cherrypick-robot/cherry-pick-81-to-release-4.2
    770cb7bf Merge pull request openshift#73 from danwinship/egressip-cleanup-4.2

  So the fix landed after 4.2.16's 770cb7bf.

    $ oc adm release info --commits registry.svc.ci.openshift.org/ocp/release:4.3.0-rc.0 | grep ' sdn '
      sdn                                           https://github.com/openshift/sdn                                           d4e36d5019ef0e130e0d246581508821a7322753
    $ git --no-pager log --first-parent --oneline -5 origin/release-4.3
    490a574e (origin/release-4.3) Merge pull request openshift#98 from openshift-cherrypick-robot/cherry-pick-96-to-release-4.3
    85ab1033 Merge pull request openshift#78 from openshift-cherrypick-robot/cherry-pick-57-to-release-4.3
    d4e36d50 Merge pull request openshift#85 from openshift-cherrypick-robot/cherry-pick-84-to-release-4.3
    dabc4ef5 Merge pull request openshift#83 from dougbtv/backport-build-use-host-local
    74a8aee3 Merge pull request openshift#81 from openshift-cherrypick-robot/cherry-pick-79-to-release-4.3

  So the fix landed before rc.0's d4e36d50.

* GCP update CI fails with [11]:

    Could not reach HTTP service through <ip>:80 after 2m0s

  in 4.2.16 -> 4.3.0-rc.0 [12], 4.2.16 -> 4.3.0-rc.3 [13,14,15], and
  4.2.18 -> 4.3.1 [16].  This doesn't happen every time though; at
  least one 4.2.16 -> 4.3.0-rc.3 has passed on GCP [17].  We don't
  have a root-cause yet, but the final failure matches [8] discussed
  above.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1786993
[2]: openshift/machine-config-operator#1358 (comment)
[3]: openshift/machine-config-operator#1359 (comment)
[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1787635
[5]: openshift/machine-config-operator#1362 (comment)
[6]: openshift/machine-config-operator#1361 (comment)
[7]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/214#1:build-log.txt%3A414
[8]: https://bugzilla.redhat.com/show_bug.cgi?id=1781763
[9]: openshift/sdn#90 (comment)
[10]: openshift/sdn#81 (comment)
[11]: https://bugzilla.redhat.com/show_bug.cgi?id=1785457
[12]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/216
[13]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/232
[14]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/233
[15]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/234
[16]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/286
[17]: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-origin-installer-e2e-gcp-upgrade/230
wking referenced this pull request in wking/cincinnati-graph-data Mar 18, 2020
…1810036

Also tombstone affected releases to avoid further channel promotion
for affected releases.  Details on the bug:

* 4.5: Fixed by [1], service-ca-operator 74b5ce2 [2], which included library-go
  d9c73bb [3].

* 4.4: Introduced by [4] (no PR?).  Fixed by [5], service-ca-operator
  e5a04d6 [6], which included library-go 3c25293 [7].

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ git --no-pager log -2 --first-parent --oneline origin/release-4.4
  e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial
  094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics

  So both RCs are affected.

* 4.3: Introduced by [8], service-ca-operator 8395d65 [9]. Fixed by
  [10], service-ca-operator dd7235b [11], which includes library-go
  5844159 [12].

  Fix has not been released yet.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           774c394da334dec446703545d4baaf89611ccb9d
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           8395d65888b0a4249277989f18ee03f45383e409

  So this was introduced in 4.3.5 (there was no 4.3.4).

* 4.2: Introduced by [13], service-ca-operator 0324055 [14], which
  includes library-go 2cf86bb [15] and API 8ce0047 [16].  Fix in
  flight with [17,18].  [19] has already landed with library-go
  d58edcb.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           f6720573b9b63147436374e51e6fda44683b1e9f
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           0324055c3bad3a857dcf3471c024bf42c20d549e

  So this was introduced in 4.2.22.

* 4.1: Backport stream introducing the bug is still ASSIGNED [20], so
  no 4.1 impact yet.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036
[2]: openshift/service-ca-operator#110 (comment)
[3]: openshift/library-go#726 (comment)
[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121
[5]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418
[6]: openshift/service-ca-operator#111 (comment)
[7]: openshift/library-go#728 (comment)
[8]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179
[9]: openshift/service-ca-operator#104 (comment)
[10]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420
[11]: openshift/service-ca-operator#112 (comment)
[12]: openshift/library-go#729 (comment)
[13]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156
[14]: openshift/service-ca-operator#105 (comment)
[15]: openshift/library-go#684 (comment)
[16]: openshift/api#577 (comment)
[17]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421
[18]: openshift/service-ca-operator#113
[19]: openshift/library-go#730 (comment)
[20]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157
wking referenced this pull request in wking/cincinnati-graph-data Mar 18, 2020
…1810036

Also tombstone affected releases to avoid further channel promotion
for affected releases.  Details on the bug:

* 4.5: Fixed by [1], service-ca-operator 74b5ce2 [2], which included library-go
  d9c73bb [3].

* 4.4: Introduced by [4] (no PR?).  Fixed by [5], service-ca-operator
  e5a04d6 [6], which included library-go 3c25293 [7].

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ git --no-pager log -2 --first-parent --oneline origin/release-4.4
  e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial
  094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics

  So both RCs are affected.

* 4.3: Introduced by [8], service-ca-operator 8395d65 [9]. Fixed by
  [10], service-ca-operator dd7235b [11], which includes library-go
  5844159 [12].

  Fix has not been released yet.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           774c394da334dec446703545d4baaf89611ccb9d
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           8395d65888b0a4249277989f18ee03f45383e409

  So this was introduced in 4.3.5 (there was no 4.3.4).

* 4.2: Introduced by [13], service-ca-operator 0324055 [14], which
  includes library-go 2cf86bb [15] and API 8ce0047 [16].  Fix in
  flight with [17,18].  [19] has already landed with library-go
  d58edcb.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           f6720573b9b63147436374e51e6fda44683b1e9f
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           0324055c3bad3a857dcf3471c024bf42c20d549e

  So this was introduced in 4.2.22.

* 4.1: Backport stream introducing the bug is still ASSIGNED [20], so
  no 4.1 impact yet.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036
[2]: openshift/service-ca-operator#110 (comment)
[3]: openshift/library-go#726 (comment)
[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121
[5]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418
[6]: openshift/service-ca-operator#111 (comment)
[7]: openshift/library-go#728 (comment)
[8]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179
[9]: openshift/service-ca-operator#104 (comment)
[10]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420
[11]: openshift/service-ca-operator#112 (comment)
[12]: openshift/library-go#729 (comment)
[13]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156
[14]: openshift/service-ca-operator#105 (comment)
[15]: openshift/library-go#684 (comment)
[16]: openshift/api#577 (comment)
[17]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421
[18]: openshift/service-ca-operator#113
[19]: openshift/library-go#730 (comment)
[20]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157
wking referenced this pull request in wking/cincinnati-graph-data Mar 18, 2020
…1810036

Also tombstone affected releases to avoid further channel promotion
for affected releases.  Details on the bug:

* 4.5: Fixed by [1], service-ca-operator 74b5ce2 [2], which included library-go
  d9c73bb [3].

* 4.4: Introduced by [4] (no PR?).  Fixed by [5], service-ca-operator
  e5a04d6 [6], which included library-go 3c25293 [7].

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ git --no-pager log -2 --first-parent --oneline origin/release-4.4
  e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial
  094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics

  So both RCs are affected.

* 4.3: Introduced by [8], service-ca-operator 8395d65 [9]. Fixed by
  [10], service-ca-operator dd7235b [11], which includes library-go
  5844159 [12].

  Fix has not been released yet.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           774c394da334dec446703545d4baaf89611ccb9d
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           8395d65888b0a4249277989f18ee03f45383e409

  So this was introduced in 4.3.5 (there was no 4.3.4).

* 4.2: Introduced by [13], service-ca-operator 0324055 [14], which
  includes library-go 2cf86bb [15] and API 8ce0047 [16].  Fix in
  flight with [17,18].  [19] has already landed with library-go
  d58edcb.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           f6720573b9b63147436374e51e6fda44683b1e9f
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           0324055c3bad3a857dcf3471c024bf42c20d549e

  So this was introduced in 4.2.22.

* 4.1: Backport stream introducing the bug is still ASSIGNED [20], so
  no 4.1 impact yet.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036
[2]: openshift/service-ca-operator#110 (comment)
[3]: openshift/library-go#726 (comment)
[4]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121
[5]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418
[6]: openshift/service-ca-operator#111 (comment)
[7]: openshift/library-go#728 (comment)
[8]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179
[9]: openshift/service-ca-operator#104 (comment)
[10]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420
[11]: openshift/service-ca-operator#112 (comment)
[12]: openshift/library-go#729 (comment)
[13]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156
[14]: openshift/service-ca-operator#105 (comment)
[15]: openshift/library-go#684 (comment)
[16]: openshift/api#577 (comment)
[17]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421
[18]: openshift/service-ca-operator#113
[19]: openshift/library-go#730 (comment)
[20]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157
wking referenced this pull request in wking/cincinnati-graph-data Mar 18, 2020
…1810036

The bugs were introduced by the [1] series, and fixed by the
combination of [2,3].  This commit also tombstones affected releases
to avoid further channel promotion.  Details on the bug:

* 4.5: Introduced by [1] (no PR?).  Fixed by [2], service-ca-operator
  74b5ce2 [4], which included library-go d9c73bb [5].

  Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the
  4.4/4.5 split.

* 4.4: Introduced by [1] (no PR?).  Fixed by [7], service-ca-operator
  e5a04d6 [7], which included library-go 3c25293 [9].

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ git --no-pager log -2 --first-parent --oneline origin/release-4.4
  e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial
  094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics

  So both RCs are affected.

  Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the
  4.4/4.5 split.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep oauth-proxy
    oauth-proxy                                    https://github.com/openshift/oauth-proxy                                    3d0621eb72c9dd1c036505363032468a9016f381
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep oauth-proxy
  oauth-proxy                                    https://github.com/openshift/oauth-proxy                                    3d0621eb72c9dd1c036505363032468a9016f381

  So both RCs have OAuth fix, but neither has the service-ca-operator
  fix.

* 4.3: Introduced by [10], service-ca-operator 8395d65 [11]. Fixed by
  [12], service-ca-operator dd7235b [13], which includes library-go
  5844159 [14].

  Fix has not been released yet.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           774c394da334dec446703545d4baaf89611ccb9d
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           8395d65888b0a4249277989f18ee03f45383e409

  So this was introduced in 4.3.5 (there was no 4.3.4).

  Fix also requires the OAuth proxy fix [15,16], which is still in
  flight.

* 4.2: Introduced by [17], service-ca-operator 0324055 [18], which
  includes library-go 2cf86bb [19] and API 8ce0047 [20].  Fix in
  flight with [21,22].  [23] has already landed with library-go
  d58edcb.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           f6720573b9b63147436374e51e6fda44683b1e9f
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           0324055c3bad3a857dcf3471c024bf42c20d549e

  So this was introduced in 4.2.22.

  Fix also requires the OAuth proxy fix [24,25], which is still in
  flight.

* 4.1: Backport stream introducing the bug is still ASSIGNED [26], so
  no 4.1 impact yet.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1801573
[4]: openshift/service-ca-operator#110 (comment)
[5]: openshift/library-go#726 (comment)
[6]: openshift/oauth-proxy#152 (comment)
[7]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418
[8]: openshift/service-ca-operator#111 (comment)
[9]: openshift/library-go#728 (comment)
[10]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179
[11]: openshift/service-ca-operator#104 (comment)
[12]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420
[13]: openshift/service-ca-operator#112 (comment)
[14]: openshift/library-go#729 (comment)
[15]: https://bugzilla.redhat.com/show_bug.cgi?id=1809253
[16]: openshift/oauth-proxy#160
[17]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156
[18]: openshift/service-ca-operator#105 (comment)
[19]: openshift/library-go#684 (comment)
[20]: openshift/api#577 (comment)
[21]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421
[22]: openshift/service-ca-operator#113
[23]: openshift/library-go#730 (comment)
[24]: https://bugzilla.redhat.com/show_bug.cgi?id=1809258
[25]: openshift/oauth-proxy#164
[26]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157
wking referenced this pull request in wking/cincinnati-graph-data Mar 18, 2020
…1810036

The bugs were introduced by the [1] series, and fixed by the
combination of [2,3].  This commit also tombstones affected releases
to avoid further channel promotion.  Details on the bug:

* 4.5: Introduced by [1] (no linked PR, so not sure exactly when it
  was introduced).  Fixed by [2], service-ca-operator 74b5ce2 [4],
  which included library-go d9c73bb [5].

  Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the
  4.4/4.5 split.

* 4.4: Introduced by [1] (no linked PR, so not sure exactly when it
  was introduced).  Fixed by [7], service-ca-operator e5a04d6 [7],
  which included library-go 3c25293 [9].

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ git --no-pager log -2 --first-parent --oneline origin/release-4.4
  e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial
  094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics

  So both RCs are affected.

  Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the
  4.4/4.5 split.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep oauth-proxy
    oauth-proxy                                    https://github.com/openshift/oauth-proxy                                    3d0621eb72c9dd1c036505363032468a9016f381
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep oauth-proxy
  oauth-proxy                                    https://github.com/openshift/oauth-proxy                                    3d0621eb72c9dd1c036505363032468a9016f381

  So both RCs have OAuth fix, but neither has the service-ca-operator
  fix.

* 4.3: Introduced by [10], service-ca-operator 8395d65 [11]. Fixed by
  [12], service-ca-operator dd7235b [13], which includes library-go
  5844159 [14].

  Fix has not been released yet.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           774c394da334dec446703545d4baaf89611ccb9d
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           8395d65888b0a4249277989f18ee03f45383e409

  So this was introduced in 4.3.5 (there was no 4.3.4).

  Fix also requires the OAuth proxy fix [15,16], which is still in
  flight.

* 4.2: Introduced by [17], service-ca-operator 0324055 [18], which
  includes library-go 2cf86bb [19] and API 8ce0047 [20].  Fix in
  flight with [21,22].  [23] has already landed with library-go
  d58edcb.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           f6720573b9b63147436374e51e6fda44683b1e9f
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           0324055c3bad3a857dcf3471c024bf42c20d549e

  So this was introduced in 4.2.22.

  Fix also requires the OAuth proxy fix [24,25], which is still in
  flight.

* 4.1: Backport stream introducing the bug is still ASSIGNED [26], so
  no 4.1 impact yet.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1801573
[4]: openshift/service-ca-operator#110 (comment)
[5]: openshift/library-go#726 (comment)
[6]: openshift/oauth-proxy#152 (comment)
[7]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418
[8]: openshift/service-ca-operator#111 (comment)
[9]: openshift/library-go#728 (comment)
[10]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179
[11]: openshift/service-ca-operator#104 (comment)
[12]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420
[13]: openshift/service-ca-operator#112 (comment)
[14]: openshift/library-go#729 (comment)
[15]: https://bugzilla.redhat.com/show_bug.cgi?id=1809253
[16]: openshift/oauth-proxy#160
[17]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156
[18]: openshift/service-ca-operator#105 (comment)
[19]: openshift/library-go#684 (comment)
[20]: openshift/api#577 (comment)
[21]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421
[22]: openshift/service-ca-operator#113
[23]: openshift/library-go#730 (comment)
[24]: https://bugzilla.redhat.com/show_bug.cgi?id=1809258
[25]: openshift/oauth-proxy#164
[26]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157
wking referenced this pull request in wking/cincinnati-graph-data Mar 18, 2020
…1810036

The bugs were introduced by the [1] series, and fixed by the
combination of [2,3].  This commit also tombstones affected releases
to avoid further channel promotion.  Details on the bug:

* 4.5: Introduced by [1] (no linked PR, so not sure exactly when it
  was introduced).  Fixed by [2], service-ca-operator 74b5ce2 [4],
  which included library-go d9c73bb [5].

  Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the
  4.4/4.5 split.

* 4.4: Introduced by [1] (no linked PR, so not sure exactly when it
  was introduced).  Fixed by [7], service-ca-operator e5a04d6 [7],
  which included library-go 3c25293 [9].

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep service-ca-operator
    service-ca-operator                            https://github.com/openshift/service-ca-operator                            094a9ad02dbe3bcb57d5fbad301cfcfcd48bd2ed
  $ git --no-pager log -2 --first-parent --oneline origin/release-4.4
  e5a04d6a (origin/release-4.4) Merge pull request openshift#111 from marun/4.4-unique-ca-serial
  094a9ad0 Merge pull request #95 from vareti/signer-ca-metrics

  So both RCs are affected.

  Also fixed by [3], oauth-proxy 3d0621e [6], which landed before the
  4.4/4.5 split.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep oauth-proxy
    oauth-proxy                                    https://github.com/openshift/oauth-proxy                                    3d0621eb72c9dd1c036505363032468a9016f381
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.1-x86_64 | grep oauth-proxy
  oauth-proxy                                    https://github.com/openshift/oauth-proxy                                    3d0621eb72c9dd1c036505363032468a9016f381

  So both RCs have OAuth fix, but neither has the service-ca-operator
  fix.

* 4.3: Introduced by [10], service-ca-operator 8395d65 [11]. Fixed by
  [12], service-ca-operator dd7235b [13], which includes library-go
  5844159 [14].

  Fix has not been released yet.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           774c394da334dec446703545d4baaf89611ccb9d
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           8395d65888b0a4249277989f18ee03f45383e409

  So this was introduced in 4.3.5 (there was no 4.3.4).

  Fix also requires the OAuth proxy fix [15,16], which is still in
  flight.

* 4.2: Introduced by [17], service-ca-operator 0324055 [18], which
  includes library-go 2cf86bb [19] and API 8ce0047 [20].  Fix in
  flight with [21,22].  [23] has already landed with library-go
  d58edcb.

  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           f6720573b9b63147436374e51e6fda44683b1e9f
  $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.22-x86_64 | grep service-ca-operator
    service-ca-operator                           https://github.com/openshift/service-ca-operator                           0324055c3bad3a857dcf3471c024bf42c20d549e

  So this was introduced in 4.2.22.

  Fix also requires the OAuth proxy fix [24,25], which is still in
  flight.

* 4.1: Backport stream introducing the bug is still ASSIGNED [26], so
  no 4.1 impact yet.

[1]: https://bugzilla.redhat.com/show_bug.cgi?id=1774121
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=1810036
[3]: https://bugzilla.redhat.com/show_bug.cgi?id=1801573
[4]: openshift/service-ca-operator#110 (comment)
[5]: openshift/library-go#726 (comment)
[6]: openshift/oauth-proxy#152 (comment)
[7]: https://bugzilla.redhat.com/show_bug.cgi?id=1810418
[8]: openshift/service-ca-operator#111 (comment)
[9]: openshift/library-go#728 (comment)
[10]: https://bugzilla.redhat.com/show_bug.cgi?id=1788179
[11]: openshift/service-ca-operator#104 (comment)
[12]: https://bugzilla.redhat.com/show_bug.cgi?id=1810420
[13]: openshift/service-ca-operator#112 (comment)
[14]: openshift/library-go#729 (comment)
[15]: https://bugzilla.redhat.com/show_bug.cgi?id=1809253
[16]: openshift/oauth-proxy#160
[17]: https://bugzilla.redhat.com/show_bug.cgi?id=1774156
[18]: openshift/service-ca-operator#105 (comment)
[19]: openshift/library-go#684 (comment)
[20]: openshift/api#577 (comment)
[21]: https://bugzilla.redhat.com/show_bug.cgi?id=1810421
[22]: openshift/service-ca-operator#113
[23]: openshift/library-go#730 (comment)
[24]: https://bugzilla.redhat.com/show_bug.cgi?id=1809258
[25]: openshift/oauth-proxy#164
[26]: https://bugzilla.redhat.com/show_bug.cgi?id=1774157
@sdodson sdodson deleted the candidate-1-4.1.133-RTOG1IX5JO branch October 12, 2020 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants