Bug 1800319: kubelet: add kube reservation for kubernetes components #1450

rphillips · 2020-02-07T02:33:22Z

- What I did
bumps systemReserved. Fixes an issue where the kubelet and node could become starved causing the node to hang.

upstream doc
BZ1800319

- How to verify it
The BZ has an attached test case.

- Description for the changelog
NONE

/cc @smarterclayton @sjenning @mrunalp

openshift-ci-robot · 2020-02-07T02:33:27Z

@rphillips: This pull request references Bugzilla bug 1800319, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Bug 1800319: kubelet: add kube reservation for kubernetes components

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-07T02:38:00Z

@rphillips: This pull request references Bugzilla bug 1800319, which is valid.

Details

In response to this:

Bug 1800319: kubelet: add kube reservation for kubernetes components

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rphillips · 2020-02-07T13:33:47Z

/retest

rphillips · 2020-02-10T19:31:28Z

ready for review

rphillips · 2020-02-10T19:32:26Z

/cc @smarterclayton @sjenning

rphillips · 2020-02-10T21:04:17Z

/retest

rphillips · 2020-02-10T21:24:54Z

/cc @kikisdeliveryservice @runcom

rphillips · 2020-02-10T22:20:49Z

/retest

kikisdeliveryservice · 2020-02-10T22:39:41Z

seems sane leaving LGTM to @smarterclayton @sjenning @mrunalp

/approve

cfilleke · 2020-02-11T00:59:28Z

Hi, @rphillips -- the symptoms of this bug we're seeing in OpenShift on Z are very different from the symptoms on x86. I had understood from @jaypoulz that a CI system for Z would be available, and since this bug shows up very differently on the two platforms, perhaps it would be prudent to have some CI tests be performed on that. I'm also pursuing access to a 4.x install on the Power architecture to see what the symptoms of memory stress are there. Thanks!

includes: kubelet, container runtime

rphillips · 2020-02-11T16:14:40Z

768m cpu reservations looks like a requirement. The node dies with a 500m reservation.

sjenning · 2020-02-11T16:58:50Z

/lgtm

openshift-ci-robot · 2020-02-11T17:00:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kikisdeliveryservice, rphillips, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [kikisdeliveryservice]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-02-11T17:08:51Z

@rphillips: This pull request references Bugzilla bug 1800319, which is valid.

Details

In response to this:

Bug 1800319: kubelet: add kube reservation for kubernetes components

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-11T17:28:35Z

@rphillips: All pull requests linked via external trackers have merged. Bugzilla bug 1800319 has been moved to the MODIFIED state.

Details

In response to this:

Bug 1800319: kubelet: add kube reservation for kubernetes components

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot · 2020-02-11T19:13:27Z

@rphillips: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-aws-scaleup-rhel7	`b811616`	link	`/test e2e-aws-scaleup-rhel7`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3

Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * It's not clear to me that this is a regression. Does this actually affect update edges at all? Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * It's not clear to me that this is a regression. Does this actually affect update edges at all? Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * This doesn't seem to be a regression, and doesn't seem to be update-specific. UpgradeBlocker was dropped from the 4.3 bug in [3]. So I'm just removing references to this series from blocked-edges. Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1808429#c6

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? But we don't need to bottom that out, because of the next point. * The impact is only a minute or so of unreachable workloads, after which the issue resolves automatically. The UpgradeBlocker was dropped from the 4.3 bug in [3]. So I'm just removing references to this series from blocked-edges. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * This doesn't seem to be a regression, and doesn't seem to be update-specific. UpgradeBlocker was dropped from the 4.3 bug in [4]. So I'm just removing references to this series from blocked-edges. Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1805444#c9 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1808429#c6

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87) and relaxing the blocks from 1161d00 (Added 4.3.2->4.3.3 to channels, 2020-03-06, openshift#100). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? But we don't need to bottom that out, because of the next point. * The impact is only a minute or so of unreachable workloads, after which the issue resolves automatically. The UpgradeBlocker was dropped from the 4.3 bug in [3]. So I'm just removing references to this series from blocked-edges. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * This doesn't seem to be a regression, and doesn't seem to be update-specific. UpgradeBlocker was dropped from the 4.3 bug in [4]. So I'm just removing references to this series from blocked-edges. Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Block 4.2 -> 4.3 for 4.3 before 4.3.5 because of the port bug. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). I think 1161d00's broader .* blocks (which I'm relaxing to 4.2.* blocks) on the earlier 4.3 releases were because the Multus bug affects all updates. But since the Multus bug is no longer a blocker, we can restore 4.3->4.3 edges. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1805444#c9 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1808429#c6

openshift-ci-robot requested review from mrunalp, sjenning and smarterclayton February 7, 2020 02:33

openshift-ci-robot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 7, 2020

openshift-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Feb 8, 2020

rphillips force-pushed the fixes/add_kube_reservation branch from a0f197d to 4e505a3 Compare February 10, 2020 18:51

rphillips changed the title ~~Bug 1800319: kubelet: add kube reservation for kubernetes components~~ WIP: Bug 1800319: kubelet: add kube reservation for kubernetes components Feb 10, 2020

openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 10, 2020

rphillips changed the title ~~WIP: Bug 1800319: kubelet: add kube reservation for kubernetes components~~ Bug 1800319: kubelet: add kube reservation for kubernetes components Feb 10, 2020

openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 10, 2020

openshift-ci-robot requested review from kikisdeliveryservice and runcom February 10, 2020 21:24

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 10, 2020

kubelet: add more kube reservation to protect node

b811616

includes: kubelet, container runtime

rphillips force-pushed the fixes/add_kube_reservation branch from 4e505a3 to b811616 Compare February 11, 2020 15:12

openshift-ci-robot assigned sjenning Feb 11, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 11, 2020

rphillips mentioned this pull request Feb 11, 2020

[release-4.3] Bug 1801824: kubelet: add more system reservation to protect node #1458

Merged

This was referenced Feb 11, 2020

[release-4.2] Bug 1801826: kubelet: add more kube reservation to protect node #1459

Closed

[release-4.1] Bug 1801829: kubelet: add more kube reservation to protect node #1460

Closed

openshift-merge-robot merged commit 01e20aa into openshift:master Feb 11, 2020

smarterclayton mentioned this pull request Feb 13, 2020

DO NOT MERGE: Test with lower reservations for debugging the real failure #1471

Closed

rphillips mentioned this pull request Feb 14, 2020

Revert "kubelet: add kube reservation for kubernetes components" #1475

Closed

wking mentioned this pull request Mar 14, 2020

blocked-edges: Details on bugs for 4.3.2 and 4.3.3 openshift/cincinnati-graph-data#119

Merged

Bug 1800319: kubelet: add kube reservation for kubernetes components #1450

Bug 1800319: kubelet: add kube reservation for kubernetes components #1450

Uh oh!

Conversation

rphillips commented Feb 7, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Feb 7, 2020

Uh oh!

openshift-ci-robot commented Feb 7, 2020

Uh oh!

rphillips commented Feb 7, 2020

Uh oh!

rphillips commented Feb 10, 2020

Uh oh!

rphillips commented Feb 10, 2020

Uh oh!

rphillips commented Feb 10, 2020

Uh oh!

rphillips commented Feb 10, 2020

Uh oh!

rphillips commented Feb 10, 2020

Uh oh!

kikisdeliveryservice commented Feb 10, 2020

Uh oh!

cfilleke commented Feb 11, 2020

Uh oh!

rphillips commented Feb 11, 2020

Uh oh!

sjenning commented Feb 11, 2020

Uh oh!

openshift-ci-robot commented Feb 11, 2020

Uh oh!

openshift-ci-robot commented Feb 11, 2020

Uh oh!

openshift-ci-robot commented Feb 11, 2020

Uh oh!

openshift-ci-robot commented Feb 11, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

rphillips commented Feb 7, 2020 •

edited

Loading