[release-4.11] Bug 2104687: drain controller: don't skip the MCC pod drain #3234
Conversation
In 4.11 we moved the drain operation to a centralized controller. This drain controller doesn't drain the MCC today. In theory this means the pod can finish and then gracefully terminate. In practice this is problematic since the pod never gets scheduled off the node, meaning it thinks its still running (according to the API) when the master node shuts down, and won't be reachable until the master node reboots. For some metal setups the reboot could take up to 30 minutes, this means that we won't have a MCC for 30 minutes, which would be very problematic as other pools waiting for drains/updates are stalled as well. We should instead just drain the controller pod. The next one will immediately pick up where this one left off, so there shouldn't be any conflicts. The pod when drained will restart existing operations in ~10 seconds from testing, which would be much faster.
|
@openshift-cherrypick-robot: Bugzilla bug 2103786 has been cloned as Bugzilla bug 2104687. Retitling PR to link against new bug. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Not a blocker for 4.11, but a definite nice-to-have as soon as we can |
openshift-ci
bot
changed the title
openshift-ci
bot
added
bugzilla/severity-high
|
@openshift-cherrypick-robot: This pull request references Bugzilla bug 2104687, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 6 validation(s) were run on this bug
Requesting review from QA contact: DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
@openshift-cherrypick-robot: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
/label cherry-pick-approved |
openshift-ci
bot
added
the
cherry-pick-approved
|
clean backport |
openshift-ci
bot
added
the
backport-risk-assessed
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: openshift-cherrypick-robot, sinnykumari The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
@openshift-cherrypick-robot: All pull requests linked via external trackers have merged: Bugzilla bug 2104687 has been moved to the MODIFIED state. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
6 participants
This is an automated cherry-pick of #3212
/assign yuqi-zhang