Graduate "Pop pod from backoffQ when activeQ is empty" to GA

[ania-borowiec]

• One-line PR description: Graduate the feature "Pop pod from backoffQ when activeQ is empty" to GA

• Issue link: Pop pod from backoffQ when activeQ is empty #5142

• Other comments:

[ania-borowiec]

@sanposhiho @macsko I'm a bit lost here, I tried to make a change similar to this one f65b96f but I clearly got something wrong, as my PR fails the tests.
Do you know how to fix this?

[sanposhiho]

nits only from me

[sanposhiho]

/assign @macsko

I'd like to make sure it's aligned with @macsko as well.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ania-borowiec
Once this PR has been reviewed and has the lgtm label, please ask for approval from macsko, wojtek-t. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS
• keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[macsko]

I think we should put it on hold, as we spot some problems related to this feature: kubernetes/kubernetes#134249

/hold

[dom4ha]

I think we should put it on hold, as we spot some problems related to this feature: kubernetes/kubernetes#134249

The feature itself is technically correct, but in its current form it brings a risk of creating a tight loops whenever other parts of the scheduling framework have gaps.

It's a similar problem why we cannot turn off periodic flushing of the unschedulable queue, as it's very hard to exclude a possibility a system would miss some type of a notification waking up a pod.

Even if we can make the in-tree version 100% correct, other plugins may do things wrong, so we need safety nets based on some backoff and periodic flushing.

[macsko]

I think we should put it on hold, as we spot some problems related to this feature: kubernetes/kubernetes#134249

The feature itself is technically correct, but in its current form it brings a risk of creating a tight loops whenever other parts of the scheduling framework have gaps.

It's a similar problem why we cannot turn off periodic flushing of the unschedulable queue, as it's very hard to exclude a possibility a system would miss some type of a notification waking up a pod.

Even if we can make the in-tree version 100% correct, other plugins may do things wrong, so we need safety nets based on some backoff and periodic flushing.

I agree. The feature itself is correct, but it relies on the correctness of the entire kube-scheduler, which is difficult to ensure. We can consider changing the KEP approach to somehow mitigate such unrelated errors, but for now we have more important things to deal with.

[dom4ha]

We can consider changing the KEP approach to somehow mitigate such unrelated errors, but for now we have more important things to deal with.

Exactly, so I'm supporting not only finding a way to mitigate, but also turn it off by default for now until we find that mitigation.

We obviously could turn off async preemption feature instead, but we see a bigger value in having async preemption in the current form as it most likely brings a higher value than the pop from backoff feature.

[macsko]

Exactly, so I'm supporting not only finding a way to mitigate, but also turn it off by default for now until we find that mitigation.

We obviously could turn off async preemption feature instead, but we see a bigger value in having async preemption in the current form as it most likely brings a higher value than the pop from backoff feature.

Agree, that was my proposal as well: kubernetes/kubernetes#134249 (comment)

[ania-borowiec]

@macsko @dom4ha
Does anything change if we merge kubernetes/kubernetes#134294 ?

If not, do you think we're able to define any conditions under which we want to have this feature enabled by default or graduated to GA?