Add support for DRA devices in kueue workloads #5873

alaypatel07 · 2025-07-03T12:55:08Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR implements the support for DRA devices in Kueue workloads.

Which issue(s) this PR fixes:

Fixes #2941

Special notes for your reviewer:

Work Item Checklist for reviewers:

Nice to haves:

Add integration for config change Add support for DRA devices in kueue workloads #5873 (comment)
support for resource claim templates with DeviceAllocationModeAll Add support for DRA devices in kueue workloads #5873 (comment)
Add event+status condition for dra processing failure for UX
Check if it will be valuable to assert presence of device classes when dra configuration is loaded in kueue: Add support for DRA devices in kueue workloads #5873 (comment)
Move the pre-existing logic in workload_controller from event handlers to Reconcile: https://github.com/kubernetes-sigs/kueue/pull/5873#issuecomment-3299784921, related issue: Reconciliation logic should not happen in predicates #5310
Re-evaluate a need for an extra check in scheduler about existence of ResourceClaimTemplates / ResourceClaims: Add support for DRA devices in kueue workloads #5873 (comment) before entering Beta

Notes:

document the side-effects of enabling->disabling->re-enabling this feature

Does this PR introduce a user-facing change?

DRA: Alpha support for Dynamic Resource Allocation in Kueue.

netlify · 2025-07-03T12:55:15Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`badf257`
🔍 Latest deploy log	https://app.netlify.com/projects/kubernetes-sigs-kueue/deploys/68ca0a5cf3de3e0008716948

k8s-ci-robot · 2025-07-03T12:55:18Z

Hi @alaypatel07. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mimowo · 2025-07-03T12:57:22Z

/ok-to-test

kannon92 · 2025-07-03T14:23:52Z

/ok-to-test

mimowo · 2025-07-09T08:53:30Z

pkg/controller/core/dynamicresourceallocationconfig_controller.go

+}
+
+func (r *DynamicResourceAllocationConfigReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	if !features.Enabled(features.DynamicResourceAllocation) {


No need to check the feature gate inside the reconciler since the controller installation is guarded by the feature gate already.

pkg/controller/core/workload_controller.go

Signed-off-by: Alay Patel <[email protected]>

… function Signed-off-by: Alay Patel <[email protected]>

Signed-off-by: Alay Patel <[email protected]>

…gate check in event handlers Signed-off-by: Alay Patel <[email protected]>

alaypatel07 · 2025-09-17T01:37:52Z

/test pull-kueue-test-e2e-main-1-32
/test pull-kueue-test-e2e-certmanager-main

mimowo · 2025-09-17T08:59:30Z

test/integration/singlecluster/controller/dra/dra_test.go

+			wl.Spec.PodSets[0].Template.Spec.ResourceClaims = []corev1.PodResourceClaim{
+				{Name: "device", ResourceClaimName: ptr.To("test-rc-large")},
+			}
+			wl.Spec.PodSets[0].Template.Spec.Containers[0].Resources.Claims = []corev1.ResourceClaim{
+				{Name: "device"},
+			}


We could replace this PodSets[0]... = with ResourceClaim() wrapper. Also in other places.

mimowo · 2025-09-17T09:05:16Z

test/integration/singlecluster/controller/dra/suite_test.go

+			"DynamicResourceAllocation=true",
+		},
+		APIServerRuntimeConfig: []string{
+			"resource.k8s.io/v1beta2=true",


Folks, can we please start with supporting v1 only? My thinking:

we already have k8s v1.34 with v1 released, so the path for "early adopters" already exists

v1beta2 is disabled by default in the core k8s as of 1.33 so users on most clouds (like GKE where the control-plane manifest is locked) would not benefit from this, just on-prem. The on prem users are often early adopters who can afford using v1.34 (see the k8s feature gate enablement history)

we will need to migrate to v1 eventually, which just delays the process, and may slow down the integration in Kueue if we need some new fields (basically adoption of new DRA features)

I have not yet had an explicit user request to support of DRA-Kueue integration, which suggests even the early adopters are probably ok to wait for the integration a little bit (and if they really really need, then we can suggest upgrading clusters to 1.34, which on clouds would need to be our answer anyway).

cc @alaypatel07 @klueska @tenzen-y @kannon92 @mwysokin

I am fine with supporting 1.34 and on for this feature.

This may mess up our k8s support though.

I think we are saying that DRA + Kueue would only be available on 1.34 and on (so this feature would not be valid for 1.31, 1.32 or 1.33).

I think we are saying that DRA + Kueue would only be available on 1.34 and on (so this feature would not be valid for 1.31, 1.32 or 1.33).

Yes, we can document that. It would not be valid for 1.31 anyway as 1.31 is v1beta1.

For 1.32 and 1.33, yes, but note that while DRA is Beta it remains disabled, which makes it anyway unsable on most clouds where the feature gates are locked: https://github.com/kubernetes/kubernetes/blob/69aca29e6def5873779cbd392bd9a1bad124c586/pkg/features/kube_features.go#L1236-L1240

Ah, as I synced with @alaypatel07 it turns out that some cloud providers have offering which enables DRA in 1.33 despite it is disabled in the OSS: EKS, and GKE, despite the feature being disabled in the core.

@johnbelamaric do you know why it was decided to go with this approach - disabled in core, but enabled by providers?

In any case, for example it remains unclear if supported by Azure.

I think in Kueue we generally have the approach of supporting what is enabled in the core. OTOH, if there are some users waiting for the DRA-Kueue integration in 1.33, then I don't want to block anyone's business, so feel free to speak up.

I'm basically concerned with the future maintanence cost, if we don't have a clear justification for supporting v1beta2. Suppose scenario when the Kueue-DRA integration is in Beta, and we have adoption, but then we need to support a new field which is only in v1, then I'm afraid we will need to choose between breaking existing users in Beta, or moving forward fast and supporting the new feature (field).

So, I would like to make at least the support for "only v1" the graduation criteria for Kueue into Beta.

GKE allows it but the user has to explicitly enable it. EKS I believe may enable it by default in order to support the NVIDIA GB200 on 1.33.

All new endpoints in K8s are disabled at beta by default. Feature gates are enabled by default at beta. That means that beta fields in existing v1 APIs will become available. But v1betaX endpoints are disabled by default, because the encoding in the API object meta data means massive migrations for customers when they write manifests that way.

I think it's OK to do v1 only, unless we have specific use cases like GB200 that people need to do in 1.33.

I can confirm EKS added it to support GB200 on 1.33

Ok, I have synced with @mwysokin and it seems we don't have users for which Kueue-DRA in 1.33 is a blocker.

Actually, we observe for now limited interest as the integration does not support yet TAS (Topology-Aware Scheduling) which is a major investment for Kueue, and recommended solution to consume new big hardware.

So, it seems for the adoption of the DRA-Kueue integration we need to support TAS in the picture which is maybe 0.15, but most likely 0.16+ (depends on the prioritizing of the effort).

Let's assume we support TAS+DRA in 0.16, this is Jan'26, by then k8s 1.34 will be much more widely used and we will have k8s 1.35. So, unlikely v1 would be a blocker for early adopters.

OTOH, I understand the temptation to support v1beta2 "just in case".

I have two open-ended proposals:

if you see some potential adoption in 1.33, or just want to "stay safe" feel free to keep v1beta2, but please add graduation criteria to the KEP update for Beta to "migrate for using v1"

otherwise let's just migrate to v1 in a follow up.

I dont see any potential users for v1beta2 yet, just waiting for @jackfrancis to confirm. With that I would be happy to move to v1 in a follow-up PR

mimowo · 2025-09-17T09:09:33Z

Thank you @alaypatel07, amazing work, and everyone contributing review comments 👍

While I still have some open comments, this comment. I believe it is ready to go as Alpha, we still have time to address this comment before the release.

The PR is getting large, with complex history, so I believe it will be much more productive to merge and address follow ups, rather than resolving conflicts constantly. I double checked the new code is well isolated by the check for the feature gate enablement.

/lgtm
/approve

k8s-ci-robot · 2025-09-17T09:09:41Z

LGTM label has been added.

Git tree hash: cd83bf17096a6e3315877a204fa11cbc914f1d59

k8s-ci-robot · 2025-09-17T09:09:50Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alaypatel07, mimowo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [mimowo]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mimowo · 2025-09-18T15:33:44Z

/release-note-edit

DRA: Alpha support for Dynamic Resource Allocation in Kueue.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 3, 2025

k8s-ci-robot requested a review from kannon92 July 3, 2025 12:55

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 3, 2025

k8s-ci-robot requested a review from mbobrovskyi July 3, 2025 12:55

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jul 3, 2025

alaypatel07 marked this pull request as draft July 3, 2025 12:56

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jul 3, 2025

alaypatel07 force-pushed the dra-implementation branch from b9b6085 to 891d3c8 Compare July 3, 2025 13:43

alaypatel07 force-pushed the dra-implementation branch 2 times, most recently from 6821c91 to e54fb3f Compare July 3, 2025 15:22

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 4, 2025

mimowo reviewed Jul 9, 2025

View reviewed changes

pkg/controller/core/workload_controller.go Outdated Show resolved Hide resolved

mimowo reviewed Jul 9, 2025

View reviewed changes

pkg/controller/core/workload_controller.go Outdated Show resolved Hide resolved

alaypatel07 force-pushed the dra-implementation branch 4 times, most recently from 0bbec13 to 6e6e2cf Compare July 14, 2025 20:39

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 14, 2025

alaypatel07 force-pushed the dra-implementation branch 3 times, most recently from bc817d9 to 1e4d9bd Compare July 14, 2025 23:15

alaypatel07 added 6 commits September 16, 2025 16:59

remove CRD and use configmap for defining the mapping

85c9930

Signed-off-by: Alay Patel <[email protected]>

remove support for resource claims

ca0b72c

Signed-off-by: Alay Patel <[email protected]>

fixups from review

939371c

Signed-off-by: Alay Patel <[email protected]>

remove WorkloadOptions and move processing dra resources to reconcile…

a076a31

… function Signed-off-by: Alay Patel <[email protected]>

don't block workloads when feature gate is disabled

f3aec20

Signed-off-by: Alay Patel <[email protected]>

add more validation tests

d60d7f8

Signed-off-by: Alay Patel <[email protected]>

alaypatel07 force-pushed the dra-implementation branch from bbae069 to bbeac13 Compare September 16, 2025 21:00

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 16, 2025

alaypatel07 force-pushed the dra-implementation branch 3 times, most recently from 29d9c09 to a8732e2 Compare September 17, 2025 00:48

refactor workload controller tests, fix resource counts, fix feature …

badf257

…gate check in event handlers Signed-off-by: Alay Patel <[email protected]>

alaypatel07 force-pushed the dra-implementation branch from a8732e2 to badf257 Compare September 17, 2025 01:09

mimowo reviewed Sep 17, 2025

View reviewed changes

k8s-ci-robot assigned mimowo Sep 17, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 17, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 17, 2025

k8s-ci-robot merged commit 1ffa6bd into kubernetes-sigs:main Sep 17, 2025
23 checks passed

k8s-ci-robot added this to the v0.14 milestone Sep 17, 2025

github-project-automation bot moved this from In Review to Done in Kueue Release Tracking Sep 17, 2025

kannon92 mentioned this pull request Sep 17, 2025

[WIP] DRA e2e test to validate Nvidia gpu openshift/origin#29842

Open

nojnhuh mentioned this pull request Sep 17, 2025

WIP: Add integration test for DRA config change #6904

Closed

mimowo mentioned this pull request Sep 30, 2025

Release v0.14.0 #6756

Closed

36 tasks

mimowo mentioned this pull request Oct 9, 2025

bump DRA version from v1beta2 to v1 #7212

Merged

Add support for DRA devices in kueue workloads #5873

Add support for DRA devices in kueue workloads #5873

Uh oh!

Conversation

alaypatel07 commented Jul 3, 2025 • edited by k8s-ci-robot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Uh oh!

netlify bot commented Jul 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Uh oh!

k8s-ci-robot commented Jul 3, 2025

Uh oh!

mimowo commented Jul 3, 2025

Uh oh!

kannon92 commented Jul 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alaypatel07 commented Sep 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mimowo commented Sep 17, 2025

Uh oh!

k8s-ci-robot commented Sep 17, 2025

Uh oh!

k8s-ci-robot commented Sep 17, 2025

Uh oh!

Uh oh!

mimowo commented Sep 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

alaypatel07 commented Jul 3, 2025 •

edited by k8s-ci-robot

Loading

netlify bot commented Jul 3, 2025 •

edited

Loading

mimowo Sep 17, 2025 •

edited

Loading

mimowo Sep 17, 2025 •

edited

Loading

mimowo Sep 18, 2025 •

edited

Loading