Skip to content

[fix] Release slots with spec decode + disagg #5975

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
schetlur-nv merged 2 commits into from
Jul 14, 2025
Merged

[fix] Release slots with spec decode + disagg #5975

schetlur-nv merged 2 commits into from
Jul 14, 2025

Conversation

@Tabrizian
Copy link
Member

@Tabrizian Tabrizian commented Jul 11, 2025

[fix] Release slots with spec decode + disagg

For disagg with eagle the resources where not freed running into no free slot errors.

@Tabrizian Tabrizian requested a review from a team as a code owner July 11, 2025 21:02
@Tabrizian Tabrizian force-pushed the user/imant/disaggRelease branch from 9dcb5aa to 7966503 Compare July 11, 2025 21:02
@Tabrizian
Copy link
Member Author

Tabrizian commented Jul 11, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 11, 2025

PR_Github #11675 [ run ] triggered by Bot

@Tabrizian Tabrizian force-pushed the user/imant/disaggRelease branch from 7966503 to 750329e Compare July 11, 2025 21:58
@Tabrizian Tabrizian requested a review from pcastonguay July 11, 2025 22:00
@Tabrizian
Copy link
Member Author

Tabrizian commented Jul 12, 2025

/bot run

@Tabrizian Tabrizian force-pushed the user/imant/disaggRelease branch from 750329e to 9734d1a Compare July 12, 2025 00:57
@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 12, 2025

PR_Github #11688 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 12, 2025

PR_Github #11675 [ run ] completed with state ABORTED

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 12, 2025

PR_Github #11688 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #236 completed with status: 'FAILURE'

mikeiovine
mikeiovine approved these changes Jul 12, 2025
View reviewed changes
@Tabrizian Tabrizian force-pushed the user/imant/disaggRelease branch from 9734d1a to 3dee5fb Compare July 14, 2025 04:51
@Tabrizian
Remove drafter
771d29d 
Signed-off-by: Iman Tabrizian <[email protected]>
@Tabrizian
Copy link
Member Author

Tabrizian commented Jul 14, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 14, 2025

PR_Github #11763 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 14, 2025

PR_Github #11763 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #241 completed with status: 'FAILURE'

@Tabrizian
Copy link
Member Author

Tabrizian commented Jul 14, 2025

/bot run

@Tabrizian
Copy link
Member Author

Tabrizian commented Jul 14, 2025

/bot run --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 14, 2025

PR_Github #11826 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 14, 2025

PR_Github #11827 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 14, 2025

PR_Github #11826 [ run ] completed with state ABORTED

@tensorrt-cicd
Copy link
Collaborator

tensorrt-cicd commented Jul 14, 2025

PR_Github #11827 [ run ] completed with state SUCCESS
/LLM/release-0.21/L0_MergeRequest_PR pipeline #242 completed with status: 'SUCCESS'

@schetlur-nv
Copy link
Collaborator

schetlur-nv commented Jul 14, 2025

Based on offline discussion:

  1. There is a time-sensitive customer requirement for disagg+LLAMA4+eagle3.
  2. Without this change, it's not really in working condition (will error out very quickly).
  3. Surface area is contained - only affects disagg+spec dec on 2 model path.

@schetlur-nv schetlur-nv merged commit 2e7da20 into NVIDIA:release/0.21 Jul 14, 2025
3 checks passed
@Tabrizian Tabrizian deleted the user/imant/disaggRelease branch July 14, 2025 23:15
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 14, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
33e2508 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 14, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
77af18b 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 15, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
8fc44aa 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 16, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
4bfaa04 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 16, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
e9f6d1f 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 16, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
92f1c33 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
@Wanli-Jiang Wanli-Jiang mentioned this pull request Jul 17, 2025
Closed
Wanli-Jiang pushed a commit to Wanli-Jiang/TensorRT-LLM that referenced this pull request Jul 17, 2025
@Tabrizian @Wanli-Jiang
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
b76a8ce 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Wanli Jiang <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 17, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
905d0dd 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit to Tabrizian/TensorRT-LLM that referenced this pull request Jul 17, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (NVIDIA#5975)
652c26b 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Tabrizian added a commit that referenced this pull request Jul 17, 2025
@Tabrizian
[fix] Release slots with spec decode + disagg (#5975) (#6032)
d4d21a1 
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
yizhang-nv pushed a commit to yizhang-nv/TensorRT-LLM that referenced this pull request Jul 17, 2025
@Tabrizian @yizhang-nv
[fix] Release slots with spec decode + disagg (NVIDIA#5975) (NVIDIA#6032
f84bf50 
)

Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Signed-off-by: Iman Tabrizian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mikeiovine mikeiovine mikeiovine approved these changes

@pcastonguay pcastonguay pcastonguay approved these changes

@shaharmor98 shaharmor98 Awaiting requested review from shaharmor98 shaharmor98 is a code owner automatically assigned from NVIDIA/trt-llm-torch-runtime-devs

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Tabrizian @tensorrt-cicd @schetlur-nv @mikeiovine @pcastonguay