[refactor] Simplification of Speculative decoding configs - Part 2 #5936
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wili-65535
force-pushed
the
user/wili/spec-configs
branch
from
b773dd3 to
5f205d9
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #11653 [ run ] triggered by Bot |
mikeiovine
approved these changes
Jul 11, 2025
Collaborator
|
PR_Github #11653 [ run ] completed with state |
wili-65535
force-pushed
the
user/wili/spec-configs
branch
from
57ec521 to
fb30ad5
Compare
Contributor
## Walkthrough
This update refactors speculative decoding utility access across several modules, replacing direct attribute and method calls on configuration objects with new standalone utility functions. It also removes related methods and fields from configuration classes, updates public exports, and modifies a test prompt. These changes centralize speculative decoding logic and validation.
## Changes
| File(s) | Change Summary |
|---------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|
| tensorrt_llm/_torch/pyexecutor/_util.py<br>tensorrt_llm/_torch/pyexecutor/model_engine.py<br>tensorrt_llm/_torch/pyexecutor/py_executor_creator.py<br>tensorrt_llm/_torch/pyexecutor/resource_manager.py | Refactored to use new utility functions (`get_num_extra_kv_tokens`, `update_spec_config_from_model_config`) instead of direct config attribute/method access. |
| tensorrt_llm/_torch/speculative/__init__.py | Added new utility functions to public exports (`__all__`). |
| tensorrt_llm/_torch/speculative/model_drafter.py | Replaced method call for draft prompt with new utility function. |
| tensorrt_llm/_torch/speculative/utils.py | Introduced new utility functions for speculative decoding modes and config updates. |
| tensorrt_llm/llmapi/llm_args.py | Removed obsolete fields and methods from decoding config classes; added stricter validation for draft model presence. |
| tests/unittest/_torch/speculative/test_draft_target.py | Switched test prompt from Germany to France by uncommenting and removing lines. |
## Sequence Diagram(s)
```mermaid
sequenceDiagram
participant User
participant PyExecutor
participant SpecConfig
participant Utils
User->>PyExecutor: create_py_executor()
PyExecutor->>Utils: get_num_extra_kv_tokens(spec_config)
Utils-->>PyExecutor: num_extra_kv_tokens
PyExecutor->>SpecConfig: update_spec_config_from_model_config()
SpecConfig-->>PyExecutor: updated spec_config
PyExecutor-->>User: Executor readyEstimated code review effort3 (90–240 minutes) Suggested reviewers
Poem
|
wili-65535
force-pushed
the
user/wili/spec-configs
branch
from
072d36a to
01ef697
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #12440 [ run ] completed with state |
Signed-off-by: wili-65535 <[email protected]>
Signed-off-by: wili-65535 <[email protected]>
Signed-off-by: wili-65535 <[email protected]>
Signed-off-by: wili-65535 <[email protected]>
Signed-off-by: wili-65535 <[email protected]>
wili-65535
force-pushed
the
user/wili/spec-configs
branch
from
d1df81a to
ae34c89
Compare
Collaborator
Author
|
/bot run --disable-fail-fast |
Collaborator
|
PR_Github #12527 [ run ] triggered by Bot |
Collaborator
|
PR_Github #12527 [ run ] completed with state |
syuoni
approved these changes
Jul 23, 2025
NVShreyas
pushed a commit
to NVShreyas/TensorRT-LLM
that referenced
this pull request
Jul 28, 2025
…VIDIA#5936) Signed-off-by: wili-65535 <[email protected]> Co-authored-by: wili-65535 <[email protected]> Signed-off-by: Shreyas Misra <[email protected]>
Ransiki
pushed a commit
to Ransiki/TensorRT-LLM
that referenced
this pull request
Jul 29, 2025
…VIDIA#5936) Signed-off-by: wili-65535 <[email protected]> Co-authored-by: wili-65535 <[email protected]> Signed-off-by: Ransiki Zhang <[email protected]>
This was referenced Aug 18, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Previous PR5639.
Here we replace the class methods with stand-alone tool functions, including:
update_from_model_config,get_draft_model_prompt, andget_num_extra_kv_tokens.Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
Details
run [--disable-fail-fast --skip-test --stage-list "A10-1, xxx" --gpu-type "A30, H100_PCIe" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-[Post-Merge]-1, xxx"]Launch build/test pipelines. All previously running jobs will be killed.
--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests. Will also run L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-[Post-Merge]-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-[Post-Merge]-1, xxx".For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.md.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.
Summary by CodeRabbit
New Features
Refactor
Bug Fixes
Tests