docs: add aggregated example turning on MTP with DeepSeek R1 #1421
Conversation
richardhuo-nv
requested review from
GuanLuo,
rmccorm4 and
tanmayv25
as code owners
WalkthroughA new example for aggregated serving using Multi-Token Prediction (MTP) with the DeepSeek R1 model has been added. This includes documentation updates and two new YAML configuration files specifying the serving architecture and runtime parameters for deploying the DeepSeek-R1-FP4 model with TensorRT LLM and MTP decoding. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Frontend
participant TensorRTLLMWorker
User->>Frontend: Send inference request
Frontend->>TensorRTLLMWorker: Route request (round robin)
TensorRTLLMWorker->>TensorRTLLMWorker: Run DeepSeek-R1-FP4 with MTP decoding
TensorRTLLMWorker->>Frontend: Return prediction
Frontend->>User: Return response
Poem
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
richardhuo-nv
force-pushed
the
rihuo/add_agg_mtp_example
branch
from
7ceafe7 to
7f41cdb
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
examples/tensorrt_llm/README.md (2)
121-125: Ensure consistency in formatting and path references
- Add a space before the parentheses in the heading:
Multi-Token Prediction (MTP)- Prefix the config path with
./to match other examples-#### Aggregated serving with Multi-Token Prediction(MTP) and DeepSeek R1 +#### Aggregated serving with Multi-Token Prediction (MTP) and DeepSeek R1 -```bash -dynamo serve graphs.agg:Frontend -f configs/deepseek_r1/mtp/mtp_agg.yaml -``` +```bash +dynamo serve graphs.agg:Frontend -f ./configs/deepseek_r1/mtp/mtp_agg.yaml +```
126-131: Enhance notes with performance insights and formatting consistency
- Add a bullet on observed stability and throughput gains:
- Aggregated MTP serving remains stable up to 256 concurrency and yields ~10% TPS improvement.- Wrap the
cuda_graph_padding_enabled: falsesetting in backticks for clarity.examples/tensorrt_llm/configs/deepseek_r1/mtp/mtp_agg.yaml (1)
22-30: Use consistent path notation forengine_args
Other examples use./configs/...relative paths. Consider updating to:-TensorRTLLMWorker: - engine_args: "configs/deepseek_r1/mtp/mtp_agg_llm_api_config.yaml" +TensorRTLLMWorker: + engine_args: "./configs/deepseek_r1/mtp/mtp_agg_llm_api_config.yaml"
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
examples/tensorrt_llm/README.md(1 hunks)examples/tensorrt_llm/configs/deepseek_r1/mtp/mtp_agg.yaml(1 hunks)examples/tensorrt_llm/configs/deepseek_r1/mtp/mtp_agg_llm_api_config.yaml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (1)
- GitHub Check: Build and Test - vllm
🔇 Additional comments (5)
examples/tensorrt_llm/configs/deepseek_r1/mtp/mtp_agg.yaml (2)
1-15: License header is in order
The Apache-2.0 license block is correctly applied.
16-21: Frontend configuration looks good
The served model name, endpoint, port, and router settings align with other examples.examples/tensorrt_llm/configs/deepseek_r1/mtp/mtp_agg_llm_api_config.yaml (3)
1-15: License header is correct
The Apache-2.0 block and SPDX tags are properly included.
16-32: Engine hyperparameters are appropriately set
Model, parallelism, batch size, and KV cache configurations align with FP4 and MTP requirements.
33-53: MTP decoding and CUDA graph settings are properly configured
Speculative decoding is enabled, and the workaround for the knowncuda_graph_padding_enabledbug is documented.
Overview:
Add an example about how to turning on the MTP with DeepSeek R1 in Aggregated serving.
Details:
Add an example about how to turning on the MTP with DeepSeek R1 in Aggregated serving. The serving won't crash even when concurrency reach 256. There appears to be around a 10% TPS gain, but I learned the MTP need to be benchmarked with special datasets. That's our next step.
We are still working with Tensorrt LLM team to get a stable main build that can serve the disaggregated cases.
Summary by CodeRabbit
New Features
Documentation