Skip to content

docs: Clean up incomplete recipes and clarify Kubernetes-only focus #4159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BenHamm merged 12 commits into from
Nov 18, 2025
Merged

docs: Clean up incomplete recipes and clarify Kubernetes-only focus #4159

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
1b34e17
recipes: Clean up incomplete recipes and clarify Kubernetes-only focus
BenHamm Nov 6, 2025
a024413
Remove emojis from README section headings
BenHamm Nov 6, 2025
037c3bd
docs: add hyperlinks to recipe table for easier navigation
BenHamm Nov 6, 2025
85153cd
docs: fix trailing whitespace in README
BenHamm Nov 6, 2025
a6dccf3
restore trtllm configs in examples/trtllm
biswapanda Nov 10, 2025
a6be3b9
Fix the TRTLLM engine config locations and references
tanmayv25 Nov 12, 2025
f1490c4
Fix the multinode documentation
tanmayv25 Nov 12, 2025
5e2646d
Fix the reference directory
tanmayv25 Nov 12, 2025
00c13f3
Apply suggestions from code review
tanmayv25 Nov 13, 2025
0b230f7
Add placeholder for the version
tanmayv25 Nov 13, 2025
85009c5
Update README.md with GAIE
atchernych Nov 17, 2025
f7881a0
Add a new column
tanmayv25 Nov 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: add hyperlinks to recipe table for easier navigation
  • Loading branch information
@BenHamm @tanmayv25
BenHamm authored and tanmayv25 committed Nov 18, 2025
commit 037c3bd884d4c3cd7ccf87fd9c8ba62deaf5982b
20 changes: 10 additions & 10 deletions recipes/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@ Production-tested Kubernetes deployment recipes for LLM inference using NVIDIA D

| Model | Framework | Mode | GPUs | Deployment | Benchmark Recipe | Notes |
|-------|-----------|------|------|------------|------------------|-------|
| **Llama-3-70B** | vLLM | Aggregated | 4x H100/H200 | ✅ | ✅ | FP8 dynamic quantization |
| **Llama-3-70B** | vLLM | Disagg (Single-Node) | 8x H100/H200 | ✅ | ✅ | Prefill + Decode separation |
| **Llama-3-70B** | vLLM | Disagg (Multi-Node) | 16x H100/H200 | ✅ | ✅ | 2 nodes, 8 GPUs each |
| **Qwen3-32B-FP8** | TensorRT-LLM | Aggregated | 4x GPU | ✅ | ✅ | FP8 quantization |
| **Qwen3-32B-FP8** | TensorRT-LLM | Disaggregated | 8x GPU | ✅ | ✅ | Prefill + Decode separation |
| **GPT-OSS-120B** | TensorRT-LLM | Aggregated | 4x GB200 | ✅ | ✅ | Blackwell only, WideEP |
| **GPT-OSS-120B** | TensorRT-LLM | Disaggregated | TBD | ❌ | ❌ | Engine configs only, no K8s manifest |
| **DeepSeek-R1** | SGLang | Disagg WideEP | 8x H200 | ✅ | ❌ | Benchmark recipe pending |
| **DeepSeek-R1** | SGLang | Disagg WideEP | 16x H200 | ✅ | ❌ | Benchmark recipe pending |
| **DeepSeek-R1** | TensorRT-LLM | Disagg WideEP (GB200) | 32+4 GB200 | ✅ | ✅ | Multi-node: 8 decode + 1 prefill nodes |
| **[Llama-3-70B](llama-3-70b/vllm/agg/)** | vLLM | Aggregated | 4x H100/H200 | ✅ | ✅ | FP8 dynamic quantization |
| **[Llama-3-70B](llama-3-70b/vllm/disagg-single-node/)** | vLLM | Disagg (Single-Node) | 8x H100/H200 | ✅ | ✅ | Prefill + Decode separation |
| **[Llama-3-70B](llama-3-70b/vllm/disagg-multi-node/)** | vLLM | Disagg (Multi-Node) | 16x H100/H200 | ✅ | ✅ | 2 nodes, 8 GPUs each |
| **[Qwen3-32B-FP8](qwen3-32b-fp8/trtllm/agg/)** | TensorRT-LLM | Aggregated | 4x GPU | ✅ | ✅ | FP8 quantization |
| **[Qwen3-32B-FP8](qwen3-32b-fp8/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | 8x GPU | ✅ | ✅ | Prefill + Decode separation |
| **[GPT-OSS-120B](gpt-oss-120b/trtllm/agg/)** | TensorRT-LLM | Aggregated | 4x GB200 | ✅ | ✅ | Blackwell only, WideEP |
| **[GPT-OSS-120B](gpt-oss-120b/trtllm/disagg/)** | TensorRT-LLM | Disaggregated | TBD | ❌ | ❌ | Engine configs only, no K8s manifest |
| **[DeepSeek-R1](deepseek-r1/sglang/disagg-8gpu/)** | SGLang | Disagg WideEP | 8x H200 | ✅ | ❌ | Benchmark recipe pending |
| **[DeepSeek-R1](deepseek-r1/sglang/disagg-16gpu/)** | SGLang | Disagg WideEP | 16x H200 | ✅ | ❌ | Benchmark recipe pending |
| **[DeepSeek-R1](deepseek-r1/trtllm/disagg/wide_ep/gb200/)** | TensorRT-LLM | Disagg WideEP (GB200) | 32+4 GB200 | ✅ | ✅ | Multi-node: 8 decode + 1 prefill nodes |

**Legend:**
- **Deployment**: ✅ = Complete `deploy.yaml` manifest available | ❌ = Missing or incomplete
Expand Down