docs: add GPT-OSS deployment guide #2297
Conversation
Signed-off-by: jthomson04 <[email protected]>
# Conflicts: # components/backends/trtllm/gpt-oss.md
fix: Improvements to orangina deployment guide See merge request dl/ai-dynamo/dynamo!6
nealvaidya
requested review from
a team,
alec-flowers,
ishandhanani,
nnshah1,
ptarasiewiczNV,
richardhuo-nv,
rmccorm4 and
tanmayv25
as code owners
WalkthroughThis update introduces new documentation, configuration files, and scripts to support disaggregated serving of the Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant Frontend
participant PrefillWorker
participant DecodeWorker
User->>Frontend: Send inference request
Frontend->>PrefillWorker: Route request (prefill phase)
PrefillWorker-->>Frontend: Return prefill results
Frontend->>DecodeWorker: Route request (decode phase, with KV cache via UCX)
DecodeWorker-->>Frontend: Return decode results
Frontend-->>User: Respond with final output
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~15–20 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
♻️ Duplicate comments (2)
components/backends/trtllm/engine_configs/gpt_oss/prefill.yaml (1)
15-27: Same formatting nits asdecode.yamlSee previous comment; apply the same linter fixes here for consistency.
components/backends/trtllm/gpt-oss.md (1)
73-74: Model name availability needs confirmation
openai/gpt-oss-120bis not yet visible on HF Hub (was flagged in prior review). Double-check spelling / availability before publishing docs.
🧹 Nitpick comments (8)
container/build_trtllm_wheel.sh (1)
54-56: Repository already present with different remoteIf
/tmp/TensorRT-LLMexists from a previous run but points to another remote, the wheel will be built from the wrong source. Consider validating that:remote=$(git -C TensorRT-LLM remote get-url origin 2>/dev/null) [ "$remote" != "$TRTLLM_GIT_URL" ] && rm -rf TensorRT-LLMbefore the clone.
components/backends/trtllm/engine_configs/gpt_oss/decode.yaml (1)
15-26: Missing trailing newline & inconsistent indentationMinor, but YAML linters flag both:
- File lacks a trailing newline (POSIX-style text).
- Top-level keys use 0-space indent while
moe_configuses 4; 2-space is the repo convention elsewhere.Not blocking, but worth normalising to keep diffs clean.
README.md (1)
30-33: Date format & link clarity“08/05” is ambiguous (US vs ISO). Prefer
2025-08-05to avoid confusion for international readers.-* [08/05] Deploy `openai/gpt-oss-120b` … +* [2025-08-05] Deploy `openai/gpt-oss-120b` …Also confirm the relative link renders correctly on GitHub; a broken anchor in the homepage hurts discoverability.
components/backends/trtllm/launch/gpt_oss_disagg.sh (2)
18-20: Frontend may start slowly – race condition with workersWorkers may try to register before
dynamo.frontendis listening, causing retries / failures. Consider a simple readiness loop:python3 -m dynamo.frontend --router-mode round-robin --http-port 8000 & FRONTEND_PID=$! until curl -sf http://localhost:8000/healthz >/dev/null; do sleep 1; done
23-46: Hard-coded GPU sets & duplicate flags – expose knobs insteadFixed
CUDA_VISIBLE_DEVICESand batch/token limits make the script less reusable. Export them with sensible defaults so users can override:export PREFILL_GPUS=${PREFILL_GPUS:-"0,1,2,3"} export DECODE_GPUS=${DECODE_GPUS:-"4,5,6,7"} … CUDA_VISIBLE_DEVICES=$PREFILL_GPUS python3 -m dynamo.trtllm …container/Dockerfile.tensorrt_llm_prebuilt (2)
31-38:apt-getlayer leaves cache & increases image sizeAdd
rm -rf /var/lib/apt/lists/*at the end of theapt-getRUN to shrink the final image.
54-58: Building Rust workspace directly in the runtime image bloats final sizeConsider a multi-stage build: compile in one stage, then
COPY --from=builderonly the binaries/libs needed. This keeps the runtime image slim and avoids shipping build toolchain.components/backends/trtllm/gpt-oss.md (1)
228-255: Benchmark command bundles long JSON in--extra-inputstwiceBoth
--extra-inputs ignore_eos:trueand the subsequent JSON-wrapped version set the same flag, which is redundant and may confuse users. Consider removing one of them.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (8)
README.md(1 hunks)components/backends/trtllm/engine_configs/gpt_oss/decode.yaml(1 hunks)components/backends/trtllm/engine_configs/gpt_oss/prefill.yaml(1 hunks)components/backends/trtllm/gpt-oss.md(1 hunks)components/backends/trtllm/launch/gpt_oss_disagg.sh(1 hunks)container/Dockerfile.tensorrt_llm_prebuilt(1 hunks)container/build.sh(4 hunks)container/build_trtllm_wheel.sh(2 hunks)
🧰 Additional context used
🧠 Learnings (5)
📚 Learning: the `--torch-backend=auto` flag works with vllm installations via uv pip install, even though it's n...
Learnt from: ptarasiewiczNV
PR: ai-dynamo/dynamo#2027
File: container/deps/vllm/install_vllm.sh:0-0
Timestamp: 2025-07-22T10:22:28.972Z
Learning: The `--torch-backend=auto` flag works with vLLM installations via uv pip install, even though it's not a standard pip option. This flag is processed by vLLM's build system during installation to automatically match PyTorch distribution with container CUDA versions.
Applied to files:
container/build_trtllm_wheel.shcontainer/build.sh
📚 Learning: in examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd)...
Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
Applied to files:
components/backends/trtllm/launch/gpt_oss_disagg.sh
📚 Learning: in multi-node setups with head/worker architecture, the head node typically doesn't need environment...
Learnt from: GuanLuo
PR: ai-dynamo/dynamo#1371
File: examples/llm/benchmarks/vllm_multinode_setup.sh:18-25
Timestamp: 2025-06-05T01:46:15.509Z
Learning: In multi-node setups with head/worker architecture, the head node typically doesn't need environment variables pointing to its own services (like NATS_SERVER, ETCD_ENDPOINTS) because local processes can access them via localhost. Only worker nodes need these environment variables to connect to the head node's external IP address.
Applied to files:
components/backends/trtllm/launch/gpt_oss_disagg.sh
📚 Learning: in vllm worker deployments, startup probes (with longer periods and higher failure thresholds like p...
Learnt from: nnshah1
PR: ai-dynamo/dynamo#2124
File: components/backends/vllm/deploy/disagg.yaml:54-60
Timestamp: 2025-07-25T22:34:11.384Z
Learning: In vLLM worker deployments, startup probes (with longer periods and higher failure thresholds like periodSeconds: 10, failureThreshold: 60) are used to handle the slow model loading startup phase, while liveness probes are intentionally kept aggressive (periodSeconds: 5, failureThreshold: 1) for quick failure detection once the worker is operational. This pattern separates startup concerns from operational health monitoring in GPU-heavy workloads.
Applied to files:
components/backends/trtllm/gpt-oss.md
📚 Learning: graceful fallback for pytorch wheel installation is broken on arm architecture, so immediate exit on...
Learnt from: zaristei
PR: ai-dynamo/dynamo#2020
File: container/deps/vllm/install_vllm.sh:115-118
Timestamp: 2025-07-21T00:10:56.947Z
Learning: Graceful fallback for PyTorch wheel installation is broken on ARM architecture, so immediate exit on pinned version failure is preferred over fallback mechanisms in container/deps/vllm/install_vllm.sh for ARM64.
Applied to files:
container/build.sh
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: pre-merge-rust (lib/bindings/python)
- GitHub Check: pre-merge-rust (lib/runtime/examples)
- GitHub Check: pre-merge-rust (.)
- GitHub Check: Build and Test - vllm
🔇 Additional comments (5)
container/build_trtllm_wheel.sh (1)
43-47: Good default for custom forks👍 Providing a sane fallback URL avoids hard-coding the NVIDIA fork and simplifies use of private mirrors.
container/build.sh (2)
94-98:TRTLLM_GIT_URLdefault is empty – OKVariable introduced cleanly and initialised early; no side-effects observed.
191-198: Help/option wiring looks correctParser, help text and variable storage are in sync. No issues.
components/backends/trtllm/launch/gpt_oss_disagg.sh (1)
36-46: Missing&keeps decode worker in-foreground – script exits on failure onlyThis is intentional (the script stays alive), but once decode worker ends normally the trap kills all processes. If that is not desired, add an explicit
waitfor the background PIDs instead:wait $FRONTEND_PID $PREFILL_PID $DECODE_PIDcomponents/backends/trtllm/gpt-oss.md (1)
29-33:deploy/docker-compose.ymlpath not present in repoThe guide instructs
docker compose -f deploy/docker-compose.yml up, but no such file is added in this PR. Please verify the path or include the compose definition.
|
Will address @coderabbitai stuff in a later PR |
|
Tip For best results, initiate chat on the files or code changes. @nealvaidya That sounds like a good approach! Keeping PRs focused on their original scope is a solid practice. If you'd like, I can create a follow-up issue to track any suggestions or improvements for later consideration. This would help ensure nothing gets lost and provide a centralized place to discuss the items when you're ready to address them. Let me know if you'd like me to create that issue, or if you prefer to handle the tracking yourself. |
Signed-off-by: jthomson04 <[email protected]> Co-authored-by: jthomson04 <[email protected]> Co-authored-by: John Thomson (DLAlgo) <[email protected]> Co-authored-by: Neelay Shah <[email protected]>
Summary by CodeRabbit
New Features
Documentation
Chores