Skip to content

Latest commit

 

History

History
182 lines (123 loc) · 8.13 KB

File metadata and controls

182 lines (123 loc) · 8.13 KB

LLM Deployment using SGLang

This directory contains an SGLang component for Dynamo and reference implementations for deploying Large Language Models (LLMs) in various configurations using SGLang. SGLang internally uses ZMQ to communicate between the ingress and the engine processes. For Dynamo, we leverage the runtime to communicate directly with the engine processes and handle ingress and pre/post processing on our end.

Use the Latest Release

We recommend using the latest stable release of dynamo to avoid breaking changes:

GitHub Release

You can find the latest release here and check out the corresponding branch with:

git checkout $(git describe --tags $(git rev-list --tags --max-count=1))

Table of Contents

Feature Support Matrix

Core Dynamo Features

Feature SGLang Notes
Disaggregated Serving
Conditional Disaggregation 🚧 WIP PR
KV-Aware Routing
SLA-Based Planner Planned
Load Based Planner Planned
KVBM Planned

Large Scale P/D and WideEP Features

Feature SGLang Notes
WideEP ✅/🚧 Full support on H100s/GB200 WIP PR
DP Rank Routing 🚧 Direct routing supported. Process per DP rank is not supported
GB200 Support 🚧 WIP PR

Quick Start

Below we provide a guide that lets you run all of our the common deployment patterns on a single node. See our different architectures for a high level overview of each pattern and the architecture diagram for each.

Start NATS and ETCD in the background

Start using Docker Compose

docker compose -f deploy/docker-compose.yml up -d

Build container

# pull our pre-build sglang runtime container
docker pull nvcr.io/nvidia/ai-dynamo/sglang-runtime:0.3.2
# or build from source
./container/build.sh --framework sglang

Run container

./container/run.sh -it --framework sglang

Run Single Node Examples

Important

Each example corresponds to a simple bash script that runs the OpenAI compatible server, processor, and optional router (written in Rust) and LLM engine (written in Python) in a single terminal. You can easily take each command and run them in separate terminals.

Additionally - because we use sglang's argument parser, you can pass in any argument that sglang supports to the worker!

Aggregated Serving

cd $DYNAMO_ROOT/components/backends/sglang
./launch/agg.sh

Aggregated Serving with KV Routing

Note

The current implementation of components/backends/sglang/src/dynamo/sglang/worker/main.py publishes placeholder engine metrics to keep the Dynamo KV-router happy. Real-time metrics will be surfaced directly from the SGLang engine once the following pull requests are merged: • Dynamo: ai-dynamo/dynamo #1465feat: receive kvmetrics from sglang scheduler.

After these are in, the TODOs in main.py will be resolved and the placeholder logic removed.

cd $DYNAMO_ROOT/components/backends/sglang
./launch/agg_router.sh

Disaggregated serving

Under the hood: SGLang Load Balancer vs Dynamo Discovery

SGLang uses a mini load balancer to route requests to handle disaggregated serving. The load balancer functions as follows:

  1. The load balancer receives a request from the client
  2. A random (prefill, decode) pair is selected from the pool of available workers
  3. Request is sent to both prefill and decode workers via asyncio tasks
  4. Internally disaggregation is done from prefill -> decode

Because Dynamo has a discovery mechanism, we do not use a load balancer. Instead, we first route to a random prefill worker, select a random decode worker, and then send the request to both. Internally, SGLang's bootstrap server (which is a part of the tokenizer_manager) is used in conjuction with NIXL to handle the kv transfer.

Important

Disaggregated serving in SGLang currently requires each worker to have the same tensor parallel size unless you are using an MLA based model

cd $DYNAMO_ROOT/components/backends/sglang
./launch/disagg.sh

Disaggregated Serving with Mixture-of-Experts (MoE) models and DP attention

You can use this configuration to test out disaggregated serving with dp attention and expert parallelism on a single node before scaling to the full DeepSeek-R1 model across multiple nodes.

# note this will require 4 GPUs
cd $DYNAMO_ROOT/components/backends/sglang
./launch/disagg_dp_attn.sh

When using MoE models, you can also use the our implementation of the native SGLang endpoints to record expert distribution data. The disagg_dp_attn.sh script automatically sets up the SGLang HTTP server, the environment variable that controls the expert distribution recording directory, and sets up the expert distribution recording mode to stat. You can learn more about expert parallelism load balancing here.

Request Migration

In a Distributed System, a request may fail due to connectivity issues between the Frontend and the Backend.

The Frontend will automatically track which Backends are having connectivity issues with it and avoid routing new requests to the Backends with known connectivity issues.

For ongoing requests, there is a --migration-limit flag which can be set on the Backend that tells the Frontend how many times a request can be migrated to another Backend should there be a loss of connectivity to the current Backend.

For example,

python3 -m dynamo.sglang ... --migration-limit=3

indicates a request to this model may be migrated up to 3 times to another Backend, before failing the request, should the Frontend detects a connectivity issue to the current Backend.

The migrated request will continue responding to the original request, allowing for a seamless transition between Backends, and a reduced overall request failure rate at the Frontend for enhanced user experience.

Advanced Examples

Below we provide a selected list of advanced examples. Please open up an issue if you'd like to see a specific example!

Run on multi-node

Large scale P/D disaggregation with WideEP

Supporting SGLang's native endpoints via Dynamo

Deployment

We currently provide deployment examples for Kubernetes and SLURM.

Kubernetes

SLURM