Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 20 additions & 5 deletions docs/guides/dynamo_run.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ This guide explains the`dynamo run` command.

`dynamo-run` is a CLI tool for exploring the Dynamo components. It's also an example of how to use components from Rust. If you use the Python wheel, it's available as `dynamo run` .

It supports these engines: mistralrs, llamacpp, sglang, vllm, and tensorrt-llm. `mistralrs` is the default.
It supports these engines: mistralrs, llamacpp, sglang, vllm, and tensorrt-llm.

Usage:
```
Expand All @@ -35,12 +35,16 @@ Example: `dynamo run Qwen/Qwen3-0.6B`

Set the environment variable `DYN_LOG` to adjust the logging level; for example, `export DYN_LOG=debug`. It has the same syntax as `RUST_LOG`.

`dynamo-run` is build for CUDA by default. For other builds (CPU, Metal, etc) see the [Setup section](#setup).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`dynamo-run` is build for CUDA by default. For other builds (CPU, Metal, etc) see the [Setup section](#setup).
`dynamo-run` is built with CUDA support by default. For other builds (CPU, Metal, etc) see the [Setup section](#setup).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: the build will support CPU as well as CUDA - right? maybe dynamo is built with CPU and GPU support by default ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently we have to decide that at build time. We could (probably should) release two versions, one for CPU and CUDA.

It would be valuable to investigate doing a multi-build that can support either, and detect at runtime. Need engineering time there though, so many other priorities.


## Quickstart with pip and vllm

If you used `pip` to install `dynamo`, you have the `dynamo-run` binary pre-installed with the `vllm` engine. You must be in a virtual environment with vllm installed to use this engine. To compile from source, see [Full usage details](#full-usage-details) below.

The vllm and sglang engines require [etcd](https://etcd.io/) and [nats](https://nats.io/) with jetstream (`nats-server -js`). Mistralrs and llamacpp do not.

The default engine for safetensors is `mistral.rs`. The default model for GGUF is `llama.cpp`.

### Use model from Hugging Face

To automatically downloads Qwen3 4B from Hugging Face (16 GiB download) and starts it in interactive text mode:
Expand Down Expand Up @@ -252,19 +256,30 @@ source $HOME/.cargo/env

##### Step 3: Build

- Linux with GPU and CUDA (tested on Ubuntu):
- Linux with GPU and CUDA (tested on Ubuntu). This is the default.
```
cargo build --features cuda
cargo build
```

which is equivalent to

```
cargo build --features cuda,mistralrs,llamacpp
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome!

```

- macOS with Metal:
```
cargo build --features metal
cargo build --no-default-features --features metal,mistralrs,llamacpp
```

- Vulkan (llama.cpp engine only), untested
```
cargo build --no-default-features --features vulkan,mistralrs,llamacpp
```

- CPU only:
```
cargo build
cargo build --no-default-features --features mistralrs,llamacpp
```

Optionally you can run `cargo build` from any location with arguments:
Expand Down
2 changes: 1 addition & 1 deletion launch/dynamo-run/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ description = "Dynamo Run CLI"

[features]
# Build with `--no-default-features` to disable these defaults
default = ["mistralrs", "llamacpp"]
default = ["mistralrs", "llamacpp", "cuda"]
mistralrs = ["dep:dynamo-engine-mistralrs"]
llamacpp = ["dep:dynamo-engine-llamacpp"]

Expand Down
Loading