Skip to content

chore: Make llama.cpp a default engine, speedup CI #1177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
grahamking merged 9 commits into from
May 29, 2025
Merged

chore: Make llama.cpp a default engine, speedup CI #1177

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
338f6b3
chore: Make llama.cpp a default engine
grahamking May 22, 2025
f4f8ec4
rustup profile to 'default' to include rustfmt
grahamking May 22, 2025
15b84fa
Single cargo build
grahamking May 23, 2025
b16b24b
Tidy up
grahamking May 28, 2025
95bc0a8
Add a TODO in Dockerfile.vllm
grahamking May 29, 2025
d994e95
Update container/Dockerfile.sglang
grahamking May 29, 2025
2b392d9
Update container/Dockerfile.tensorrt_llm
grahamking May 29, 2025
3684420
Update container/Dockerfile.vllm
grahamking May 29, 2025
039d1fe
Update container/Dockerfile.sglang
grahamking May 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Single cargo build 
Also fix two docker lints (undefined variables).
  • Loading branch information
@grahamking
grahamking committed May 29, 2025
commit 15b84fa666b4f72e76fa921e33f6ace3679716b9
22 changes: 8 additions & 14 deletions container/Dockerfile.sglang
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,13 +294,11 @@ COPY components /workspace/components
COPY launch /workspace/launch
COPY deploy/sdk /workspace/deploy/sdk

# Build Rust crate binaries packaged with the wheel
RUN cargo build --release --locked --features mistralrs,python,dynamo-llm/block-manager \
-p dynamo-run \
-p llmctl \
# Multiple http named crates are present in dependencies, need to specify the path
-p file://$PWD/components/http \
-p metrics
RUN cargo build \
--release \
--locked \
--features mistralrs,python,dynamo-llm/block-manager \
--workspace

# Build dynamo wheel
RUN uv build --wheel --out-dir /workspace/dist && \
Expand All @@ -323,20 +321,16 @@ ENV CARGO_TARGET_DIR=/workspace/target
WORKDIR /workspace

COPY --from=wheel_builder /workspace/dist/ /workspace/dist/
# Copy all the Rust binaries built in the previous stage
COPY --from=wheel_builder /workspace/target/ /workspace/target/
COPY --from=wheel_builder /opt/nvidia/nvda_nixl /opt/nvidia/nvda_nixl
# Copy Cargo cache to avoid re-downloading dependencies
COPY --from=wheel_builder $CARGO_HOME $CARGO_HOME

COPY . /workspace

# Build rest of the crates
# Need to figure out rust caching to avoid rebuilding and remove exclude flags
RUN cargo build --release --locked --features block-manager --workspace \
--exclude dynamo-run \
--exclude llmctl \
--exclude file://$PWD/components/http \
--exclude metrics
# Build C bindings, creates lib/bindings/c/include
RUN cd /workspace/lib/bindings/c && cargo build --release

# Package the bindings
RUN mkdir -p /opt/dynamo/bindings/wheels && \
Expand Down
22 changes: 8 additions & 14 deletions container/Dockerfile.tensorrt_llm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,13 +273,11 @@ COPY components /workspace/components
COPY launch /workspace/launch
COPY deploy/sdk /workspace/deploy/sdk

# Build Rust crate binaries packaged with the wheel
RUN cargo build --release --locked \
-p dynamo-run \
-p llmctl \
# Multiple http named crates are present in dependencies, need to specify the path
-p file://$PWD/components/http \
-p metrics
RUN cargo build \
--release \
--locked \
--features dynamo-llm/block-manager \
--workspace

# Build dynamo wheels
RUN uv build --wheel --out-dir /workspace/dist && \
Expand All @@ -298,19 +296,15 @@ FROM build AS dev

WORKDIR /workspace
COPY --from=wheel_builder /workspace/dist/ /workspace/dist/
# Copy all the Rust binaries built in the previous stage
COPY --from=wheel_builder /workspace/target/ /workspace/target/
# Copy Cargo cache to avoid re-downloading dependencies
COPY --from=wheel_builder $CARGO_HOME $CARGO_HOME

COPY . /workspace

# Build rest of the crates
# Need to figure out rust caching to avoid rebuilding and remove exclude flags
RUN cargo build --release --locked --workspace \
--exclude dynamo-run \
--exclude llmctl \
--exclude file://$PWD/components/http \
--exclude metrics
# Build C bindings, creates lib/bindings/c/include
RUN cd /workspace/lib/bindings/c && cargo build --release

# Package the bindings
RUN mkdir -p /opt/dynamo/bindings/wheels && \
Expand Down
30 changes: 11 additions & 19 deletions container/Dockerfile.vllm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,9 +100,9 @@ RUN cd /usr/local/src && \
ldconfig

ENV LD_LIBRARY_PATH=/usr/lib:/usr/local/ucx/lib:$LD_LIBRARY_PATH
ENV CPATH=/usr/include:$CPATH
ENV CPATH=/usr/include
ENV PATH=/usr/bin:$PATH
ENV PKG_CONFIG_PATH=/usr/lib/pkgconfig:$PKG_CONFIG_PATH
ENV PKG_CONFIG_PATH=/usr/lib/pkgconfig
SHELL ["/bin/bash", "-c"]

WORKDIR /workspace
Expand Down Expand Up @@ -372,13 +372,11 @@ COPY components /workspace/components
COPY launch /workspace/launch
COPY deploy/sdk /workspace/deploy/sdk

# Build Rust crate binaries packaged with the wheel
RUN cargo build --release --locked --features dynamo-llm/block-manager \
-p dynamo-run \
-p llmctl \
# Multiple http named crates are present in dependencies, need to specify the path
-p file://$PWD/components/http \
-p metrics
RUN cargo build \
--release \
--locked \
--features dynamo-llm/block-manager \
--workspace

# Build dynamo wheel
RUN uv build --wheel --out-dir /workspace/dist && \
Expand All @@ -400,21 +398,15 @@ ENV CARGO_TARGET_DIR=/workspace/target

WORKDIR /workspace

COPY --from=wheel_builder /workspace/dist/ /workspace/dist/
COPY --from=wheel_builder /workspace/target/ /workspace/target/
COPY --from=wheel_builder /workspace /workspace
COPY --from=wheel_builder /opt/nvidia/nvda_nixl /opt/nvidia/nvda_nixl
# Copy Cargo cache to avoid re-downloading dependencies
COPY --from=wheel_builder $CARGO_HOME $CARGO_HOME

COPY . /workspace
#COPY . /workspace

# Build rest of the crates
# Need to figure out rust caching to avoid rebuilding and remove exclude flags
RUN cargo build --release --locked --features block-manager --workspace \
--exclude dynamo-run \
--exclude llmctl \
--exclude file://$PWD/components/http \
--exclude metrics
# Build C bindings, creates lib/bindings/c/include
RUN cd /workspace/lib/bindings/c && cargo build --release

# Package the bindings
RUN mkdir -p /opt/dynamo/bindings/wheels && \
Expand Down