server: fix crash when batch > ubatch with embeddings (#12836) #17912
+9
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes ggml-org#12836 where the server crashes with GGML_ASSERT failure when running with embeddings enabled and n_batch > n_ubatch. Root cause: Embeddings use non-causal attention which requires all tokens to be processed within a single ubatch. When n_batch > n_ubatch, the server attempts to split processing, causing assertion failure. Solution: - Add parameter validation in main() after common_params_parse() - When embeddings enabled and n_batch > n_ubatch: * Log warnings explaining the issue * Automatically set n_batch = n_ubatch * Prevent server crash This follows the approach suggested by @ggerganov in issue ggml-org#12836. Note: This supersedes stalled PR ggml-org#12940 which attempted a runtime fix in the old examples/server/server.cpp location. This implementation validates at startup in tools/server/server.cpp (current location). Testing: - Build: Compiles successfully - Validation triggers: Warns when -b > -ub with --embedding - Auto-correction works: Adjusts n_batch = n_ubatch - No false positives: Valid params don't trigger warnings - Verified on macOS M3 Pro with embedding model
yifant-code
force-pushed
the
fix/embedding-batch-validation
branch
from
aae2567 to
2722844
Compare
Contributor
|
This is quite relevant to me. |
aviallon
approved these changes
Dec 11, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #12836
Problem
Server crashes with
GGML_ASSERTfailure when running embeddings with-b > -ub:Embeddings use non-causal attention which requires all tokens in a single ubatch. When
n_batch > n_ubatch, the server attempts to split processing, triggering the assertion.Solution
Add parameter validation in
main()aftercommon_params_parse():--embeddingis enabled andn_batch > n_ubatchn_batch = n_ubatchto prevent crashFollows @ggerganov's suggested approach in #12836.
Testing
./llama-server -m model.gguf --embedding -b 2048 -ub 512Note
Supersedes stalled PR #12940 which attempted a runtime fix in the old
examples/server/location. This implementation validates at startup intools/server/(current location) per maintainer guidance.