Skip to content
Open
Changes from 1 commit
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2d2e8df
Support disaggregation with KVBM and TRTLLM
jthomson04 Oct 9, 2025
b537618
Wire up connector to trtllm component and support trtllm in disagg de…
jthomson04 Oct 10, 2025
21a5928
Run connector with prefill worker
jthomson04 Oct 11, 2025
1cad35a
Cleanup
jthomson04 Oct 24, 2025
6927c39
fmt
jthomson04 Oct 24, 2025
fa19f67
Merge remote-tracking branch 'origin/main' into jthomson04/kvbm-trtll…
jthomson04 Nov 24, 2025
58f5968
Fix connector path
jthomson04 Nov 24, 2025
b00b282
Merge remote-tracking branch 'origin/main' into jthomson04/kvbm-trtll…
jthomson04 Nov 24, 2025
a701194
Update docs
jthomson04 Nov 24, 2025
5d134df
Check for version compatibility
jthomson04 Nov 24, 2025
667e155
precommit
jthomson04 Nov 24, 2025
cce0e3d
Merge branch 'main' into jthomson04/kvbm-trtllm-disagg
jthomson04 Nov 25, 2025
8a2d280
Merge branch 'main' into jthomson04/kvbm-trtllm-disagg
jthomson04 Nov 30, 2025
1386c07
Merge branch 'main' into jthomson04/kvbm-trtllm-disagg
jthomson04 Dec 1, 2025
41ed735
Merge branch 'main' into jthomson04/kvbm-trtllm-disagg
jthomson04 Dec 8, 2025
d856a14
Merge branch 'main' into jthomson04/kvbm-trtllm-disagg
jthomson04 Dec 12, 2025
bab61a0
Merge remote-tracking branch 'origin/main' into jthomson04/kvbm-trtll…
jthomson04 Dec 12, 2025
616cea9
Merge remote-tracking branch 'origin/main' into jthomson04/kvbm-trtll…
jthomson04 Dec 13, 2025
8fc1724
Little fixes
jthomson04 Dec 13, 2025
474fa6e
fix kv block size
jthomson04 Dec 13, 2025
0c7646b
Better error message
jthomson04 Dec 13, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix kv block size
Signed-off-by: jthomson04 <[email protected]>
  • Loading branch information
jthomson04 committed Dec 13, 2025
commit 474fa6e69bf9deaaf97751a130e5eb36a890b857
8 changes: 5 additions & 3 deletions tests/kvbm_integration/test_determinism_disagg.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,19 +181,21 @@ def _set_up_trtllm_config(self, gpu_cache_blocks):
"/tmp/kvbm_llm_api_decode_config.yaml",
)

KV_BLOCK_SIZE = 16

llm_api_config: Dict[str, Any] = {}
llm_api_config["kv_cache_config"] = {
"enable_partial_reuse": False,
"free_gpu_memory_fraction": 0.10,
"tokens_per_block": 16,
"tokens_per_block": KV_BLOCK_SIZE,
}

# GPU blocks override
if gpu_cache_blocks is not None:
del llm_api_config["kv_cache_config"]["free_gpu_memory_fraction"]
llm_api_config["kv_cache_config"]["max_tokens"] = (
int(gpu_cache_blocks) * 32
) # TRTLLM defaults 32 tokens per block
int(gpu_cache_blocks) * KV_BLOCK_SIZE
)

prefill_config = deepcopy(llm_api_config)
prefill_config["disable_overlap_scheduler"] = True
Expand Down
Loading