Skip to content
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
still confused about what if get_num_new_matched_tokens is indeed cal…
…led twice what the behavior of update_state_after_alloc should be after the subsequent calls to get_num_new_matched_tokens
  • Loading branch information
ryanolson committed Aug 5, 2025
commit a6f7f356031bff09401aa1d0951076cff052da49
Original file line number Diff line number Diff line change
Expand Up @@ -160,14 +160,11 @@ impl KvConnectorLeader {
let shared_slot = self.slot_manager.get_slot(&request_id).map_err(to_pyerr)?;
let mut slot = shared_slot.lock().map_err(to_pyerr)?;

if slot.has_matched_external_tokens() {
tracing::debug!("detected multiple calls to update_state_after_alloc; skipping lookup");
return Ok(());
}

// the second call will show num_external_tokens == 0
// this call is just letting us know the other blocks that are being used for the remainder of the prefill
if num_external_tokens > 0 {
slot.append_mutable_device_blocks(&block_ids)?;

tracing::debug!(
request_id = request_id,
"triggering onboarding for {} external tokens",
Expand Down
Loading