Skip to content

perf(agent): reuse CUDA autocast weight cache across batches - #700

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
RagingSilence:perf/autocast-cache
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
RagingSilence:perf/autocast-cache

Conversation

@RagingSilence

Copy link
Copy Markdown
Contributor

What

Keep CUDA autocast's weight-cast cache alive across eager predict_batch forward chunks. The enclosing scope is disabled for operations between forwards, respects a caller-owned autocast scope and its cache_enabled setting, and closes before prediction hooks run. If a CUDA OOM triggers the CPU retry inside this new scope, clear the cached copies before moving the model; unrelated OOM paths are unchanged. CPU/MPS, single-forward, fast, and compiled paths do not use the new scope.

Why

predict_long can split one request into many windows and feed them through several predict_batch forwards. Previously, each CUDA forward entered and exited its own autocast scope, discarding cached BF16 casts of the same FP32 weights. The next forward then repeated those casts.

On a local RTX 4060 (8 GiB), using the English checkpoint and a 30-window request with batch_size=4 (8 forwards), 12 alternating before/after pairs measured a median 664.6 ms before → 599.7 ms after (9.8% lower; all 12 pairs faster). Per-forward raw outputs and the final answer were identical. With one forward (batch_size=None), medians were 548.3 ms and 548.8 ms, within run-to-run variation. In a separate 10-pair run, peak allocated CUDA memory for the multi-forward case rose from 2359.8 MiB to 2382.5 MiB (+22.7 MiB); the cached copies remain live until the prediction finishes.

How it was verified

python tests/test_batch.py
python tests/test_runtime_fixes.py
python tests/test_router.py
python tests/test_criteria.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_predict_long.py
python tests/test_question_token_reuse.py
python -m compileall -q laya/ tests/
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120

All passed locally. The batch tests cover scope selection and cleanup before end/error hooks; the runtime tests cover clearing before the CPU move only when this batch scope is active. A CUDA run also observed one cache clear across three forwards and confirmed that a caller's cache_enabled=False was preserved.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • No public API change requiring docs or examples updates

@NandhaKishorM
NandhaKishorM merged commit 280dff8 into NandhaKishorM:main Sep 29, 2026
22 of 23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @RagingSilence! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants