Skip to content

fix(agent): synchronize GPU OOM fallback to protect concurrent in-flight inference (#649) - #810

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Swatantra-66:fix/concurrent-gpu-oom-fallback
Oct 1, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Swatantra-66:fix/concurrent-gpu-oom-fallback

Conversation

@Swatantra-66

Copy link
Copy Markdown
Contributor

What

Adds an _InferenceGate reader-writer synchronization gate to Agent in laya/agent.py to protect shared model device placement during scoped GPU out-of-memory (OOM) fallbacks:

  • Normal inference acquires read_lock(), allowing multiple threads to run concurrent forward passes on GPU without serialization or contention.
  • When an OOM occurs, the fallback path acquires write_lock(), ensuring all in-flight GPU forward passes finish cleanly before self.model.to("cpu") is executed.
  • Newly arriving requests wait until the CPU retry completes and _restore_runtime moves the model and device back to GPU, preventing RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!.
  • Includes deterministic threading.Event concurrency regression tests in tests/test_agent_gate.py covering concurrent readers, writer mutual exclusion, writer preference, and exception safety.

Why

Fixes #649. In multi-threaded environments (such as laya.serve worker pools), a memory-heavy request triggering a GPU OOM fallback previously moved the shared model to CPU while other concurrent requests were mid-inference with CUDA inputs already prepared on GPU, crashing them with a device mismatch.

How it was verified

python tests/test_agent_gate.py
# Output: 11 passed, 0 failed - all agent gate tests passed

python tests/test_packaging.py
# Output: 110 passed, 0 failed - all packaging tests passed

ruff check laya/ tests/test_agent_gate.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
# Output: All checks passed!

python -m compileall -q laya/ tests/
# Output: Clean (exit code 0)

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes #649

@Swatantra-66
Swatantra-66 force-pushed the fix/concurrent-gpu-oom-fallback branch from 93e855e to e1c14ae Compare October 1, 2026 16:36
NandhaKishorM added a commit that referenced this pull request Oct 1, 2026
…hind

Three follow-throughs the integrated branch surfaced, each the missing half of a
change that landed on its own:

- `.github/workflows/ci.yml`: #795 added `tests/test_cli_lang_guess.py` without
  registering it, and `tests/test_packaging.py` gates every suite on being wired
  into CI, so the suite that proves `--lang-guess` works never ran.
- `tests/test_cli.py`: #725's drift pin lists the predict controls with no CLI
  flag. #795 gave `lang_guess` one, so the accepted set is now the five hook
  arguments. The pin firing here is the pin working.
- `tests/test_agent_gate.py`: #810's fake-torch stub omitted `Tensor` and
  `dtype`, which `laya/common.py` reads at import time for `proper_reward`'s
  annotations, so the import died and the 11 concurrency checks never executed.
  They pass.
- `tests/test_compose_env.py`: #730 refuses an empty `LAYA_DEFAULT_MODEL` with
  `if raw is None or not raw.strip()`, a spelling the empty-default recognizer
  did not match, so a correct read was reported unsafe. The recognizer now takes
  that form as well as `if not raw`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM
NandhaKishorM merged commit e54b340 into NandhaKishorM:main Oct 1, 2026
24 of 25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.23, on e1c14ae.

A note on what happened in between, since it changes what this PR proves. I first integrated the earlier head, and there the fake-torch stub in tests/test_agent_gate.py defined neither torch.Tensor nor torch.dtype. laya/common.py reads both at import time for proper_reward's annotations, so from laya.agent import _InferenceGate raised AttributeError: module 'torch' has no attribute 'Tensor' and the file exited during import. The eleven concurrency checks had never run, on that head or in CI.

Your e1c14ae fixes that independently and in a better shape than my patch did: try: import torch / except ImportError: means the suite runs against real torch wherever torch is installed, which is every CI lane, rather than shadowing it with a stub. I took your version. I added t.dtype alongside your t.Tensor on the fallback path, because without it the torch-less branch still dies on the same annotation, one line further down.

They run now: 11 passed, 0 failed, covering reader concurrency, writer exclusion, writer preference and exception release. That is the only reason I was willing to take a concurrency change to laya/agent.py in a batch of this size, so it is worth saying plainly that the tests, not the description, are what merged this.

On attribution: #660 opened this ground and did the measurement round that settled the design on a reader-writer lock, after I asked about it there. Your implementation shipped because it was mergeable and carried runnable tests, and that is a mechanical outcome rather than a judgement between the two. I have said the same on #660.

Verified on the integrated branch: full ci.yml test lane plus the pytest lane green, and a decision diff against main on the real english and multilingual checkpoints showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: Concurrent GPU OOM fallback can break in-flight requests with a device mismatch

2 participants