Repository navigation
fix(agent): synchronize GPU OOM fallback to protect concurrent in-flight inference (#649) - #810
Conversation
93e855e to
e1c14ae
Compare
…hind Three follow-throughs the integrated branch surfaced, each the missing half of a change that landed on its own: - `.github/workflows/ci.yml`: #795 added `tests/test_cli_lang_guess.py` without registering it, and `tests/test_packaging.py` gates every suite on being wired into CI, so the suite that proves `--lang-guess` works never ran. - `tests/test_cli.py`: #725's drift pin lists the predict controls with no CLI flag. #795 gave `lang_guess` one, so the accepted set is now the five hook arguments. The pin firing here is the pin working. - `tests/test_agent_gate.py`: #810's fake-torch stub omitted `Tensor` and `dtype`, which `laya/common.py` reads at import time for `proper_reward`'s annotations, so the import died and the 11 concurrency checks never executed. They pass. - `tests/test_compose_env.py`: #730 refuses an empty `LAYA_DEFAULT_MODEL` with `if raw is None or not raw.strip()`, a spelling the empty-default recognizer did not match, so a correct read was reported unsafe. The recognizer now takes that form as well as `if not raw`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged for 0.3.23, on A note on what happened in between, since it changes what this PR proves. I first integrated the earlier head, and there the fake-torch stub in Your They run now: 11 passed, 0 failed, covering reader concurrency, writer exclusion, writer preference and exception release. That is the only reason I was willing to take a concurrency change to On attribution: #660 opened this ground and did the measurement round that settled the design on a reader-writer lock, after I asked about it there. Your implementation shipped because it was mergeable and carried runnable tests, and that is a mechanical outcome rather than a judgement between the two. I have said the same on #660. Verified on the integrated branch: full Thank you. |
What
Adds an
_InferenceGatereader-writer synchronization gate toAgentinlaya/agent.pyto protect shared model device placement during scoped GPU out-of-memory (OOM) fallbacks:read_lock(), allowing multiple threads to run concurrent forward passes on GPU without serialization or contention.write_lock(), ensuring all in-flight GPU forward passes finish cleanly beforeself.model.to("cpu")is executed._restore_runtimemoves the model and device back to GPU, preventingRuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!.threading.Eventconcurrency regression tests intests/test_agent_gate.pycovering concurrent readers, writer mutual exclusion, writer preference, and exception safety.Why
Fixes #649. In multi-threaded environments (such as
laya.serveworker pools), a memory-heavy request triggering a GPU OOM fallback previously moved the shared model to CPU while other concurrent requests were mid-inference with CUDA inputs already prepared on GPU, crashing them with a device mismatch.How it was verified
Checklist
mainFixes #649