Repository navigation
fix(agent): prevent concurrent OOM fallback device races - #660
AliAkbariAlashti wants to merge 3 commits into
Conversation
|
The dependency CVE check is failing because of the current optional dependency set. This PR does not modify dependencies, and the issue appears to be tracked separately in #645/#646. |
|
Unsubscribe
…On Mon, 28 Sept 2026, 10:23 am Ali, ***@***.***> wrote:
Fixes #649 <#649> by
serializing inference per Agent instance while preserving concurrency
across separate Agents. A deterministic two-thread regression test verifies
that a GPU OOM fallback cannot move the shared model under another
in-flight request.
------------------------------
You can view, comment on, or merge this pull request online at:
#660
Commit Summary
- 38db478
<38db478>
fix(agent): serialize inference during OOM fallback
File Changes
(2 files <https://github.com/NandhaKishorM/laya/pull/660/files>)
- *M* laya/agent.py
<https://github.com/NandhaKishorM/laya/pull/660/files#diff-4157c4dbcf565c799ba2eee88c6cd76776776520750731d05286fda100a2e497>
(93)
- *M* tests/test_runtime_fixes.py
<https://github.com/NandhaKishorM/laya/pull/660/files#diff-50589edebc6211d0c09ec27d5cdaa7ee93fb39fdc996cba622de510292b0cde2>
(72)
Patch Links:
- https://github.com/NandhaKishorM/laya/pull/660.patch
- https://github.com/NandhaKishorM/laya/pull/660.diff
—
Reply to this email directly, view it on GitHub
<#660?email_source=notifications&email_token=CC5M5MA72GY2FXNNB3S7PH35RHVFZA5CNFSNUABEM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UF42DMNJZGQYTMMJVHGTHEZLBONXW5KTTOVRHGY3SNFRGKZFFMV3GK3TUVRTG633UMVZF6Y3MNFRWW>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/CC5M5MFFCJFCUGXMU3TYXIL5RHVFZAVCNFSNUABGKJSXA33TNF2G64TZHMYTGNZVGM4DEMRVGM5US43TOVSTWNJWGA4TKMRVGU3DHILWAI>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Bruce-Yii
left a comment
There was a problem hiding this comment.
The race is real, your fix closes it, and the two-thread test is genuinely load-bearing — I ran it against main's agent.py and it goes red every time (25 runs out of 25, no spurious passes), and green on this branch. It also needs no GPU, so it will actually run in CI. The Lock → RLock change on the fallback is load-bearing too: an instance without _infer_lock falls back to the module global, which the OOM branch then re-acquires on the same thread, and I confirmed that reverting just that hunk self-deadlocks. Good catch on that one.
What I want settled before this lands is the throughput contract, because the change is broader than the OOM path and nothing in the PR says so.
with infer_lock: wraps the whole try/except, so the lock is held across the forward on every request, not only when one OOMs. I measured it with a stand-in forward, four threads times three calls: main runs them at 3.98x effective parallelism, this branch at 1.00x. That is exactly the cost issue #649 lists as Option 1's con, and #649 says a separate CPU runtime is preferable — so I don't think the approach is wrong. But it reverses three statements that are in the tree today and that this PR does not touch:
laya/router.py:427— "Inference (Agent.system_one) is deliberately left outside the lock so concurrent predictions share a checkpoint without serialising."laya/serve.py:341— the same claim, as the stated reason formax_workers=1.examples/server.py:221— "Locking around this call would undo that."
After this lands all three read as false. laya/serve.py:345 is already ThreadPoolExecutor(max_workers=1), so the bundled server loses nothing — the cost lands on people calling Router.predict or Agent.system_one from their own threads, which is the shape #649's own repro uses. Since that is user-visible, the PR should either update those three or say why the decision changed, and a line in the body stating the new guarantee ("one forward per Agent instance at a time") would help people deciding whether to share an Agent across threads.
Two smaller things:
_inferreadsself.amp_enabled/self.deviceatlaya/agent.py:804andself._fastat:806before taking the lock, while the OOM branch inside it writes exactly those fields at:856-859. The effect is benign becauserun()recomputesenabledat:816, but it means the protection has a hole at those two reads. Moving thewithabove them closes it.- #649 also asked for a test that normal-path concurrency is unchanged, and for multiple concurrent OOMs. Neither is here, and normal-path concurrency is the thing this PR actually changes. A test asserting two
system_onecalls overlap — or documenting that they no longer do — would pin it either way.
Flagging the merges too: this conflicts with #459 in laya/agent.py, which restructures the same try and adds an else: clause plus a shared failure counter, so whoever lands second has to re-thread the lock through it. It also conflicts with #700, whose torch.clear_autocast_cache() in the same OOM branch is a process-global clear that a per-instance lock does not serialise against another Agent's forward.
Requesting changes on the undocumented contract reversal rather than on the fix itself — the race fix is right and I would rather it landed than not, but reversing a published concurrency guarantee silently is the kind of thing that should be a decision, not a side effect.
|
Holding: The per-agent RLock serializes every forward pass, introducing a throughput regression for concurrent callers. Please consider using a reader-writer lock (shared lock for forwards, exclusive lock only for device fallback). |
Bruce-Yii
left a comment
There was a problem hiding this comment.
Answering the reader-writer lock question directly, because it decides whether this is a rebase-and-resubmit or a small amendment.
I built all three designs and measured them: today's main, this PR's per-instance RLock, and the shared-forwards / exclusive-for-demotion RWLock you described. Each case runs in its own process with a hard timeout, because a lock that deadlocks should report as a result rather than hang the harness.
Does the reader-writer design fix the race? Yes
Two overlapping forwards, where one is in flight when the other moves the model to CPU. The detector compares the placement a forward entered with the placement it reaches, so a legitimate post-demotion call is not miscounted as a race:
| design | result |
|---|---|
main (no instance lock) |
['RACE', 'cpu-answer'] — reproduces |
this PR (RLock) |
['cpu-answer', 'cpu-answer'] — no race |
your RWLock |
['cpu-answer', 'cpu-answer'] — no race |
So the design is sound. It is not a workaround for the bug.
The throughput complaint is real, and this is the number
Four independent forwards of 0.25s each — serialised is 1.000s, fully parallel is 0.250s:
| design | elapsed | peak concurrent forwards |
|---|---|---|
| no lock | 0.254s | 4 |
this PR's RLock |
1.004s | 1 |
your RWLock |
0.252s | 4 |
The RLock costs the full 4x, and the RWLock recovers all of it.
One trap in the implementation, which is the part that would cost you a debugging round
A thread discovers the OOM while it is holding the read lock, so it has to upgrade to write. If the upgrade is done in place, it deadlocks:
| upgrade strategy | result |
|---|---|
acquire_write() while still holding read |
DEADLOCK — no result in 12s |
release_read() then acquire_write() |
both threads complete, 2 demotions |
With two forwards hitting OOM at the same time, each holds a read lock and each waits for a write lock that cannot be granted until readers drain. Neither releases, so neither proceeds. Release-then-acquire is the correct shape, and it happens to be the pattern this PR already uses for its fallback lock — so the amendment is smaller than it looks.
Scope of the regression, so you can decide how much it matters
The RLock is per-instance, so separate Agents never contend. The exposure is concurrent forwards against one Agent, and the shipped HTTP server does not do that: docs/http-api.md:185-190 documents a single-worker executor, "one forward pass at a time", with LAYA_MAX_CONCURRENT admission held separately. The batch surfaces fold into predict_batch rather than fanning out.
Where it does bite is in-process callers sharing one Agent across a thread pool, and there is a documented one: README.md:947-953 exists precisely to steer LangChain users away from its default one-call-per-input thread pool, and notes that pool "races on MPS, where concurrent torch forwards abort the process". A user who ignores that advice and calls one Agent from several threads gets the serialisation.
What I would need to move this forward
The throughput contract is the open question and it is yours to set, not mine. Either is defensible:
- Keep the
RLockand state the guarantee — one forward perAgentinstance at a time. Three places currently promise the opposite:laya/router.py:428("concurrent predictions share a checkpoint without serialising"),laya/serve.py:591("One worker, because one forward pass at a[ time]") andexamples/server.py:290("concurrent predictions aren't serialised. Locking around this call would undo that"). The last one is now false. - Take the reader-writer route. Happy to re-review that shape; the deadlock case above is the only implementation trap I found, and release-then-acquire avoids it.
I am not asking for both. Separately, laya/agent.py:965 reads self.device.type to decide whether the OOM branch applies, and that read is still outside the lock — the reads at :976 and :980 are inside it. That is the hole I noted before, and it is unchanged; benign in practice, since the window is the gap between the raise and the lock, but it is there.
Bruce-Yii
left a comment
There was a problem hiding this comment.
This resolves the blocker from my previous review, and it does so in the shape I suggested. I re-tested all seven points on b4e9414c rather than taking the two new commits' word for it.
1. Normal forwards overlap again — 0.251s, peak 4 concurrent
Four independent 0.25s forwards through the new lock:
| design | elapsed | peak concurrent |
|---|---|---|
main (no instance lock) |
0.254s | 4 |
old head 38db478d (RLock) |
1.004s | 1 |
this head (_InferenceRWLock) |
0.251s | 4 |
The 4x serialisation regression is gone. This is the point of the change.
2. The device-move race is still prevented — verified three ways
The hazard is narrow: a forward that entered while the model was on cuda, and by the time it reaches the model another thread has moved it to cpu. Thread A parks inside the forward; thread B is submitted only once A is parked, and B OOMs and moves the model.
| tree | result |
|---|---|
main 6d942c9 |
['RACE', 'ok'] — reproduces |
old head 38db478d |
['ok', 'ok'] |
this head b4e9414c |
['ok', 'ok'] |
So the new lock keeps the guarantee the old one had, and adds the concurrency back.
3 + 5. Concurrent OOMs cannot deadlock
N threads that all hit OOM at the same instant, all forced through upgrade_to_write():
| N | reached the write lock | all completed |
|---|---|---|
| 2 | 2/2 | yes |
| 3 | 3/3 | yes |
| 4 | 4/4 | yes |
| 8 | 8/8 | yes |
Plus a mixed case — 4 normal forwards overlapping 2 OOM forwards: 6/6 completed, 2 demotions, peak 4 concurrent readers. Every case ran in a child process with a hard timeout so a deadlock would be a result rather than a hang.
4. release-read → acquire-write is implemented correctly
upgrade_to_write() is release_read() then acquire_write() — not an in-place upgrade, which is the variant that deadlocks. The accounting is balanced across the whole cycle:
before (0, False, 0) → in_read (1, False, 0) → in_write (0, True, 0)
→ after_downgrade (1, False, 0) → after_exit (0, False, 0)
_readers and _writer return to zero, so the outer with infer_lock: releases exactly one read. No leak across a fallback.
6. The public concurrency claims are accurate again — and this one needs no edit
laya/router.py:428 ("outside the lock so concurrent predictions share a checkpoint without serialising") and examples/server.py:290 ("concurrent predictions aren't serialised. Locking around this call would undo that") were both false under the old RLock and are true under this change. The shipped server's single-worker note in laya/serve.py is unaffected. The three sites are byte-identical to main — the PR does not touch them, and it no longer needs to. That is worth noting because it was one of the two options I offered.
7. The collision is real and is the one thing still outstanding
The branch is still based on 9d95567 (v0.3.21), 229 commits behind, and mergeable=dirty. I reproduced the merge. Three regions:
laya/agent.py— the lock declaration, wheremainnow reads_OOM_FALLBACK_LOCK = threading.Lock()and adds_BATCH_AUTOCAST_CACHE = ContextVar("laya_batch_autocast_cache", default=False)on the next linelaya/agent.py— the OOM block, wheremainhas gained the#700guardif self.device.type == "cuda" and _BATCH_AUTOCAST_CACHE.get(): torch.clear_autocast_cache()tests/test_runtime_fixes.py— the import line
The one to watch is _BATCH_AUTOCAST_CACHE. It does not exist anywhere on this branch. Taking this side of the conflict wholesale would silently drop 06fe814 ("perf(agent): reuse CUDA autocast weight cache across batches") — three references on main, including the cache-clear inside the very OOM path this PR is rewriting. The rebase needs to keep the ContextVar declaration and the _BATCH_AUTOCAST_CACHE.get() guard inside the demotion, while wrapping it in the new read/write structure.
What I did not accept silently: the race assertion was removed from the suite
This is the one thing I want to raise, and the reason is not that the deletion looks convenient — it does not, on the evidence.
The old test proved the race by treating any overlap on cuda as a race:
concurrent_call = self.active > 1 and self.placed == "cuda"
...
if concurrent_call:
raise RuntimeError("Expected all tensors to be on the same device, cuda:0 and cpu")That detector is wrong for this design. With a shared read lock, two concurrent forwards are the intended behaviour, so that line now fires on correct code — it had to go, and removing it was the right call. The docstring change ("so fallback serialization can be exercised") is an accurate description of what the test does now.
The consequence is that the actual hazard no longer has any regression guard. concurrent_call is still computed at tests/test_runtime_fixes.py:310 and never used, and the string same device no longer appears in the file at all. The suite now covers claims 3 and 5, which is real value, but claim 2 is unguarded: a future change that took the write lock after moving the model, or dropped the upgrade, would leave the suite green.
Reinstating it is small — the detector I used is about six lines, and it is RED on main:
entered = self.placed
... park ...
if entered == "cuda" and self.placed == "cpu":
raise RuntimeError("Expected all tensors to be on the same device, cuda:0 and cpu")That distinguishes overlap (fine) from movement (the bug), which is exactly the distinction the reader-writer lock introduces. I am not asking for it to block the merge — the design is right and I would rather you had this than not — but a lock whose whole purpose is a race should not ship with the race assertion removed and nothing in its place.
Also worth deleting while you are in there
concurrent_call at :310 is dead. Small, but it reads as if something is being detected.
Verification I ran
tests/test_runtime_fixes.py 45 passed, 0 failed; tests/test_hooks_api.py 387 passed, 0 failed; ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 → All checks passed; python -m compileall -q laya/ tests/ → exit 0.
The three-way race table, the throughput table, the N=2..8 deadlock sweep and the lock-accounting trace were each run in a child process with a 20–30s hard timeout, so a hang would have surfaced as a result rather than stalling this review. The one caveat: the OOM path is simulated with a fake model that reports a CUDA OOM, because this host has no CUDA device. That is faithful to the control flow being changed — the lock, the upgrade and the demotion are the real code — but it is not a test on real hardware, and the maintainer may reasonably want one before merging a concurrency change.
Requesting changes for the rebase only. The concurrency work itself is what I would have asked for.
b4e9414 to
f01ca62
Compare
|
Rebased onto the latest main and addressed the remaining review point. The upstream _BATCH_AUTOCAST_CACHE guard is preserved, and the regression now detects actual device movement during an in-flight forward rather than rejecting legitimate concurrent forwards. Runtime, router, lint, compile, and diff checks all pass. |
f01ca62 to
b4e9414
Compare
|
I reviewed the latest notifications and CI results. The RW-lock implementation and race fix remain sound, but the attempted rebase introduced unrelated upstream regressions across docs, Docker, Python, Windows, and TypeScript checks. I have restored the last locally verified implementation ( I will redo the rebase from a clean upstream base, preserving the current |
|
Small attribution correction to my review above: Nandha was the one who suggested considering a reader-writer lock; that was not your proposal. The throughput contract is a maintainer/project decision, not something I should have attributed to you. The technical findings in the review are unchanged. |
|
Holding this, and I owe you a straight account of why, because the design in it is the one that shipped. You did the work: the first round found the race, the two-thread test was genuinely load-bearing, and after I asked about a reader-writer lock you built it and measured it, 0.251s with peak 4 concurrent against 1.004s serialized. That is the shape 0.3.23 ships. What it ships as is #810, because this branch is conflicted. You said you had restored If you want to keep going here, the useful thing is not a rebase of this branch, it is a look at what landed: #810's Thanks for the measurement round. It is what made the design decidable. |
|
Closing this as already fixed, and the fix on
with self._gate.read_lock():
out = run()
...
with self._gate.write_lock(), _OOM_FALLBACK_LOCK:I only noticed while resolving this branch's conflict, because the conflict is your version of the lock against the merged version of the same idea. One difference worth recording, since it is the reason I would have asked for a change here even without the overlap. Your gate exposes Thank you for the issue analysis and for the deterministic two-thread test, which is the part that made #649 actionable. Nothing here is lost. |
Fixes #649 by serializing inference per Agent instance while preserving concurrency across separate Agents. A deterministic two-thread regression test verifies that a GPU OOM fallback cannot move the shared model under another in-flight request.