Skip to content

fix(agent): scope the autocast fallback to the failed request - #459

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
MoonPointer-Byte:fix/autocast-scoped-fallback
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
MoonPointer-Byte:fix/autocast-scoped-fallback

Conversation

@MoonPointer-Byte

@MoonPointer-Byte MoonPointer-Byte commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

What

Agent._infer retries an unsupported autocast op in full precision. AMP stays on until three such failures in a row, then it is turned off and a warning is printed. A forward that succeeds under autocast clears that streak.

Why

On MPS or CPU, one unsupported op used to set amp_enabled = False and dtype = float32 for the rest of the process. Leaving AMP on forever means a build that lacks the op pays for two forwards on every request. This is the autocast half of #351. Rebased onto current main; the CUDA OOM path, including the /health fallback counters, is unchanged.

How it was verified

python tests/test_runtime_fixes.py
python tests/test_portability.py
python -m compileall -q laya/ tests/
python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120

test_runtime_fixes.py: 51 passed. test_portability.py: 35 passed.

Follow-ups

None on this change.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for picking up this follow-up from #351, a single bad op should not slow the process down for good. Two things before it can merge: tests/test_runtime_fixes.py still asserts the old behaviour ('infer/falls back and disables amp') and now fails, so please update it to the new contract. Also, since an unsupported op usually fails the same way every time, please add the consecutive-failure threshold the issue mentioned (for example, disable AMP with a warning after 3 failures in a row), so a build that lacks an op does not pay for two forward passes on every request. #349 has landed, so please rebase; the only overlap is in tests/test_portability.py.

@rahul05ranjan rahul05ranjan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — correct fix for a real state-leak bug, with thorough tests.

The bug is real: when an MPS/CPU build hit an unsupported autocast op, the fallback did self.amp_enabled = False; self.dtype = torch.float32 and retried — but those mutations stuck for the life of the process, so one unsupported op silently slowed every later request (#351).

The fix changes run() to accept an enabled override and retries the single failed request with run(enabled=False), leaving the agent's AMP settings untouched. The enabled=None default preserves the existing recompute-on-retry behavior for the OOM/CPU-fallback path.

The tests are strong:

  • autocast/answers after an unsupported op verifies the retry succeeds and AMP stays enabled.
  • autocast/a later request still answers + autocast/later request is a single forward prove the fix is scoped (no lingering slowdown).
  • autocast/repeated failure leaves amp enabled covers the case where the retry also fails, confirming AMP isn't left off.

Minor nit (non-blocking): the run(enabled=None) signature uses None as a sentinel for "recompute", which is slightly subtle — a comment already explains it, but an explicit keyword like run(recompute=True) or a separate helper would be marginally clearer. Not worth blocking on.

This is a clean, well-scoped fix.

@MoonPointer-Byte
MoonPointer-Byte force-pushed the fix/autocast-scoped-fallback branch from 9e17ba2 to 0e737ce Compare September 25, 2026 12:53
@MoonPointer-Byte

Copy link
Copy Markdown
Contributor Author

Rebased onto main after #349. test_runtime_fixes.py now matches the new contract: one unsupported op retries in full precision and leaves AMP on, and AMP turns off with a warning only after three misses in a row. A forward that succeeds under autocast clears that streak. python tests/test_runtime_fixes.py (42 passed) and python tests/test_portability.py (35 passed).

umran666 added a commit to umran666/laya that referenced this pull request Sep 25, 2026
Agent exposes cpu_fallback_count and last_fallback_reason, recorded under the OOM fallback lock, and GET /health reports them per loaded agent so a silent GPU->CPU degradation is visible to operators.

Part 1 of NandhaKishorM#351 (observability); NandhaKishorM#459 covers part 2 (scoped autocast drop).
@MoonPointer-Byte
MoonPointer-Byte force-pushed the fix/autocast-scoped-fallback branch from 0e737ce to b775efc Compare September 28, 2026 06:31
@MoonPointer-Byte

Copy link
Copy Markdown
Contributor Author

Rebased onto current main. The autocast behavior is the same: one miss retries in full precision and leaves AMP on, and AMP turns off after three misses in a row. python tests/test_runtime_fixes.py (51 passed) and python tests/test_portability.py (35 passed).

Retry an unsupported autocast op in full precision. Turn AMP off only after three failures in a row, so a build that lacks the op does not pay for two forwards on every later request.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @MoonPointer-Byte! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants