Skip to content

fix(verify): isolate the soak concurrency burst in a helper process - #1074

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:w809-soak-child-isolation
Oct 11, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:w809-soak-child-isolation

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

verify/soak_check.py section 3 spawns N threads against one loaded agent in-process. On Apple silicon with torch 2.14, that exact pattern -- several threads hitting an MPS model at once -- SIGABRTs the interpreter (Metal: _status < MTLCommandBufferStatusCommitted). The driver aborts below Python, so the worker pool's try/except never sees it: the whole run dies with exit 134, sections 1-2 findings are lost, the CPU retry the header promises never happens, and the script violates its own "Exits non-zero if a check fails" contract. Serial MPS calls on the same model are fine, so this is the threads-at-once pattern, not the device.

Fix

Run the burst in a disposable helper process: same script with a new --concurrency-child flag, same loader, same threads, same single-threaded reference, one JSON verdict on stdout. The parent files the outcome under the original report names, and a dead helper (bad exit, no verdict line, timeout) becomes FAIL concurrency on <device> plus the same CPU retry the script always had. Leak sections 1-2 stay in-process untouched -- they measure the serving process, not a throwaway helper.

Verification (this machine, MPS available, cached checkpoints, HF_HUB_OFFLINE=1)

Before: default run aborted mid-section-1, exit 134, no verdict.
After (--calls 200 --cycles 4 --threads 4 defaults):

   PASS every answer identical to the first
   PASS no Python objects retained across 200 calls
   ...sections 1-2 unchanged, all green...
   FAIL concurrency on mps    helper died without a verdict (exit -6)
   the mps pass failed; retrying on cpu
   PASS all 48 concurrent calls returned (cpu)
   PASS concurrent answers match the single-threaded reference
   1 check(s) failed:
     - concurrency on mps (helper died without a verdict (exit -6))

The crash is contained and named; the CPU retry proves the model is thread-safe; exit code is non-zero. Child-flag smoke check passes (bogus ref -> exactly one JSON verdict line, exit 3). verify/ is not CI-run, so no workflow changes.

Section 3 spawns N threads against one loaded agent in-process. On Apple silicon with torch 2.14 that exact pattern -- several threads hitting an MPS model at once -- SIGABRTs the interpreter (Metal: _status < MTLCommandBufferStatusCommitted), which kills the whole run with no verdict: sections 1-2 findings are lost, the promised CPU retry never happens, and the script exits 134 instead of the non-zero verdict its header guarantees. Serial MPS on the same model is fine, so this is the threads-at-once pattern, not the device.

Run the burst in a disposable helper process (same script, `--concurrency-child`, same loader, same threads, one JSON verdict on stdout) and file the outcome under the original report names. The helper dying (bad exit, no verdict line, timeout) becomes `FAIL concurrency on <device>` plus the same CPU retry the script always had; leak sections 1-2 stay in-process untouched.

Verified on this machine: before, the default run aborted (exit 134, output stops mid-section-1); after, the full run completes -- sections 1-2 unchanged, `FAIL concurrency on mps (helper died without a verdict (exit -6))`, then the CPU pass (48/48, all match) -- and exits non-zero with the named failure instead of aborting.
@NandhaKishorM
NandhaKishorM merged commit 1672a0d into NandhaKishorM:main Oct 11, 2026
36 of 38 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.3. The diagnosis is the part I want to single out: the SIGABRT happens below Python, so the worker pool's try/except never sees it, and the script dies at exit 134 with sections 1 and 2 findings already computed and then lost. A checker that cannot report what it already found is worse than one that fails the check, and this is the second time this pass that a failure mode turned out to be about where the error surfaces rather than what the error is.

Running the burst in a disposable helper process with the same loader and the same threads keeps the thing being measured identical while moving the crash somewhere the parent can file. The --concurrency-child flag reusing the same script rather than a second file is the right call, since the two would drift.

Verified: verify/soak_check.py is outside the wheel's packages list, so this changes nothing a pip install laya user runs. All 87 suites pass on the merged tree and the 94-row real-weights decision diff is identical to 68804629.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants