Repository navigation
fix(verify): isolate the soak concurrency burst in a helper process - #1074
Conversation
Section 3 spawns N threads against one loaded agent in-process. On Apple silicon with torch 2.14 that exact pattern -- several threads hitting an MPS model at once -- SIGABRTs the interpreter (Metal: _status < MTLCommandBufferStatusCommitted), which kills the whole run with no verdict: sections 1-2 findings are lost, the promised CPU retry never happens, and the script exits 134 instead of the non-zero verdict its header guarantees. Serial MPS on the same model is fine, so this is the threads-at-once pattern, not the device. Run the burst in a disposable helper process (same script, `--concurrency-child`, same loader, same threads, one JSON verdict on stdout) and file the outcome under the original report names. The helper dying (bad exit, no verdict line, timeout) becomes `FAIL concurrency on <device>` plus the same CPU retry the script always had; leak sections 1-2 stay in-process untouched. Verified on this machine: before, the default run aborted (exit 134, output stops mid-section-1); after, the full run completes -- sections 1-2 unchanged, `FAIL concurrency on mps (helper died without a verdict (exit -6))`, then the CPU pass (48/48, all match) -- and exits non-zero with the named failure instead of aborting.
|
Merged in 0.4.3. The diagnosis is the part I want to single out: the SIGABRT happens below Python, so the worker pool's Running the burst in a disposable helper process with the same loader and the same threads keeps the thing being measured identical while moving the crash somewhere the parent can file. The Verified: |
What
verify/soak_check.pysection 3 spawns N threads against one loaded agent in-process. On Apple silicon with torch 2.14, that exact pattern -- several threads hitting an MPS model at once -- SIGABRTs the interpreter (Metal: _status < MTLCommandBufferStatusCommitted). The driver aborts below Python, so the worker pool'stry/exceptnever sees it: the whole run dies withexit 134, sections 1-2 findings are lost, the CPU retry the header promises never happens, and the script violates its own "Exits non-zero if a check fails" contract. Serial MPS calls on the same model are fine, so this is the threads-at-once pattern, not the device.Fix
Run the burst in a disposable helper process: same script with a new
--concurrency-childflag, same loader, same threads, same single-threaded reference, one JSON verdict on stdout. The parent files the outcome under the original report names, and a dead helper (bad exit, no verdict line, timeout) becomesFAIL concurrency on <device>plus the same CPU retry the script always had. Leak sections 1-2 stay in-process untouched -- they measure the serving process, not a throwaway helper.Verification (this machine, MPS available, cached checkpoints,
HF_HUB_OFFLINE=1)Before: default run aborted mid-section-1, exit 134, no verdict.
After (
--calls 200 --cycles 4 --threads 4defaults):The crash is contained and named; the CPU retry proves the model is thread-safe; exit code is non-zero. Child-flag smoke check passes (bogus ref -> exactly one JSON verdict line, exit 3).
verify/is not CI-run, so no workflow changes.