Repository navigation
fix(onnx): export a batch-dynamic graph, and check that it is one - #688
Conversation
7f87d9f to
06b05d9
Compare
e98bc5d to
2d2d31e
Compare
Bruce-Yii
left a comment
There was a problem hiding this comment.
The load-bearing claim holds up — I reproduced it by mutation — but there is one blocker, and it is the last line of the new test file.
The blocker: os._exit is unguarded, and pytest can be made to report success having run nothing
tests/test_onnx_export_batch.py:724 is os._exit(1 if FAIL else 0), at module level. There is no if __name__ == "__main__": guard anywhere in the file — the three __main__ hits are all inside comments, one of which (:666) explains that a CLI body moved into a function precisely so a check can read main's source rather than the guard.
The file matches pytest's test_*.py glob, so a plain pytest run imports it and the os._exit fires during collection. Measured on this branch, same interpreter, same tree:
| command | output | exit |
|---|---|---|
python -m pytest tests/test_runtime_fixes.py -q |
9007 bytes — 60 passed, then INTERNALERROR> Traceback |
3 |
python -m pytest tests/test_onnx_export_batch.py -q |
0 bytes | 0 |
The control is the comparison that matters. test_runtime_fixes.py ends in sys.exit(...), and pytest turns that into a loud ~30-line INTERNALERROR with exit 3. os._exit(0) produces no output at all and a success status — the file's own 83 checks never print, and pytest reports green having executed none of them.
This would be the only such suite in the repository: sys.exit appears 58 times across tests/*.py, os._exit zero.
It does not currently break CI, and I want to be precise about why, because it changes how urgent this is: ci.yml:232 runs it as python tests/test_onnx_export_batch.py (script mode, where the guard would not matter), and every pytest invocation in ci.yml:193/333/387 and release.yml names its files explicitly, with ci.yml:335-337 citing #374 for that choice. No bare pytest, no testpaths/addopts, no conftest.py. So it is a latent trap, not an active break. The exposure is a contributor running pytest tests/ locally to check their own change, who gets a green that means nothing — and this repo already has form here.
Minimal fix, which keeps the macOS workaround for the invocation CI actually uses:
if __name__ == "__main__":
sys.stdout.flush()
sys.stderr.flush()
os._exit(1 if FAIL else 0)The gate itself works
I mutated TRACE_BATCH = 2 to 1 — the original #695 bug — and the suite goes red with the consequence named:
ONNXRuntimeError: INVALID_ARGUMENT : Got invalid dimensions for input: marker_pos
index: 0 Got: 3 Expected: 1 (exit 1)
Restored, the same tree is 83 passed, 0 failed, exit 0, summary printed. Separately, the SystemExit path you added is covered on its own rather than only as a side effect: :472-513 asserts verify_batch_dynamic raises SystemExit in three separate cases, and :546 pins the shipped default via inspect.signature(...).parameters["atol"].default == 1e-3 so the expectation cannot drift with the constant it is checking.
On INT8_ATOL = 1e-1 — a guardrail, because the tightening looks tempting and is wrong
I spent time trying to show this was too loose, and it is not, at least not for the reason it first appears. A good int8 graph on this machine already drifts further than the PR's own CI log suggests. Tightening the constant until the healthy export fails:
INT8_ATOL |
result on this tree |
|---|---|
1e-2 |
exit 1 — a good graph fails |
3e-2 |
exit 1 |
5e-2 |
exit 1 |
7e-2 |
exit 0 |
1e-1 (shipped) |
exit 0 — 83 passed, 0 failed |
So on Windows here a healthy export needs somewhere between 5e-2 and 7e-2, against the 9.95e-04 you measured on macOS arm64 and the 2.53e-02 that failed your CI on Linux x86. That is a spread of roughly two orders of magnitude across three platforms, and 1e-1 is the only value in the set that covers all of them. The headroom is not 100x — on the platform that matters most here it is under 2x.
Worth stating in the docstring, because the obvious "reviewer move" here is to tighten it: any value at or below 5e-2 will reject valid exports on at least one of the three platforms, and the resulting failure would look like a broken quantizer rather than a too-tight tolerance. If you want the gate to carry more numerical weight than "does it still run at batch > 1", the place to add it is a per-platform floor recorded in the test, not a lower constant.
Requesting changes for the os._exit guard only. Everything else I checked holds.
|
Holding on the The finding is the kind I will not merge past, because of what it does rather than what it is: The change itself is worth having and your framing of it is honest: the original bug is fixed on Add the |
`scripts/export_onnx.py` traced the model with a single row while declaring a
`batch_size` axis. `torch.export` specializes every example dimension whose extent is
1, so the graph it wrote ran at batch 1 and failed inside ONNX Runtime at batch 2
("Attempting to broadcast an axis by a dimension other than 1"). `ONNXAgent` builds one
row per question, so every two-question `predict`/`system_one` call, and
`predict_batch`/`predict_long` at any size, was unservable -- while the script printed
"Successfully exported" and torch raised the problem only as a UserWarning.
Trace with two rows, and stop claiming success without checking: the export now runs the
graph in ONNX Runtime at batch 1, 2 and 3 and compares it with PyTorch, raising rather
than writing a graph that cannot serve two questions. Probabilities rather than raw
logits, because the decompositions move a raw `act_logits` value by up to 1.2e-02 on the
322M multilingual checkpoint while the probabilities agree to 6.1e-05.
Measured on that checkpoint: the five calls that failed go to 5/5, and all nine answers
over three states x choice/score/noul are identical to the torch `Agent`, max |delta|
0.00000 across 57 numeric fields.
A declared axis proved nothing here -- the broken graph declared `batch_size` too -- so
`tests/test_onnx_export_batch.py` runs the graph instead of reading its shapes. It
exports a tiny stand-in model through the same helper at batch 1, 2, 3 and 5, so it
needs no checkpoint and no network and fits the existing weight-free `onnx-export` job.
That job moves to the declared `[onnx]` extra -- which already exists upstream as
["onnx", "onnxruntime", "onnxscript"], so nothing is lost and `onnxscript` is gained, which
`torch.onnx.export` needs for the path it takes by default. Setting `TRACE_BATCH` back to 1 turns 9
of the suite's 48 checks red.
Re-proved from scratch: exporting the same tiny model with a one-row and a two-row example produces
graphs that declare the IDENTICAL `['batch_size', 'seq_len']` shape, and only the two-row one runs at
batch 2 and 3. Reading declared shapes is precisely the check that passes the broken graph. And the
batch dimension really is the question count -- `collate_items` flattens one item per question per
state, so 2 questions is batch 2 and 2 questions over 2 states is batch 4.
Verifying by default changed the command's contract in three ways the first draft missed, each now
guarded and pinned. `onnxruntime` became a hard requirement of a command that never needed it --
`torch.onnx.export` does not touch it -- so an install with onnx + onnxscript printed "Successfully
exported" and then died on a bare ModuleNotFoundError; the import is guarded and names both
`pip install laya[onnx]` and `--no-verify`. `--quantize` ran after the function returned, so the INT8
copy that docs/evals.md tells people to deploy was written unverified, which is precisely the "runs but
returns different numbers" case this verifier is for; `export_to_onnx` now returns the model so the
quantized graph is checked too, at its own measured tolerance (weight-only INT8 drifts up to 9.95e-04
across the swept shapes, already at the fp32 atol, so INT8_ATOL = 2e-2). And the success message
claimed "runs at batch > 1 and matches PyTorch" while the sweep never used num_markers=1: marker width
is not dynamic, because `DecisionModel.forward` branches on `p.size(-1) >= 2` in Python, so a graph
traced at width 2 raises an ONNX Runtime TopK error at width 1 -- reachable, since a one-criterion
`score` question is accepted and yields one marker. That limitation predates this change and making
width dynamic means removing a branch from the model, so the message, the docstring and docs/evals.md
now state it instead of overclaiming.
Four checks were added because a mutation showed the verifier itself was unpinned: flipping `verify`
to default False, removing the call to it, disabling its numeric comparison and widening its
tolerance to 1e9 all left the suite green. The numeric check is now pinned by exporting one model and
verifying a perturbed copy against that graph -- a graph that runs at every batch and quietly returns
different values, which is the only case the probability comparison exists for. The perturbation is
non-uniform and on a weight rather than a final bias, because a uniform shift of the last bias moves
no probability at all: softmax is shift-invariant, the blind spot the verifier's own docstring
records.
Three smaller things the export needed on the way, each of which cost a confusing failure first:
* `example_inputs` reads `TRACE_BATCH` at call time instead of binding it as a signature
default, so `export_onnx.TRACE_BATCH = 1` actually reaches it -- the one experiment anybody
revisiting this bug will run, and it silently did nothing before.
* `marker_pos` indexes the sequence axis, so `num_markers >= seq_len` made the head's marker
gather read out of bounds: `IndexError` in torch, and in ONNX Runtime the much less obvious
"GatherElements op: Out of range value in index tensor". It now raises with the reason.
* the padding written per row was `attention_mask[row, seq_len - row:]`, which reaches 0 at
`row == seq_len` and goes negative past it, handing the model an all-padding row for any
`batch > seq_len`. Torch and ONNX Runtime legitimately disagree on such a row, and the caller
would read that as an export failure. Clamped with `max(1, ...)`, so every row keeps at least one
real token. Measured at batch 32, seq_len 16, where 16 of the 32 rows come out entirely padding:
2.76e-01 on `act_logits` and 1.12e-01 on `logits`, against 2.38e-07 and 1.19e-07 once clamped.
The `dynamic_axes` call now carries the reason it is still there rather than the `dynamic_shapes`
torch prefers: measured, `dynamic_shapes` with an explicit `torch.export.Dim("batch")` from a
one-row example still exports `input_ids` as `[1, "seq"]` and still fails at batch 2, so it is
`TRACE_BATCH` and not the axis spec that fixes this. `dynamic_axes` is nonetheless load-bearing --
drop it and every axis exports static -- and torch already calls it deprecated, so on the release
that removes it this export goes *silently* static: a graph that serves one question, no error,
exactly the failure this file just had. `verify_batch_dynamic` is what would catch that day, which
is why it runs by default and compares numbers rather than only shapes.
…path; correct the width claim From an adversarial review. The functional change is sound and the review confirmed it independently: `TRACE_BATCH = 2` is sufficient (batch 1..128 x marker width 2..16 all match PyTorch to 3e-07, and re-tracing at 3 markers is identical), `verify_batch_dynamic` genuinely rejects a real batch-1 trace, the INT8 path really does preserve batch dynamism, and the 9.95e-04 / 1e-3 / 2e-2 numbers in the comments reproduce exactly. Every HIGH finding was a COVERAGE failure: 28 mutants survived both suites, concentrated on the three things that actually reach a user. 1. THE I/O NAME CONTRACT. `laya/onnx_agent.py` builds its feed from hardcoded strings and calls `session.run(["logits", "act_logits"])`. Every check here built its feed from `INPUT_NAMES` too, so a permutation was self-consistent: swapping the two `OUTPUT_NAMES` exported a graph whose `logits` is the act head, and `ONNXAgent` then reads the act head as the option scores -- silently wrong answers on every request, no error anywhere. Swapping `input_ids`/`attention_mask` (same dtype and shape, so ORT accepts it) gave a max deviation of 5.27 from PyTorch. Now asserted against literals, against the strings `onnx_agent.py` actually contains, and by running the graph BY NAME so the shapes each name carries are pinned (`logits` is (batch, num_markers), `act_logits` is (batch, 2)). 2. THE QUANTIZE PATH -- the one `docs/evals.md` tells people to deploy -- was never exercised on a real torch export. `tests/test_onnx_quantize.py` builds a one-node MatMul graph with no intermediate `value_info`, so it cannot reach `del model.graph.value_info[:]`; this suite exported a real graph but never quantized it. Deleting that one line left both suites green and made `python scripts/export_onnx.py --quantize` crash outright with `InferenceError: Inferred shape and existing shape differ in dimension 0`. The suite now quantizes the graph it already exports, verifies it at the INT8 tolerance, and checks it kept a symbolic batch axis. 3. THE CLI WIRING was pinned by grepping `__main__` for substrings, and the stubs recorded only THAT a function was called. So `batches=(1,)` on either call, `--no-verify` flipped to `store_false`, `verify=False`, `atol=1e9` on the INT8 check, and quantizing over the fp32 output all shipped a success message while checking nothing. The CLI body is now `main(argv=None)` -- the smallest change that makes it callable -- and a test drives it with a stubbed `Agent` and asserts the `batches`, the `atol` and the output path it actually passes. 4. `logits` WAS UNPROTECTED by the drift check. The perturbation picks the last 2-D parameter, which lives in `act_head`, so only `act_logits` moved; skipping `logits` in the comparison loop left the suite green. No perturbation isolates `logits` in this model -- the act head consumes the scorer's output, so every weight reaches both, and whichever output the loop still compares catches the drift. Pinned instead on what the verifier reports: both names, at every batch it sweeps. AND ONE CLAIM THAT WAS MEASURABLY FALSE. Three places said marker width "is not dynamic" and is "fixed at the traced value". Measured on a graph traced at 2 markers: widths 2, 3, 4, 5 and 8 all run and match. Only width 1 fails, via `TopK: k argument [2] should not be greater than specified axis dim value [1]`, because `DecisionModel.forward` branches on `p.size(-1) >= 2` in Python and that BRANCH is what the trace bakes in. As written a user would conclude they must re-export per criteria count. Corrected in the docstring, in the success message, and in docs/evals.md, which now also notes that `collate_items` sizes markers to the batch maximum, so a one-criterion question only hits it when every item in the batch has exactly one marker. Verified: tests/test_onnx_export_batch.py 69 passed (was 48), tests/test_onnx_quantize.py 14 passed, all 61 ci.yml script suites, ruff and compileall clean. All 9 mutants from the review now killed, 0 survivors. Re-exported the real 322M multilingual checkpoint with this branch's script: verification passes at batch 1, 2, 3 and the graph runs at 1, 2, 3, 5 and 8 -- the bug this PR fixes stays fixed.
`tests/test_evals.py::test_docs_and_the_cli_name_the_same_flags` was already red on this branch before today's changes, and I missed it: my earlier CI sweep across the five PRs checked `dependency CVEs` and `tests (windows)` and not this job. It passes on `upstream/main` and on every other branch. The test collects every `--flag` in `docs/evals.md`, skipping lines that mention `export_onnx.py` because "lines that run another script teach that script's flags, not these", and asserts the rest are registered by an `evals` subcommand. Upstream only ever names `--quantize` on lines that also name `scripts/export_onnx.py`. The paragraph this branch added introduced `--quantize` and `--no-verify` on lines that did not, so both were collected as `evals` flags and the assertion failed with "the page teaches ['--no-verify', '--quantize'], which no subcommand registers". Both mentions now name `scripts/export_onnx.py`, which is what they are flags of, and the marker-width explanation moves into its own paragraph so the prose still reads sensibly. tests/test_evals.py: 58 passed. test_evals_api, test_evals_onnx (20) and tests/test_onnx_export_batch.py (69) green. All 61 ci.yml script suites pass. ruff clean.
…zation on Windows Two CI failures, both from the checks added in the previous commit. 1. `onnx export` went red with `logits max abs prob diff 1.03e-02` against the 1e-3 default. The model IS seeded, so this is not run-to-run noise -- it is cross-MACHINE: the same graph and the same weights drift 4.69e-04 here and 1.03e-02 on a GitHub runner, because a 32-dim random model has near-tied logits and softmax amplifies the difference between one machine's fp32 kernels and another's. 1e-3 sits inside that gap, so the suite failed by hardware. The previous commit did not introduce the fragility -- an unperturbed `verify_batch_dynamic` call was already there -- it added a second call and a second chance for it to fire. The tiny-model checks now pass an explicit `TINY_ATOL = 5e-2`. The 1e-3 DEFAULT is untouched, which matters: it is what the 322M checkpoint is held to, and it passes at 6.14e-05. Measured separation on the tiny model: unperturbed 8.9e-08 (and 1.03e-02 at its worst across machines), perturbed 6.4e-01. So 5e-2 is 5x above the worst drift observed and 13x below the signal the check exists to catch. Re-ran every mutant afterwards: all 9 still killed, 0 survivors, so the looser tolerance did not make the guards vacuous. 2. `tests (windows)` died with an illegal instruction (exit 132) inside onnxruntime, quantizing the real torch-exported transformer graph the new check builds. It is a hard crash, not an exception, so it cannot be caught -- `tests/test_onnx_quantize.py` survives on Windows only because its graph is a single MatMul node. Both the quantize check and the CLI check (which exports and quantizes) now skip on Windows and say so, leaving them gated by the `onnx export (quantization, weight-free)` job on Linux, which is the job the documented `--quantize` invocation has to keep green. tests/test_onnx_export_batch.py: 69 passed. tests/test_onnx_quantize.py: 14 passed. All 61 ci.yml script suites. ruff clean. Mutation sweep: 0 of 9 survive.
…rances that actually ship Re-review of fcf7bb5 found four problems with the checks the previous two commits added. All four were mine. 1. THE SUITE EXITED 134 INTERMITTENTLY, AFTER PRINTING "0 failed". The new checks open ONNX Runtime sessions -- one per `verify_batch_dynamic` call, plus two the name-contract and quantize blocks hold -- and leaving them alive means interpreter shutdown dies with `libc++abi: recursive_mutex lock failed`. A red CI job with a green summary, on the exact command .github/workflows/ci.yml runs in the `onnx-export` job. Measured here: 1 of 14 at fcf7bb5, and a reviewer saw 5 of 20; `verify_batch_dynamic` now releases its session in a `finally`, the two block-local sessions are deleted where they are used, and the three long-lived module-level ones are released before the file ends. Re-measured: 0 of 30. I had earlier seen this message once and called it a transient teardown artifact; it was not. 2. THE TOLERANCE THAT SHIPS WAS PINNED BY NOTHING. `export_to_onnx` calls `verify_batch_dynamic` with no `atol`, so the default is what every real export is held to -- and changing it to 1e9 left all three suites green. Its value is asserted now, and so is that it REJECTS a calibrated error: one scorer weight scaled by 1.25, not the linspace(0, 5) sledgehammer that proves nothing about a realistic corruption. 3. THE TINY_ATOL COMMENT WAS FALSE. It said the 1e-3 default "is what bounds quality and it is asserted separately"; nothing asserted it. Worse, 5e-2 is blind to a scorer weight scaled by 3.0, which the default catches from 1.25 upward -- so the four checks using it demonstrate that the graph RUNS at several batches, not that the verifier would catch a bad one. The comment now says exactly that, and the sensitivity proof lives at the shipping tolerance where it belongs. 4. `INT8_ATOL` WAS BOUNDED ONLY FROM BELOW. `> 1e-3` let 1e9 through, and my first attempt at a two-sided bound (`1e-3 < x < 0.1`) let 1.1e-3 through -- which is BELOW the 9.95e-04 drift the tolerance has to accept, so a real quantized graph would have started failing. Now `5e-3 < INT8_ATOL < 5e-2`: above the drift it must accept, well under the 0.29 that per-tensor quantization produces. I also removed a source-text check I wrote for the session release: the mutation that disables the release leaves the string in the file, so it could not distinguish and was worse than nothing. The release is evidenced by the flake rate, which is what it actually protects. Verified: tests/test_onnx_export_batch.py 73 passed, 0 failed, and 0 of 30 runs exit nonzero. tests/test_onnx_quantize.py 14 passed, tests/test_evals.py 58 passed, all 61 ci.yml script suites, ruff and compileall clean. Tolerance mutants: default atol -> 1e9, INT8_ATOL -> 1.1e-3, INT8_ATOL -> 0.29 and disabling the diff comparison are all killed.
…checks were weaker than claimed The previous commit claimed to have fixed an intermittent exit 134 by releasing ONNX Runtime sessions. Re-measuring says that is wrong, so it is corrected here rather than left standing. THE SESSION RELEASE IS NOT WHAT FIXED IT. With every session explicitly released: 3 of 30 runs abort. With the release in `verify_batch_dynamic` removed: 0 of 30. The earlier "0 of 30" was luck, not a fix, and I reported it as one. Nor is it onnxruntime by itself -- 1 or 3 bare sessions opened and held in a fresh process abort 0 of 15 times. What this file actually does that nothing else in the suite does is put torch, onnxruntime and onnxruntime's quantizer in one process, and on macOS that combination aborts during C++ static destruction, after the summary has printed "0 failed": `libc++abi: ... recursive_mutex lock failed: Invalid argument`. A green report and a red job. Python cannot reorder two native libraries' destructors, so the suite no longer runs them: it flushes and calls `os._exit` with the status it computed. Every test has completed and its result is printed by then. Measured: 0 of 40 runs abort, against 4 of 42 exiting normally, and the status still propagates -- a deliberately failing check exits 1, and TRACE_BATCH=1 still exits 1. Three of the checks added in the last two commits were weaker than their names claimed: 1. ONLY THE LEADING AXIS WAS CHECKED FOR BEING SYMBOLIC. Declaring the marker axis of `logits` static left 73 checks green, so the graph could tell a consumer the option count is fixed at the traced width. Now every axis is checked, against a table written out LITERALLY -- my first attempt read the expectation from `export_onnx.DYNAMIC_AXES`, the constant under test, so shrinking it shrank the expectation too and the check was vacuous. 2. THE VERIFIER'S INPUT VARIETY WAS UNPINNED. `verify_batch_dynamic` prints "batch 1, 2 and 3 with 2-4 markers", and sizing every batch with the same shapes left the suite green -- the sequence and marker axes were then never exercised at more than one width, so a marker-static graph would have verified clean. The shapes it asks for are now spied on and asserted distinct. 3. `example_inputs` DID NOT HAVE TO VARY `qtype`. Flattening it to one value traced a single branch of `DecisionModel.forward` with 73 checks green. I also removed a weakref check I wrote for the session release: on the success path the local dies on return either way, and on the raising path the traceback is cleared before any observer can look, so it could not fail. That is the second unfalsifiable pin I have written for this line; the line stays as hygiene, credited with nothing. Verified: 78 passed, 0 failed, 0 of 40 nonzero exits. Mutation table re-run at this head: 19 of 19 killed, including all three gaps above. test_onnx_quantize 14 passed, test_evals 58 passed, ruff and compileall clean.
….Dim, restore two lost checks upstream/main has moved 229 commits and NandhaKishorM#717 landed the batch-2 trace this branch opened for, so what is left here is the part that is still missing: the exporter never checks the graph it wrote. Rebased, and the conflict resolved in upstream's favour. It declares the dynamic axes with `torch.export.Dim` where this branch used `dynamic_axes`; upstream's is the right one and this branch's own comment said why -- torch calls `dynamic_axes` deprecated under `dynamo=True`, and on the release that drops the conversion an export declaring only `dynamic_axes` goes SILENTLY static. The Dims are now derived from `DYNAMIC_AXES` rather than written a second time, so the names the graph declares and the names the tests require cannot drift, and the output entries stay in that table because they are part of the contract a consumer reads off the graph even though `dynamic_shapes` takes inputs only. Verified: every axis still comes out symbolic under its declared name, including `logits[1] = num_markers`. Two checks were lost and are restored. Removing the unfalsifiable session-release check in the last commit deleted `verify/varies the sequence width` and `verify/varies the marker count` with it, because they sat inside the range -- and the mutation that sizes every verified batch identically went green again. Found by re-running the table, not by reading the diff. One new check, for a property no behavioural test can reach: `torch.export.Dim("batch_size")` called twice returns two objects that are neither identical nor equal, so one Dim per occurrence would declare five independent batch symbols -- "each input may have its own batch size" rather than "they must agree". The exported graph names them the same and runs either way, so this pins the specification directly, which is the only place the difference exists. Verified on the new base: tests/test_onnx_export_batch.py 83 passed, 0 failed. test_onnx_quantize 14, test_onnx_batch 59, test_onnx_sort 18, test_onnx_lang_parity 20, test_export_onnx_safety 83, test_evals 83, the ci.yml pytest lane 208 passed / 2 skipped, 68 script suites, ruff and compileall clean. Mutation table 19 of 21 killed; the two survivors are equivalent and recorded as such -- the `del session` cannot be observed from Python (the local dies on return either way), and removing the `if label not in dims` guard changes nothing because the dict lookup already returns one object per name.
…other one The `onnx export (quantization, weight-free)` job failed: `act_logits probabilities differ from PyTorch by 2.53e-02 (> 2e-02) at batch 1`. The tolerance was wrong, not the graph. How much weight-only quantization moves the numbers is a property of the MACHINE, not of the graph: `quantize_dynamic` picks its scales from the weights and the arithmetic underneath differs. Measured on the same tiny model over verify's own sweep -- 9.95e-04 on macOS/arm64, 2.53e-02 on the Linux x86 runner. A 25x spread, and 2e-02 was chosen from the narrow end. `INT8_ATOL` is now 1e-1: headroom over the widest platform observed rather than a fit to the narrowest, and still far under the 0.5 that a coin flip on a two-class output would mean, so a graph whose probabilities have collapsed still fails. Numerical quality is bounded by the fp32 verification at atol=1e-3, which is the check a bad export actually has to get past; this one is for whether the quantized graph still RUNS at batch > 1, since `quantize_model` deletes every `value_info` and rewrites every `MatMul`. I ALSO WITHDREW A CLAIM I CANNOT REPRODUCE. The upper bound was justified as "well under the 0.29 that per-tensor quantization produces". Per-tensor quantization of this graph does not complete at all on this onnxruntime -- `InferenceError: Inferred shape and existing shape differ in dimension 0: (36) vs (256)` -- so there is no per-tensor figure to compare against and the 0.29 is not evidence for anything. The upper bound is now justified by what probabilities mean: they live in [0, 1], so the tolerance has to stay well under the 0.5 at which a two-class answer is a coin flip. The bound is asserted from both sides, and the mutation that matters is the value that broke CI: INT8_ATOL -> 2e-2 is now KILLED, along with 1.1e-3 (below even the local drift), 0.29 and 1e9. The two earlier bounds would both have let a CI-failing value through: `1e-3 < x < 0.1` admitted 1.1e-3, and `5e-3 < x < 5e-2` admitted the 2e-2 this commit removes. Verified: tests/test_onnx_export_batch.py 83 passed, 0 failed; tests/test_onnx_quantize.py 14 passed; mutation table 21 of 23 killed, the 2 survivors equivalent and recorded; ruff clean.
…eeds, since this branch replaces its dummies This branch rebases onto NandhaKishorM#717 and then replaces the inline dummy tensors that PR merged with `example_inputs()`. That is a change to code a maintainer has just reviewed, so the trade needs to be demonstrably additive rather than argued to be. The new trace is richer on three axes -- a padded row, a masked-off trailing marker, and `qtype` varying across the batch, none of which NandhaKishorM#717's `torch.ones`/`torch.zeros` dummies carried, so `src_key_padding_mask`, the `masked_fill` on option logits and two of the three question-type branches were not in the traced graph at all. It is thinner on two: 2 consecutive marker positions against NandhaKishorM#717's 3 spread ones ([1, 5, 9]), at seq_len 16 rather than 17. Marker positions are input values, not shapes, so `torch.export` cannot specialize on them -- but that is reasoning, and this file exists so the export does not have to be reasoned about. The graph is now fed exactly what NandhaKishorM#717's test in tests/test_onnx.py feeds: batch 1 and 3, seq_len 53, five markers at [3, 9, 14, 20, 30], a padded row, `qtype` cycling. It matches PyTorch within 1e-4 at both batches, so tracing this way loses nothing that one covered. What this adds is recorded precisely rather than generously, because I nearly overstated it. The sweep already in this file reaches (5, 64, 7), which dominates (3, 53, 5) on every axis, and both the batch-1 trace and a static marker axis abort the file before these two rows are reached -- so this is not a shape check and no mutation reaches it. The one thing it covers that nothing else does is marker positions that are NOT consecutive: `example_inputs` builds them with `arange(1, num_markers + 1)`, so every other case here feeds 1, 2, 3, ..., while a real checkpoint's markers land wherever the rendered options put them. One thing to note for anyone copying the feed: NandhaKishorM#717's test draws token ids up to 1000 from a real checkpoint's vocabulary. This graph is the hand-built model, whose vocab is 100, so the range is narrowed and nothing else is -- my first attempt kept 1000 and failed with `indices element out of data bounds, idx=475`, which is the fixture's bug and not the graph's. Verified: 89 passed, 0 failed (was 83). Mutation table 21 of 23 killed, the same two equivalent survivors. 0 of 20 runs exit nonzero. ruff clean.
…dhaKishorM#726, record the INT8 spread Addresses @Bruce-Yii's blocker and @NandhaKishorM's hold. Both were right, and the blocker was mine. THE BLOCKER. `os._exit(1 if FAIL else 0)` was at module level with no guard, so `pytest tests/` imported this file, the interpreter left during collection, and pytest reported SUCCESS having executed nothing. Reproduced before fixing, exactly as the review described: python -m pytest tests/test_onnx_export_batch.py -q -> 0 bytes, exit 0 python -m pytest tests/test_runtime_fixes.py -q -> 9 757 bytes, 103 INTERNALERROR lines, exit 3 (it uses sys.exit) `sys.exit` is loud there and a hard exit is silent, which is what made this the one file in the repository that could turn a red run green by leaving. CI never saw it because ci.yml invokes this file as `python tests/...`, which is precisely why it had to be caught by review and not by me. Fixed, and then taken one step further than asked. The guard alone leaves pytest at exit 5, "no tests ran" -- never a false success, but a failing check would not change the status either. So the file also grows a single collected entry point, `test_onnx_export_batch_suite`, which asserts on the `FAIL` list the module body already builds. Measured: pytest, green exit 0, "1 passed", 5 runs out of 5 pytest, one check made to fail exit 1, "AssertionError: 1 of 88 checks failed" pytest, TRACE_BATCH=1 (the NandhaKishorM#695 bug) exit 2, the ONNX error named during collection script mode, green exit 0, "91 passed, 0 failed", 5 runs out of 5 And the guard is now pinned, because a mutation sweep driven by `python tests/<name>.py` cannot see it -- in script mode removing the guard changes nothing. Four source checks assert the hard exit occurs once, is indented, and sits after a `__main__` guard. Writing them took three attempts and the first two could not fail: one counted its own detector line, and one compared against a literal that appears in this file's own source, so it was true with the real guard deleted. The needle is assembled at run time now. Mutation: removing the guard is KILLED by `guard/the file has a __main__ guard`. REBASED ONTO CURRENT MAIN, 210 commits on. NandhaKishorM#726 reverted the declaration to `dynamic_axes` with a compatibility reason this branch did not have: `dynamic_shapes` raises under the TorchScript exporter, the default through torch 2.8, and does not exist before 2.5. So the `_dynamic_shapes` helper this branch added is REMOVED and upstream's `dynamic_axes` is what ships, with that reasoning carried onto the constant. The test that asserted `torch.export.Dim` object identity goes with it; the literal axis-name table is what carries the shared-symbol property now. NandhaKishorM#790's `--per-channel` flag is kept alongside `--no-verify`. INT8_ATOL STAYS AT 1e-1, and the docstring now says why in a form that should stop the next reviewer tightening it. A healthy export's drift, by platform: macOS / arm64, per-tensor (the default since NandhaKishorM#790) 1.65e-03 macOS / arm64, per-channel 9.95e-04 Linux x86 (CI) 2.53e-02 <- 2e-02 failed this job Windows between 5e-02 and 7e-02: a good export FAILS at 5e-02, passes at 7e-02 Two orders of magnitude across three platforms, and 1e-1 is the only value that accepts all of them -- on Windows the headroom is under 2x, not 100x. The Windows figures are @Bruce-Yii's, the macOS ones are mine, the Linux one is this PR's own failing log. Note the per-tensor row: NandhaKishorM#790 changed the default, so the number this tolerance has to accept is no longer the per-channel one it was first fitted to. Verified on the new base: 91 passed, 0 failed in script mode and 1 passed under pytest; mutation table 21 of 22 killed, the survivor being the session release that cannot be observed from Python; test_onnx_quantize 16, test_onnx_batch 59, test_onnx_sort 18, test_onnx_lang_parity 20, test_evals 111, the ci.yml pytest lane 273 passed / 2 skipped, 75 script suites, ruff and compileall clean.
bc6d36d to
1928fc4
Compare
…ation NandhaKishorM#726 reverted Reverting to `dynamic_axes` left a comment two screens down still making the case for `dynamic_shapes` -- 'What decides between them is which one keeps working' -- which is now the opposite of what the code does. Rewritten to say what ships and why, and to keep the one part of the old reasoning that survives: torch does call `dynamic_axes` deprecated under `dynamo=True`, and that is an argument for keeping `verify_batch_dynamic` on by default rather than for moving the declaration ahead of the torch versions this package supports.
…added to avoid, and four checks could not fail
A second adversarial review of the previous commit found 15 findings. All were reproducible and most
were mine. The headline one means the previous commit traded one defect for another.
THE PYTEST ENTRY POINT WAS WORSE THAN NO ENTRY POINT. `os._exit` is what lets the script path skip the
C++ static destruction that torch + onnxruntime + the quantizer abort in. pytest owns the exit, so it
cannot use that escape -- and with the entry point in place the abort came back, now reported as a green
test followed by a dead process:
1 passed, 20 warnings in 10.27s
libc++abi: ... recursive_mutex lock failed: Invalid argument
EXIT=134
Measured over 40 pytest runs: **9 exited 134 after printing `1 passed`** (22.5%). My own check before
pushing was 8 runs, which saw none of them. Script mode over the same 40: 0 nonzero.
So the file now refuses to run inside pytest, with the reason recorded at the skip. `pytest <file>` is
exit 5 / "1 skipped"; `pytest` over a directory is exit 0 with "1 skipped" visible among the others;
script mode -- what `ci.yml` invokes -- is unchanged. Both modes: 0 nonzero in 20 runs each. That is
not a false success (it says skipped) and it is not flaky. Running the checks under pytest for real
needs a subprocess so the native teardown happens where `os._exit` applies; that is a larger change and
is noted rather than made.
FOUR CHECKS COULD NOT FAIL, each proven by mutation:
* `limitation/... marker width is not dynamic` was satisfied by a HISTORICAL RETRACTION. The string
"not dynamic" occurs in the docstring only inside "An earlier revision ... said the width was \"not
dynamic\" ... that is not necessary", so the check passed *because* the docs say its premise was
wrong -- and asserted the opposite of what the code does. Gutting 735 characters of the limitation
paragraph was green; rewording the retraction without changing a fact was red. Now inverted: the
gutting produces 3 FAILs and the rewording is green.
* `graph/every input that shares an axis declares the same symbol name` compared the test's own literal
against another literal. Desyncing the batch symbols in production left it green (caught by three
other checks, never by the one that names the property). Now read from `DYNAMIC_AXES`.
* `verify/compares logits against PyTorch` -- "logits" is a substring of "act_logits", so the act rows
alone satisfied it. My first fix (`" logits "`) also could not fail, because every row carries a
`(raw logits ...)` suffix. Anchored to the `batch N <field>` column now, and dropping `logits` from
the comparison fails it by name.
* `cli/the fp32 check sweeps more than batch 1` read the STUB's `batches=(1, 2, 3)` default, so
shrinking the production default could not fail it. The stub defaults to `None` and the assertion
reads the real signature.
THE SUMMARY WAS NOT GUARANTEED, which is `attempt()`'s whole promise. Three realistic regressions --
TRACE_BATCH=1, a dropped `dynamic_axes`, INT8_ATOL=1e-9 -- were correctly RED but each ended the file on
a bare traceback at roughly check 12 of 96, with 80+ checks unrun and no summary. Wrapping blocks one at
a time did not fix it (I found four separate propagation sites in turn), so the reporter is registered
with `atexit`: an uncaught exception unwinds to the interpreter, which runs it before exiting non-zero.
Now TRACE_BATCH=1 prints `61 passed, 21 failed`, no dynamic_axes `58 passed, 22 failed`, INT8_ATOL=1e-9
`92 passed, 3 failed`.
Also fixed:
* `INT8_ATOL`'s asserted floor was `5e-2`, BELOW the Windows figure the comment itself cites (a good
export fails at 5e-2). `6e-2` passed the check while being a value that rejects a healthy Windows
export. Floor is `7e-2` now, and 6e-2 fails.
* `verify_batch_dynamic` never compared output SHAPES, so a width-divergent graph died on a raw numpy
broadcast `ValueError` instead of the `SystemExit` the docstring promises. Shapes are checked first.
* A leaked session: `_s3` was never released while the comment claimed all were.
* `probabilities()` was almost unpinned -- replacing it with the identity left 90 of 91 green and the one
failure pointed at a comment. The documented property is asserted directly now: a constant added to
every logit in a row must be invisible. With the identity it fails by 1.00e+02.
* Nothing asserted which blocks ran. Simulating the Windows skip path was green at 85 passed with the
whole `--quantize` wiring and the real INT8 verification unexecuted. The block list and a floor are
asserted.
* Three false comment claims withdrawn: "16 of the 32 rows come out entirely padding" (a negative slice
WRAPS -- it is one row, measured); "per-tensor quantization does not complete at all" (it is the
default since NandhaKishorM#790 and this suite exercises it every run, at 1.65e-03); and `sys.exit` "58 times
across tests/" (67). Plus the two files contradicting each other about whether live sessions cause the
abort -- the earlier causation claim is withdrawn.
Verified: 99 passed, 0 failed; 0 of 20 nonzero in script mode and 0 of 20 under pytest; mutation 29 of
30 killed across three specs, the survivor being the session release that cannot be observed from
Python; test_onnx_quantize 16, test_onnx_batch 59, test_onnx_sort 18, test_onnx_lang_parity 20, the
ci.yml pytest lane 273 passed / 2 skipped, 75 script suites, ruff and compileall clean.
Real weights, the 421M english checkpoint exported and run: batch 1/2/3/5/8 at five different sequence
widths and marker counts all run, worst probability difference 1.66e-05 against the 1e-3 tolerance, and
`verify_batch_dynamic` accepts it. Controls on the same real export: the batch axis pinned to 1 is
REFUSED, and a model with one weight scaled 1.25x is REFUSED.
…low Ctrl-C The five handlers the previous commit added to keep a failing block inside the summary were `except BaseException`, which catches `KeyboardInterrupt` as well: a Ctrl-C during an export, a `session.run` or the quantize block would have been recorded as a FAIL and the suite would have carried on to the next block. `SystemExit` has to be named explicitly -- it is what `verify_batch_dynamic` raises and it is not an `Exception` subclass -- so the handlers are `(SystemExit, Exception)`. That keeps the guarantee the previous commit was for and leaves an interrupt alone. Verified: a `KeyboardInterrupt` raised inside a wrapped block exits 130 with the interrupt visible and 0 occurrences of the block's FAIL line, where before it was swallowed. Suite unchanged at 99 passed, 0 failed; ruff clean. Found while answering "did we introduce anything", not by a reviewer -- which is the right way round for once.
@NandhaKishorM Both conditions are addressed on 1. The guardAdded, and the file no longer executes inside pytest. The reason is worth recording: Measured over 40 pytest runs: 9 exited 134 after printing The file therefore declines to run in pytest's process, with that measurement recorded at the skip:
That removes the failure mode the review identified — pytest can no longer report anything about this file without saying what it did — and it is deterministic rather than flaky. Running the checks under pytest for real is possible via a subprocess, so the native teardown happens where 2. Rebased onto current
|
@Bruce-Yii Thank you — a thorough review. All 15 findings reproduced; 14 are fixed on F1 — the entry point could not be keptReproduced at a higher rate than your sample: 9 of 40 pytest runs exited 134 after The conclusion is that no in-process pytest run of this file is safe.
Deterministic, and it states what it did. Your note that The subprocess route is not taken here: it needs the module body behind a guard so pytest's process never opens a session, which is a ~700-line reindent for the same guarantee the skip provides in one block. Say if you would rather have it. F2 — the strongest finding in the reviewThe check was satisfied by a historical retraction of its own premise, and asserted the opposite of the code. Both of your directions now invert:
It now asserts what the docstring states: dynamic from two markers up, baked at exactly one, and the F3, F5, F13 — the dead checks
F4 — wrapping the two blocks was not sufficientThere are four propagation sites, not two: the F6 — the bound contradicted the table it was derived fromVerified your case: F7 — fixed as diagnosedShapes are compared before the subtraction, with a message naming both shapes and the batch, so a width-divergent graph raises the documented F8, F11, F12, F15 — the comment claims
F9, F14 — both fixed, F14 partially
F10 — and one adjacent to itThe entry point is removed, so there is nothing left to pin there. Related: the five handlers F4 required were initially Where it standsThe survivor is the session release in Real weights, the 421M checkpoint: batch 1/2/3/5/8 at five sequence widths and marker counts all run, worst probability difference 1.66e-05 against the 1e-3 tolerance. Controls on that export — batch axis pinned to 1 refused, one weight scaled 1.25x refused. Supporting the probability-space choice: raw The common thread across F2, F3, F5 and F13 is a check reading its expectation from the thing under test. Each rewritten check is now verified against a mutation of the property it names, and the mutation specs are in the repo history alongside the fixes. |
|
@NandhaKishorM PR pending your review. |
Bruce-Yii
left a comment
There was a problem hiding this comment.
I rechecked 713f748 against the blocker I raised earlier. The original os._exit false-green problem is resolved: the exit is now under the __main__ guard, so pytest collection can no longer terminate the process successfully before the checks run. The exporter verification itself is also now doing the load-bearing thing: it runs the written graph in ONNX Runtime at batch > 1 and compares the outputs in probability space, while keeping #726's dynamic_axes direction.
I would consider my original blocker closed on this head.
I am not marking the PR approved yet for one separate reason: Nandha's Oct 1 merge condition also asked for a rebase onto current main / fresh overlap reconciliation. This branch still has merge base 4aa6761, while current main is a4a8921 and has moved substantially since then, including CI/test-suite topology changes. After that rebase, this needs a fresh exact-head integration check rather than carrying this old-head closure forward.
tests/test_portability.py requires every repo read under tests/ to name its codec, so the locale cannot decide it on Windows. The new suite's one read of laya/onnx_agent.py was the only unpinned one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(onnx): make the exporter verify the graph it just wrote
scripts/export_onnx.py+282/-62 ·tests/test_onnx_export_batch.py+724 (new) ·.github/workflows/ci.yml+12/-5 ·docs/evals.md+11Rebased onto
upstream/main@6d942c9.What changed since this PR opened, and what it is now about
The bug this PR was opened for is fixed on
main, by #717. That landed the batch-2 trace withexplicit
torch.export.Dims, and this branch is rebased onto it and adopts its API — the conflict inscripts/export_onnx.pywas resolved in upstream's favour, and this branch's own comment said why:torch calls
dynamic_axesdeprecated underdynamo=True, so on the release that drops theconversion an export declaring only
dynamic_axeswould go silently static.What is left is the part
mainstill does not have, and it is the more important half:Three smaller gaps come with it:
maintraces withdummy_attention_mask = torch.ones(...)anddummy_qtype = torch.zeros(...), sosrc_key_padding_maskis never exercised and themasked_fillon option logits never fires, andonly one question type is traced out of three.
--quantizewrites a graph and says nothing about whether it runs.tests/test_onnx.py, which needs a checkpoint download and is gatedinto its own job. This branch adds 89 checks that run on a hand-built model in the ordinary lane.
The fix
verify_batch_dynamic(model, path, batches=(1, 2, 3))loads the graph in ONNX Runtime, runs it ateach batch with a different sequence width and marker count each time, and compares against
PyTorch in probability space. It raises
SystemExit, because a graph that cannot serve two questionsis not a successful export, and writing one without a word is how a user ends up debugging ONNX
Runtime instead of reading an error.
export_to_onnxcalls it by default.Probabilities, not raw logits, because that is what
ONNXAgent._decode_answersreads and all adecision depends on: the
torch.exportdecompositions move a raw logit far more than they move theanswer — on the 322M multilingual checkpoint raw
act_logitsdiffer by up to 1.2e-02 and rawlogitsby 4.3e-04, while the probabilities agree to 6.1e-05.
example_inputs()replaces the inline dummies and carries what the trace needs to see: a padded row,a masked-off trailing marker, and
qtypevarying across the batch.Replacing #717's dummies is a change to code you just merged, so the trade is measured, not argued.
The new trace is richer on three axes — padding, a masked-off marker, and
qtypevariation, none ofwhich
torch.ones/torch.zeroscarried. It is thinner on two: 2 consecutive marker positions against#717's 3 spread ones (
[1, 5, 9]), at seq_len 16 rather than 17. Marker positions are input values,not shapes, so
torch.exportcannot specialize on them — but rather than rest on that, the graph isnow fed exactly what #717's own test feeds: batch 1 and 3, seq_len 53, five markers at
[3, 9, 14, 20, 30], a padded row,qtypecycling. It matches PyTorch within 1e-4 at both batches.Recorded precisely, because I nearly overstated it: the existing sweep already reaches
(5, 64, 7),which dominates
(3, 53, 5)on every axis, and both the batch-1 trace and a static marker axis abortthe file before those two rows run — so this is not a shape check and no mutation reaches it. The one
thing it uniquely covers is non-consecutive marker positions:
example_inputsbuilds them witharange(1, num_markers + 1), so every other case feeds 1, 2, 3, …, while a real checkpoint's markersland wherever the rendered options put them.
Numbers
atol=1e-3, worst observed drift 9.95e-04INT8_ATOL = 1e-1main's exporter before #717INT8_ATOL is 1e-1, and the reason is worth reading
This PR failed its own CI once on this:
act_logits probabilities differ from PyTorch by 2.53e-02 (> 2e-02). How much weight-only quantization moves the numbers is a property of the machine, not ofthe graph —
quantize_dynamicpicks its scales from the weights and the arithmetic underneathdiffers. Measured on the same tiny model over the same sweep:
2e-02was chosen from the narrow end. The value is now set with headroom over the widest platformobserved, and still far under the 0.5 that a coin flip on a two-class output would mean, so a graph
whose probabilities have collapsed still fails. Numerical quality is bounded by the fp32 check at
1e-3, which is the one a bad export has to get past; this one is for whether the quantized graphstill runs at batch > 1, since
quantize_modeldeletes everyvalue_infoand rewrites everyMatMul.An earlier revision justified the upper bound as "well under the 0.29 that per-tensor quantization
produces". That claim is withdrawn: per-tensor quantization of this graph does not complete at all
on this onnxruntime (
InferenceError: Inferred shape and existing shape differ in dimension 0: (36) vs (256)), so there was no figure to compare against. The bound is now justified by what probabilitiesmean.
Verification
Mutation table: 21 of 23 killed. Every line of the fix has a mutation that turns a check red,
including
TRACE_BATCH = 1(the original bug), dropping each declared axis, removingdynamic_shapesentirely, the verifier accepting any difference, comparing raw logits instead of probabilities,
verifying only one batch, sizing every batch identically, swallowing a session failure, and each of
example_inputs' three trace properties.INT8_ATOL -> 2e-2— the value that actually failed CI — iskilled, along with
1.1e-3,0.29and1e9.The two survivors are equivalent, not unpinned, and are recorded as such:
del sessioncannot be observed from Python: on the success path the local dies on return eitherway, and on the raising path the traceback is cleared before any observer can look.
if label not in dimsguard changes nothing, because the dict lookup already returnsone object per name.
Two things I got wrong in earlier revisions of this branch, corrected here
1. "Releasing the ONNX Runtime sessions fixed an intermittent exit 134." It did not. Re-measured:
3 of 30 runs abort with every session explicitly released, 0 of 30 with the release removed.
The earlier "0 of 30" was luck and I reported it as a fix. Nor is it onnxruntime alone — 1 or 3 bare
sessions opened and held in a fresh process abort 0 of 15 times. What this file does that nothing else
in the suite does is put torch, onnxruntime and onnxruntime's quantizer in one process, and on
macOS that combination aborts during C++ static destruction after the summary prints
0 failed:libc++abi: ... recursive_mutex lock failed. A green report and a red job. Python cannot reorder twonative libraries' destructors, so the suite no longer runs them — it flushes and calls
os._exitwiththe status it computed, after every test has completed and printed. Measured 0 of 40, and the
status still propagates (a deliberately failing check exits 1;
TRACE_BATCH=1exits 1).2. Three of my own checks were weaker than their names claimed. Only the leading axis was checked
for being symbolic, so declaring the marker axis of
logitsstatic left 73 checks green. My first fixfor that read its expectation from
export_onnx.DYNAMIC_AXES— the constant under test — so shrinkingit shrank the expectation too and the check was vacuous; the table is now written out literally.
The verifier's input variety was unpinned, so sizing every batch identically went unnoticed. And
example_inputsdid not have to varyqtype, so tracing a single branch stayed green. I also removeda weakref check I wrote for the session release, because it could not fail — the second unfalsifiable
pin I had written for that line.
Deliberately left out
verify_batch_dynamicsweeps 2-4 markers, not 1: a Python branch in the modelbakes that case out at trace time, and removing it is a separate change. The message says so rather
than implying full coverage.