Skip to content

fix(onnx): export a batch-dynamic graph, and check that it is one - #688

Merged
NandhaKishorM merged 13 commits into
NandhaKishorM:mainfrom
keemsisi:fix/onnx-export-dynamic-batch
Oct 17, 2026
Merged

NandhaKishorM merged 13 commits into
NandhaKishorM:mainfrom
keemsisi:fix/onnx-export-dynamic-batch

Conversation

@keemsisi

@keemsisi keemsisi commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

fix(onnx): make the exporter verify the graph it just wrote

scripts/export_onnx.py +282/-62 · tests/test_onnx_export_batch.py +724 (new) ·
.github/workflows/ci.yml +12/-5 · docs/evals.md +11
Rebased onto upstream/main @ 6d942c9.

What changed since this PR opened, and what it is now about

The bug this PR was opened for is fixed on main, by #717. That landed the batch-2 trace with
explicit torch.export.Dims, and this branch is rebased onto it and adopts its API — the conflict in
scripts/export_onnx.py was resolved in upstream's favour, and this branch's own comment said why:
torch calls dynamic_axes deprecated under dynamo=True, so on the release that drops the
conversion an export declaring only dynamic_axes would go silently static.

What is left is the part main still does not have, and it is the more important half:

The exporter never checks the graph it writes. There is no InferenceSession anywhere in
scripts/export_onnx.py. A batch-1-only graph and a working one declare the same thing — both
report batch_size on the input — so nothing at export time distinguishes them. #695 was found by
tests/test_onnx.py::test_onnx_numerical_parity failing on a clean checkout — that is, by running
the graph, downstream of an export that had reported success.

Three smaller gaps come with it:

  • The trace never sees two of the model's own branches. main traces with
    dummy_attention_mask = torch.ones(...) and dummy_qtype = torch.zeros(...), so
    src_key_padding_mask is never exercised and the masked_fill on option logits never fires, and
    only one question type is traced out of three.
  • No INT8 verification. --quantize writes a graph and says nothing about whether it runs.
  • The one new test lives in tests/test_onnx.py, which needs a checkpoint download and is gated
    into its own job. This branch adds 89 checks that run on a hand-built model in the ordinary lane.

The fix

verify_batch_dynamic(model, path, batches=(1, 2, 3)) loads the graph in ONNX Runtime, runs it at
each batch with a different sequence width and marker count each time, and compares against
PyTorch in probability space. It raises SystemExit, because a graph that cannot serve two questions
is not a successful export, and writing one without a word is how a user ends up debugging ONNX
Runtime instead of reading an error. export_to_onnx calls it by default.

Probabilities, not raw logits, because that is what ONNXAgent._decode_answers reads and all a
decision depends on: the torch.export decompositions move a raw logit far more than they move the
answer — on the 322M multilingual checkpoint raw act_logits differ by up to 1.2e-02 and raw logits
by 4.3e-04, while the probabilities agree to 6.1e-05.

example_inputs() replaces the inline dummies and carries what the trace needs to see: a padded row,
a masked-off trailing marker, and qtype varying across the batch.

Replacing #717's dummies is a change to code you just merged, so the trade is measured, not argued.
The new trace is richer on three axes — padding, a masked-off marker, and qtype variation, none of
which torch.ones/torch.zeros carried. It is thinner on two: 2 consecutive marker positions against
#717's 3 spread ones ([1, 5, 9]), at seq_len 16 rather than 17. Marker positions are input values,
not shapes, so torch.export cannot specialize on them — but rather than rest on that, the graph is
now fed exactly what #717's own test feeds: batch 1 and 3, seq_len 53, five markers at
[3, 9, 14, 20, 30], a padded row, qtype cycling. It matches PyTorch within 1e-4 at both batches.

Recorded precisely, because I nearly overstated it: the existing sweep already reaches (5, 64, 7),
which dominates (3, 53, 5) on every axis, and both the batch-1 trace and a static marker axis abort
the file before those two rows run — so this is not a shape check and no mutation reaches it. The one
thing it uniquely covers is non-consecutive marker positions: example_inputs builds them with
arange(1, num_markers + 1), so every other case feeds 1, 2, 3, …, while a real checkpoint's markers
land wherever the rendered options put them.

Numbers

fp32 verification tolerance atol=1e-3, worst observed drift 9.95e-04
INT8 tolerance INT8_ATOL = 1e-1
the same export with main's exporter before #717 ran at batch 1, failed at 2, 3, 5, 8

INT8_ATOL is 1e-1, and the reason is worth reading

This PR failed its own CI once on this: act_logits probabilities differ from PyTorch by 2.53e-02 (> 2e-02). How much weight-only quantization moves the numbers is a property of the machine, not of
the graph
— quantize_dynamic picks its scales from the weights and the arithmetic underneath
differs. Measured on the same tiny model over the same sweep:

macOS / arm64      9.95e-04
Linux x86 (CI)     2.53e-02      <- 25x wider

2e-02 was chosen from the narrow end. The value is now set with headroom over the widest platform
observed, and still far under the 0.5 that a coin flip on a two-class output would mean, so a graph
whose probabilities have collapsed still fails. Numerical quality is bounded by the fp32 check at
1e-3, which is the one a bad export has to get past; this one is for whether the quantized graph
still runs at batch > 1, since quantize_model deletes every value_info and rewrites every
MatMul.

An earlier revision justified the upper bound as "well under the 0.29 that per-tensor quantization
produces". That claim is withdrawn: per-tensor quantization of this graph does not complete at all
on this onnxruntime (InferenceError: Inferred shape and existing shape differ in dimension 0: (36) vs (256)), so there was no figure to compare against. The bound is now justified by what probabilities
mean.

Verification

tests/test_onnx_export_batch.py        89 passed, 0 failed     (new file)
tests/test_onnx_quantize.py            14 passed
tests/test_onnx_batch.py               59 passed
tests/test_onnx_sort.py                18 passed
tests/test_onnx_lang_parity.py         20 passed
tests/test_export_onnx_safety.py       83 passed
tests/test_evals.py                    83 passed
the ci.yml pytest lane                208 passed, 2 skipped    (unchanged from main)
script-style suites from ci.yml         68 ok, 0 failed
ruff + compileall                       clean

Mutation table: 21 of 23 killed. Every line of the fix has a mutation that turns a check red,
including TRACE_BATCH = 1 (the original bug), dropping each declared axis, removing dynamic_shapes
entirely, the verifier accepting any difference, comparing raw logits instead of probabilities,
verifying only one batch, sizing every batch identically, swallowing a session failure, and each of
example_inputs' three trace properties. INT8_ATOL -> 2e-2 — the value that actually failed CI — is
killed, along with 1.1e-3, 0.29 and 1e9.

The two survivors are equivalent, not unpinned, and are recorded as such:

  • del session cannot be observed from Python: on the success path the local dies on return either
    way, and on the raising path the traceback is cleared before any observer can look.
  • Removing the if label not in dims guard changes nothing, because the dict lookup already returns
    one object per name.

Two things I got wrong in earlier revisions of this branch, corrected here

1. "Releasing the ONNX Runtime sessions fixed an intermittent exit 134." It did not. Re-measured:
3 of 30 runs abort with every session explicitly released, 0 of 30 with the release removed.
The earlier "0 of 30" was luck and I reported it as a fix. Nor is it onnxruntime alone — 1 or 3 bare
sessions opened and held in a fresh process abort 0 of 15 times. What this file does that nothing else
in the suite does is put torch, onnxruntime and onnxruntime's quantizer in one process, and on
macOS that combination aborts during C++ static destruction after the summary prints 0 failed:
libc++abi: ... recursive_mutex lock failed. A green report and a red job. Python cannot reorder two
native libraries' destructors, so the suite no longer runs them — it flushes and calls os._exit with
the status it computed, after every test has completed and printed. Measured 0 of 40, and the
status still propagates (a deliberately failing check exits 1; TRACE_BATCH=1 exits 1).

2. Three of my own checks were weaker than their names claimed. Only the leading axis was checked
for being symbolic, so declaring the marker axis of logits static left 73 checks green. My first fix
for that read its expectation from export_onnx.DYNAMIC_AXES — the constant under test — so shrinking
it shrank the expectation too and the check was vacuous; the table is now written out literally.
The verifier's input variety was unpinned, so sizing every batch identically went unnoticed. And
example_inputs did not have to vary qtype, so tracing a single branch stayed green. I also removed
a weakref check I wrote for the session release, because it could not fail — the second unfalsifiable
pin I had written for that line.

Deliberately left out

  • Width-1 markers. verify_batch_dynamic sweeps 2-4 markers, not 1: a Python branch in the model
    bakes that case out at trace time, and removing it is a separate change. The message says so rather
    than implying full coverage.
  • A real-checkpoint export in CI. It needs a download; the hand-built model covers the contract.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The load-bearing claim holds up — I reproduced it by mutation — but there is one blocker, and it is the last line of the new test file.

The blocker: os._exit is unguarded, and pytest can be made to report success having run nothing

tests/test_onnx_export_batch.py:724 is os._exit(1 if FAIL else 0), at module level. There is no if __name__ == "__main__": guard anywhere in the file — the three __main__ hits are all inside comments, one of which (:666) explains that a CLI body moved into a function precisely so a check can read main's source rather than the guard.

The file matches pytest's test_*.py glob, so a plain pytest run imports it and the os._exit fires during collection. Measured on this branch, same interpreter, same tree:

command output exit
python -m pytest tests/test_runtime_fixes.py -q 9007 bytes — 60 passed, then INTERNALERROR> Traceback 3
python -m pytest tests/test_onnx_export_batch.py -q 0 bytes 0

The control is the comparison that matters. test_runtime_fixes.py ends in sys.exit(...), and pytest turns that into a loud ~30-line INTERNALERROR with exit 3. os._exit(0) produces no output at all and a success status — the file's own 83 checks never print, and pytest reports green having executed none of them.

This would be the only such suite in the repository: sys.exit appears 58 times across tests/*.py, os._exit zero.

It does not currently break CI, and I want to be precise about why, because it changes how urgent this is: ci.yml:232 runs it as python tests/test_onnx_export_batch.py (script mode, where the guard would not matter), and every pytest invocation in ci.yml:193/333/387 and release.yml names its files explicitly, with ci.yml:335-337 citing #374 for that choice. No bare pytest, no testpaths/addopts, no conftest.py. So it is a latent trap, not an active break. The exposure is a contributor running pytest tests/ locally to check their own change, who gets a green that means nothing — and this repo already has form here.

Minimal fix, which keeps the macOS workaround for the invocation CI actually uses:

if __name__ == "__main__":
    sys.stdout.flush()
    sys.stderr.flush()
    os._exit(1 if FAIL else 0)

The gate itself works

I mutated TRACE_BATCH = 2 to 1 — the original #695 bug — and the suite goes red with the consequence named:

ONNXRuntimeError: INVALID_ARGUMENT : Got invalid dimensions for input: marker_pos
  index: 0 Got: 3 Expected: 1     (exit 1)

Restored, the same tree is 83 passed, 0 failed, exit 0, summary printed. Separately, the SystemExit path you added is covered on its own rather than only as a side effect: :472-513 asserts verify_batch_dynamic raises SystemExit in three separate cases, and :546 pins the shipped default via inspect.signature(...).parameters["atol"].default == 1e-3 so the expectation cannot drift with the constant it is checking.

On INT8_ATOL = 1e-1 — a guardrail, because the tightening looks tempting and is wrong

I spent time trying to show this was too loose, and it is not, at least not for the reason it first appears. A good int8 graph on this machine already drifts further than the PR's own CI log suggests. Tightening the constant until the healthy export fails:

INT8_ATOL result on this tree
1e-2 exit 1 — a good graph fails
3e-2 exit 1
5e-2 exit 1
7e-2 exit 0
1e-1 (shipped) exit 0 — 83 passed, 0 failed

So on Windows here a healthy export needs somewhere between 5e-2 and 7e-2, against the 9.95e-04 you measured on macOS arm64 and the 2.53e-02 that failed your CI on Linux x86. That is a spread of roughly two orders of magnitude across three platforms, and 1e-1 is the only value in the set that covers all of them. The headroom is not 100x — on the platform that matters most here it is under 2x.

Worth stating in the docstring, because the obvious "reviewer move" here is to tighten it: any value at or below 5e-2 will reject valid exports on at least one of the three platforms, and the resulting failure would look like a broken quantizer rather than a too-tight tolerance. If you want the gate to carry more numerical weight than "does it still run at batch > 1", the place to add it is a per-platform floor recorded in the test, not a lower constant.

Requesting changes for the os._exit guard only. Everything else I checked holds.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding on the os._exit blocker, which has not been addressed on the current head.

The finding is the kind I will not merge past, because of what it does rather than what it is: tests/test_onnx_export_batch.py ends in a module-level os._exit(1 if FAIL else 0) with no __main__ guard, so under pytest the process exits during collection and the run reports success having asserted nothing. A test file that can make the suite green by leaving is worse than a missing test, and this one is the only thing standing behind the export.

The change itself is worth having and your framing of it is honest: the original bug is fixed on main by #717, you rebased onto it and adopted its API, and what is left is the exporter verifying the graph it just wrote. That is a real gap, and #726 landed in 0.3.23 moving the declaration back to dynamic_axes with an assertion that the symbolic axes are declared by name, so there is now some overlap to reconcile too.

Add the if __name__ == "__main__": guard so pytest collects and runs the file normally, rebase onto current main so the scripts/export_onnx.py overlap with #726 is resolved in one place, and I will take it.

`scripts/export_onnx.py` traced the model with a single row while declaring a
`batch_size` axis. `torch.export` specializes every example dimension whose extent is
1, so the graph it wrote ran at batch 1 and failed inside ONNX Runtime at batch 2
("Attempting to broadcast an axis by a dimension other than 1"). `ONNXAgent` builds one
row per question, so every two-question `predict`/`system_one` call, and
`predict_batch`/`predict_long` at any size, was unservable -- while the script printed
"Successfully exported" and torch raised the problem only as a UserWarning.

Trace with two rows, and stop claiming success without checking: the export now runs the
graph in ONNX Runtime at batch 1, 2 and 3 and compares it with PyTorch, raising rather
than writing a graph that cannot serve two questions. Probabilities rather than raw
logits, because the decompositions move a raw `act_logits` value by up to 1.2e-02 on the
322M multilingual checkpoint while the probabilities agree to 6.1e-05.

Measured on that checkpoint: the five calls that failed go to 5/5, and all nine answers
over three states x choice/score/noul are identical to the torch `Agent`, max |delta|
0.00000 across 57 numeric fields.

A declared axis proved nothing here -- the broken graph declared `batch_size` too -- so
`tests/test_onnx_export_batch.py` runs the graph instead of reading its shapes. It
exports a tiny stand-in model through the same helper at batch 1, 2, 3 and 5, so it
needs no checkpoint and no network and fits the existing weight-free `onnx-export` job.
That job moves to the declared `[onnx]` extra -- which already exists upstream as
["onnx", "onnxruntime", "onnxscript"], so nothing is lost and `onnxscript` is gained, which
`torch.onnx.export` needs for the path it takes by default. Setting `TRACE_BATCH` back to 1 turns 9
of the suite's 48 checks red.

Re-proved from scratch: exporting the same tiny model with a one-row and a two-row example produces
graphs that declare the IDENTICAL `['batch_size', 'seq_len']` shape, and only the two-row one runs at
batch 2 and 3. Reading declared shapes is precisely the check that passes the broken graph. And the
batch dimension really is the question count -- `collate_items` flattens one item per question per
state, so 2 questions is batch 2 and 2 questions over 2 states is batch 4.

Verifying by default changed the command's contract in three ways the first draft missed, each now
guarded and pinned. `onnxruntime` became a hard requirement of a command that never needed it --
`torch.onnx.export` does not touch it -- so an install with onnx + onnxscript printed "Successfully
exported" and then died on a bare ModuleNotFoundError; the import is guarded and names both
`pip install laya[onnx]` and `--no-verify`. `--quantize` ran after the function returned, so the INT8
copy that docs/evals.md tells people to deploy was written unverified, which is precisely the "runs but
returns different numbers" case this verifier is for; `export_to_onnx` now returns the model so the
quantized graph is checked too, at its own measured tolerance (weight-only INT8 drifts up to 9.95e-04
across the swept shapes, already at the fp32 atol, so INT8_ATOL = 2e-2). And the success message
claimed "runs at batch > 1 and matches PyTorch" while the sweep never used num_markers=1: marker width
is not dynamic, because `DecisionModel.forward` branches on `p.size(-1) >= 2` in Python, so a graph
traced at width 2 raises an ONNX Runtime TopK error at width 1 -- reachable, since a one-criterion
`score` question is accepted and yields one marker. That limitation predates this change and making
width dynamic means removing a branch from the model, so the message, the docstring and docs/evals.md
now state it instead of overclaiming.

Four checks were added because a mutation showed the verifier itself was unpinned: flipping `verify`
to default False, removing the call to it, disabling its numeric comparison and widening its
tolerance to 1e9 all left the suite green. The numeric check is now pinned by exporting one model and
verifying a perturbed copy against that graph -- a graph that runs at every batch and quietly returns
different values, which is the only case the probability comparison exists for. The perturbation is
non-uniform and on a weight rather than a final bias, because a uniform shift of the last bias moves
no probability at all: softmax is shift-invariant, the blind spot the verifier's own docstring
records.

Three smaller things the export needed on the way, each of which cost a confusing failure first:

  * `example_inputs` reads `TRACE_BATCH` at call time instead of binding it as a signature
    default, so `export_onnx.TRACE_BATCH = 1` actually reaches it -- the one experiment anybody
    revisiting this bug will run, and it silently did nothing before.
  * `marker_pos` indexes the sequence axis, so `num_markers >= seq_len` made the head's marker
    gather read out of bounds: `IndexError` in torch, and in ONNX Runtime the much less obvious
    "GatherElements op: Out of range value in index tensor". It now raises with the reason.
  * the padding written per row was `attention_mask[row, seq_len - row:]`, which reaches 0 at
    `row == seq_len` and goes negative past it, handing the model an all-padding row for any
    `batch > seq_len`. Torch and ONNX Runtime legitimately disagree on such a row, and the caller
    would read that as an export failure. Clamped with `max(1, ...)`, so every row keeps at least one
    real token. Measured at batch 32, seq_len 16, where 16 of the 32 rows come out entirely padding:
    2.76e-01 on `act_logits` and 1.12e-01 on `logits`, against 2.38e-07 and 1.19e-07 once clamped.

The `dynamic_axes` call now carries the reason it is still there rather than the `dynamic_shapes`
torch prefers: measured, `dynamic_shapes` with an explicit `torch.export.Dim("batch")` from a
one-row example still exports `input_ids` as `[1, "seq"]` and still fails at batch 2, so it is
`TRACE_BATCH` and not the axis spec that fixes this. `dynamic_axes` is nonetheless load-bearing --
drop it and every axis exports static -- and torch already calls it deprecated, so on the release
that removes it this export goes *silently* static: a graph that serves one question, no error,
exactly the failure this file just had. `verify_batch_dynamic` is what would catch that day, which
is why it runs by default and compares numbers rather than only shapes.
…path; correct the width claim

From an adversarial review. The functional change is sound and the review confirmed it
independently: `TRACE_BATCH = 2` is sufficient (batch 1..128 x marker width 2..16 all match PyTorch
to 3e-07, and re-tracing at 3 markers is identical), `verify_batch_dynamic` genuinely rejects a real
batch-1 trace, the INT8 path really does preserve batch dynamism, and the 9.95e-04 / 1e-3 / 2e-2
numbers in the comments reproduce exactly. Every HIGH finding was a COVERAGE failure: 28 mutants
survived both suites, concentrated on the three things that actually reach a user.

1. THE I/O NAME CONTRACT. `laya/onnx_agent.py` builds its feed from hardcoded strings and calls
   `session.run(["logits", "act_logits"])`. Every check here built its feed from `INPUT_NAMES` too,
   so a permutation was self-consistent: swapping the two `OUTPUT_NAMES` exported a graph whose
   `logits` is the act head, and `ONNXAgent` then reads the act head as the option scores -- silently
   wrong answers on every request, no error anywhere. Swapping `input_ids`/`attention_mask` (same
   dtype and shape, so ORT accepts it) gave a max deviation of 5.27 from PyTorch. Now asserted
   against literals, against the strings `onnx_agent.py` actually contains, and by running the graph
   BY NAME so the shapes each name carries are pinned (`logits` is (batch, num_markers),
   `act_logits` is (batch, 2)).

2. THE QUANTIZE PATH -- the one `docs/evals.md` tells people to deploy -- was never exercised on a
   real torch export. `tests/test_onnx_quantize.py` builds a one-node MatMul graph with no
   intermediate `value_info`, so it cannot reach `del model.graph.value_info[:]`; this suite exported
   a real graph but never quantized it. Deleting that one line left both suites green and made
   `python scripts/export_onnx.py --quantize` crash outright with
   `InferenceError: Inferred shape and existing shape differ in dimension 0`. The suite now quantizes
   the graph it already exports, verifies it at the INT8 tolerance, and checks it kept a symbolic
   batch axis.

3. THE CLI WIRING was pinned by grepping `__main__` for substrings, and the stubs recorded only THAT
   a function was called. So `batches=(1,)` on either call, `--no-verify` flipped to `store_false`,
   `verify=False`, `atol=1e9` on the INT8 check, and quantizing over the fp32 output all shipped a
   success message while checking nothing. The CLI body is now `main(argv=None)` -- the smallest
   change that makes it callable -- and a test drives it with a stubbed `Agent` and asserts the
   `batches`, the `atol` and the output path it actually passes.

4. `logits` WAS UNPROTECTED by the drift check. The perturbation picks the last 2-D parameter, which
   lives in `act_head`, so only `act_logits` moved; skipping `logits` in the comparison loop left the
   suite green. No perturbation isolates `logits` in this model -- the act head consumes the scorer's
   output, so every weight reaches both, and whichever output the loop still compares catches the
   drift. Pinned instead on what the verifier reports: both names, at every batch it sweeps.

AND ONE CLAIM THAT WAS MEASURABLY FALSE. Three places said marker width "is not dynamic" and is
"fixed at the traced value". Measured on a graph traced at 2 markers: widths 2, 3, 4, 5 and 8 all run
and match. Only width 1 fails, via `TopK: k argument [2] should not be greater than specified axis
dim value [1]`, because `DecisionModel.forward` branches on `p.size(-1) >= 2` in Python and that
BRANCH is what the trace bakes in. As written a user would conclude they must re-export per criteria
count. Corrected in the docstring, in the success message, and in docs/evals.md, which now also notes
that `collate_items` sizes markers to the batch maximum, so a one-criterion question only hits it when
every item in the batch has exactly one marker.

Verified: tests/test_onnx_export_batch.py 69 passed (was 48), tests/test_onnx_quantize.py 14 passed,
all 61 ci.yml script suites, ruff and compileall clean. All 9 mutants from the review now killed, 0
survivors. Re-exported the real 322M multilingual checkpoint with this branch's script: verification
passes at batch 1, 2, 3 and the graph runs at 1, 2, 3, 5 and 8 -- the bug this PR fixes stays fixed.
`tests/test_evals.py::test_docs_and_the_cli_name_the_same_flags` was already red on this branch
before today's changes, and I missed it: my earlier CI sweep across the five PRs checked
`dependency CVEs` and `tests (windows)` and not this job. It passes on `upstream/main` and on every
other branch.

The test collects every `--flag` in `docs/evals.md`, skipping lines that mention `export_onnx.py`
because "lines that run another script teach that script's flags, not these", and asserts the rest
are registered by an `evals` subcommand. Upstream only ever names `--quantize` on lines that also
name `scripts/export_onnx.py`. The paragraph this branch added introduced `--quantize` and
`--no-verify` on lines that did not, so both were collected as `evals` flags and the assertion failed
with "the page teaches ['--no-verify', '--quantize'], which no subcommand registers".

Both mentions now name `scripts/export_onnx.py`, which is what they are flags of, and the marker-width
explanation moves into its own paragraph so the prose still reads sensibly.

tests/test_evals.py: 58 passed. test_evals_api, test_evals_onnx (20) and
tests/test_onnx_export_batch.py (69) green. All 61 ci.yml script suites pass. ruff clean.
…zation on Windows

Two CI failures, both from the checks added in the previous commit.

1. `onnx export` went red with `logits max abs prob diff 1.03e-02` against the 1e-3 default. The
   model IS seeded, so this is not run-to-run noise -- it is cross-MACHINE: the same graph and the
   same weights drift 4.69e-04 here and 1.03e-02 on a GitHub runner, because a 32-dim random model
   has near-tied logits and softmax amplifies the difference between one machine's fp32 kernels and
   another's. 1e-3 sits inside that gap, so the suite failed by hardware. The previous commit did
   not introduce the fragility -- an unperturbed `verify_batch_dynamic` call was already there -- it
   added a second call and a second chance for it to fire.

   The tiny-model checks now pass an explicit `TINY_ATOL = 5e-2`. The 1e-3 DEFAULT is untouched,
   which matters: it is what the 322M checkpoint is held to, and it passes at 6.14e-05. Measured
   separation on the tiny model: unperturbed 8.9e-08 (and 1.03e-02 at its worst across machines),
   perturbed 6.4e-01. So 5e-2 is 5x above the worst drift observed and 13x below the signal the
   check exists to catch. Re-ran every mutant afterwards: all 9 still killed, 0 survivors, so the
   looser tolerance did not make the guards vacuous.

2. `tests (windows)` died with an illegal instruction (exit 132) inside onnxruntime, quantizing the
   real torch-exported transformer graph the new check builds. It is a hard crash, not an exception,
   so it cannot be caught -- `tests/test_onnx_quantize.py` survives on Windows only because its
   graph is a single MatMul node. Both the quantize check and the CLI check (which exports and
   quantizes) now skip on Windows and say so, leaving them gated by the
   `onnx export (quantization, weight-free)` job on Linux, which is the job the documented
   `--quantize` invocation has to keep green.

tests/test_onnx_export_batch.py: 69 passed. tests/test_onnx_quantize.py: 14 passed. All 61 ci.yml
script suites. ruff clean. Mutation sweep: 0 of 9 survive.
…rances that actually ship

Re-review of fcf7bb5 found four problems with the checks the previous two commits added. All four
were mine.

1. THE SUITE EXITED 134 INTERMITTENTLY, AFTER PRINTING "0 failed". The new checks open ONNX Runtime
   sessions -- one per `verify_batch_dynamic` call, plus two the name-contract and quantize blocks
   hold -- and leaving them alive means interpreter shutdown dies with
   `libc++abi: recursive_mutex lock failed`. A red CI job with a green summary, on the exact command
   .github/workflows/ci.yml runs in the `onnx-export` job. Measured here: 1 of 14 at fcf7bb5, and a
   reviewer saw 5 of 20; `verify_batch_dynamic` now releases its session in a `finally`, the two
   block-local sessions are deleted where they are used, and the three long-lived module-level ones
   are released before the file ends. Re-measured: 0 of 30. I had earlier seen this message once and
   called it a transient teardown artifact; it was not.

2. THE TOLERANCE THAT SHIPS WAS PINNED BY NOTHING. `export_to_onnx` calls `verify_batch_dynamic` with
   no `atol`, so the default is what every real export is held to -- and changing it to 1e9 left all
   three suites green. Its value is asserted now, and so is that it REJECTS a calibrated error: one
   scorer weight scaled by 1.25, not the linspace(0, 5) sledgehammer that proves nothing about a
   realistic corruption.

3. THE TINY_ATOL COMMENT WAS FALSE. It said the 1e-3 default "is what bounds quality and it is
   asserted separately"; nothing asserted it. Worse, 5e-2 is blind to a scorer weight scaled by 3.0,
   which the default catches from 1.25 upward -- so the four checks using it demonstrate that the
   graph RUNS at several batches, not that the verifier would catch a bad one. The comment now says
   exactly that, and the sensitivity proof lives at the shipping tolerance where it belongs.

4. `INT8_ATOL` WAS BOUNDED ONLY FROM BELOW. `> 1e-3` let 1e9 through, and my first attempt at a
   two-sided bound (`1e-3 < x < 0.1`) let 1.1e-3 through -- which is BELOW the 9.95e-04 drift the
   tolerance has to accept, so a real quantized graph would have started failing. Now
   `5e-3 < INT8_ATOL < 5e-2`: above the drift it must accept, well under the 0.29 that per-tensor
   quantization produces.

I also removed a source-text check I wrote for the session release: the mutation that disables the
release leaves the string in the file, so it could not distinguish and was worse than nothing. The
release is evidenced by the flake rate, which is what it actually protects.

Verified: tests/test_onnx_export_batch.py 73 passed, 0 failed, and 0 of 30 runs exit nonzero.
tests/test_onnx_quantize.py 14 passed, tests/test_evals.py 58 passed, all 61 ci.yml script suites,
ruff and compileall clean. Tolerance mutants: default atol -> 1e9, INT8_ATOL -> 1.1e-3, INT8_ATOL ->
0.29 and disabling the diff comparison are all killed.
…checks were weaker than claimed

The previous commit claimed to have fixed an intermittent exit 134 by releasing ONNX Runtime
sessions. Re-measuring says that is wrong, so it is corrected here rather than left standing.

THE SESSION RELEASE IS NOT WHAT FIXED IT. With every session explicitly released: 3 of 30 runs
abort. With the release in `verify_batch_dynamic` removed: 0 of 30. The earlier "0 of 30" was luck,
not a fix, and I reported it as one. Nor is it onnxruntime by itself -- 1 or 3 bare sessions opened
and held in a fresh process abort 0 of 15 times. What this file actually does that nothing else in
the suite does is put torch, onnxruntime and onnxruntime's quantizer in one process, and on macOS
that combination aborts during C++ static destruction, after the summary has printed "0 failed":
`libc++abi: ... recursive_mutex lock failed: Invalid argument`. A green report and a red job.

Python cannot reorder two native libraries' destructors, so the suite no longer runs them: it
flushes and calls `os._exit` with the status it computed. Every test has completed and its result is
printed by then. Measured: 0 of 40 runs abort, against 4 of 42 exiting normally, and the status still
propagates -- a deliberately failing check exits 1, and TRACE_BATCH=1 still exits 1.

Three of the checks added in the last two commits were weaker than their names claimed:

1. ONLY THE LEADING AXIS WAS CHECKED FOR BEING SYMBOLIC. Declaring the marker axis of `logits`
   static left 73 checks green, so the graph could tell a consumer the option count is fixed at the
   traced width. Now every axis is checked, against a table written out LITERALLY -- my first
   attempt read the expectation from `export_onnx.DYNAMIC_AXES`, the constant under test, so
   shrinking it shrank the expectation too and the check was vacuous.

2. THE VERIFIER'S INPUT VARIETY WAS UNPINNED. `verify_batch_dynamic` prints "batch 1, 2 and 3 with
   2-4 markers", and sizing every batch with the same shapes left the suite green -- the sequence and
   marker axes were then never exercised at more than one width, so a marker-static graph would have
   verified clean. The shapes it asks for are now spied on and asserted distinct.

3. `example_inputs` DID NOT HAVE TO VARY `qtype`. Flattening it to one value traced a single branch
   of `DecisionModel.forward` with 73 checks green.

I also removed a weakref check I wrote for the session release: on the success path the local dies
on return either way, and on the raising path the traceback is cleared before any observer can look,
so it could not fail. That is the second unfalsifiable pin I have written for this line; the line
stays as hygiene, credited with nothing.

Verified: 78 passed, 0 failed, 0 of 40 nonzero exits. Mutation table re-run at this head: 19 of 19
killed, including all three gaps above. test_onnx_quantize 14 passed, test_evals 58 passed, ruff and
compileall clean.
….Dim, restore two lost checks

upstream/main has moved 229 commits and NandhaKishorM#717 landed the batch-2 trace this branch opened for, so
what is left here is the part that is still missing: the exporter never checks the graph it wrote.

Rebased, and the conflict resolved in upstream's favour. It declares the dynamic axes with
`torch.export.Dim` where this branch used `dynamic_axes`; upstream's is the right one and this
branch's own comment said why -- torch calls `dynamic_axes` deprecated under `dynamo=True`, and on
the release that drops the conversion an export declaring only `dynamic_axes` goes SILENTLY static.
The Dims are now derived from `DYNAMIC_AXES` rather than written a second time, so the names the
graph declares and the names the tests require cannot drift, and the output entries stay in that
table because they are part of the contract a consumer reads off the graph even though
`dynamic_shapes` takes inputs only. Verified: every axis still comes out symbolic under its declared
name, including `logits[1] = num_markers`.

Two checks were lost and are restored. Removing the unfalsifiable session-release check in the last
commit deleted `verify/varies the sequence width` and `verify/varies the marker count` with it,
because they sat inside the range -- and the mutation that sizes every verified batch identically
went green again. Found by re-running the table, not by reading the diff.

One new check, for a property no behavioural test can reach: `torch.export.Dim("batch_size")` called
twice returns two objects that are neither identical nor equal, so one Dim per occurrence would
declare five independent batch symbols -- "each input may have its own batch size" rather than "they
must agree". The exported graph names them the same and runs either way, so this pins the
specification directly, which is the only place the difference exists.

Verified on the new base: tests/test_onnx_export_batch.py 83 passed, 0 failed. test_onnx_quantize 14,
test_onnx_batch 59, test_onnx_sort 18, test_onnx_lang_parity 20, test_export_onnx_safety 83,
test_evals 83, the ci.yml pytest lane 208 passed / 2 skipped, 68 script suites, ruff and compileall
clean. Mutation table 19 of 21 killed; the two survivors are equivalent and recorded as such -- the
`del session` cannot be observed from Python (the local dies on return either way), and removing the
`if label not in dims` guard changes nothing because the dict lookup already returns one object per
name.
…other one

The `onnx export (quantization, weight-free)` job failed: `act_logits probabilities differ from
PyTorch by 2.53e-02 (> 2e-02) at batch 1`. The tolerance was wrong, not the graph.

How much weight-only quantization moves the numbers is a property of the MACHINE, not of the graph:
`quantize_dynamic` picks its scales from the weights and the arithmetic underneath differs. Measured
on the same tiny model over verify's own sweep -- 9.95e-04 on macOS/arm64, 2.53e-02 on the Linux x86
runner. A 25x spread, and 2e-02 was chosen from the narrow end. `INT8_ATOL` is now 1e-1: headroom
over the widest platform observed rather than a fit to the narrowest, and still far under the 0.5
that a coin flip on a two-class output would mean, so a graph whose probabilities have collapsed
still fails. Numerical quality is bounded by the fp32 verification at atol=1e-3, which is the check a
bad export actually has to get past; this one is for whether the quantized graph still RUNS at
batch > 1, since `quantize_model` deletes every `value_info` and rewrites every `MatMul`.

I ALSO WITHDREW A CLAIM I CANNOT REPRODUCE. The upper bound was justified as "well under the 0.29
that per-tensor quantization produces". Per-tensor quantization of this graph does not complete at
all on this onnxruntime -- `InferenceError: Inferred shape and existing shape differ in dimension 0:
(36) vs (256)` -- so there is no per-tensor figure to compare against and the 0.29 is not evidence
for anything. The upper bound is now justified by what probabilities mean: they live in [0, 1], so
the tolerance has to stay well under the 0.5 at which a two-class answer is a coin flip.

The bound is asserted from both sides, and the mutation that matters is the value that broke CI:
INT8_ATOL -> 2e-2 is now KILLED, along with 1.1e-3 (below even the local drift), 0.29 and 1e9. The
two earlier bounds would both have let a CI-failing value through: `1e-3 < x < 0.1` admitted 1.1e-3,
and `5e-3 < x < 5e-2` admitted the 2e-2 this commit removes.

Verified: tests/test_onnx_export_batch.py 83 passed, 0 failed; tests/test_onnx_quantize.py 14 passed;
mutation table 21 of 23 killed, the 2 survivors equivalent and recorded; ruff clean.
…eeds, since this branch replaces its dummies

This branch rebases onto NandhaKishorM#717 and then replaces the inline dummy tensors that PR merged with
`example_inputs()`. That is a change to code a maintainer has just reviewed, so the trade needs to be
demonstrably additive rather than argued to be.

The new trace is richer on three axes -- a padded row, a masked-off trailing marker, and `qtype`
varying across the batch, none of which NandhaKishorM#717's `torch.ones`/`torch.zeros` dummies carried, so
`src_key_padding_mask`, the `masked_fill` on option logits and two of the three question-type branches
were not in the traced graph at all. It is thinner on two: 2 consecutive marker positions against
NandhaKishorM#717's 3 spread ones ([1, 5, 9]), at seq_len 16 rather than 17.

Marker positions are input values, not shapes, so `torch.export` cannot specialize on them -- but that
is reasoning, and this file exists so the export does not have to be reasoned about. The graph is now
fed exactly what NandhaKishorM#717's test in tests/test_onnx.py feeds: batch 1 and 3, seq_len 53, five markers at
[3, 9, 14, 20, 30], a padded row, `qtype` cycling. It matches PyTorch within 1e-4 at both batches, so
tracing this way loses nothing that one covered.

What this adds is recorded precisely rather than generously, because I nearly overstated it. The sweep
already in this file reaches (5, 64, 7), which dominates (3, 53, 5) on every axis, and both the
batch-1 trace and a static marker axis abort the file before these two rows are reached -- so this is
not a shape check and no mutation reaches it. The one thing it covers that nothing else does is marker
positions that are NOT consecutive: `example_inputs` builds them with `arange(1, num_markers + 1)`, so
every other case here feeds 1, 2, 3, ..., while a real checkpoint's markers land wherever the rendered
options put them.

One thing to note for anyone copying the feed: NandhaKishorM#717's test draws token ids up to 1000 from a real
checkpoint's vocabulary. This graph is the hand-built model, whose vocab is 100, so the range is
narrowed and nothing else is -- my first attempt kept 1000 and failed with `indices element out of data
bounds, idx=475`, which is the fixture's bug and not the graph's.

Verified: 89 passed, 0 failed (was 83). Mutation table 21 of 23 killed, the same two equivalent
survivors. 0 of 20 runs exit nonzero. ruff clean.
…dhaKishorM#726, record the INT8 spread

Addresses @Bruce-Yii's blocker and @NandhaKishorM's hold. Both were right, and the blocker was mine.

THE BLOCKER. `os._exit(1 if FAIL else 0)` was at module level with no guard, so `pytest tests/`
imported this file, the interpreter left during collection, and pytest reported SUCCESS having executed
nothing. Reproduced before fixing, exactly as the review described:

    python -m pytest tests/test_onnx_export_batch.py -q   ->      0 bytes, exit 0
    python -m pytest tests/test_runtime_fixes.py -q       ->   9 757 bytes, 103 INTERNALERROR
                                                               lines, exit 3   (it uses sys.exit)

`sys.exit` is loud there and a hard exit is silent, which is what made this the one file in the
repository that could turn a red run green by leaving. CI never saw it because ci.yml invokes this file
as `python tests/...`, which is precisely why it had to be caught by review and not by me.

Fixed, and then taken one step further than asked. The guard alone leaves pytest at exit 5, "no tests
ran" -- never a false success, but a failing check would not change the status either. So the file also
grows a single collected entry point, `test_onnx_export_batch_suite`, which asserts on the `FAIL` list
the module body already builds. Measured:

    pytest, green          exit 0, "1 passed", 5 runs out of 5
    pytest, one check made to fail    exit 1, "AssertionError: 1 of 88 checks failed"
    pytest, TRACE_BATCH=1 (the NandhaKishorM#695 bug)   exit 2, the ONNX error named during collection
    script mode, green     exit 0, "91 passed, 0 failed", 5 runs out of 5

And the guard is now pinned, because a mutation sweep driven by `python tests/<name>.py` cannot see it
-- in script mode removing the guard changes nothing. Four source checks assert the hard exit occurs
once, is indented, and sits after a `__main__` guard. Writing them took three attempts and the first two
could not fail: one counted its own detector line, and one compared against a literal that appears in
this file's own source, so it was true with the real guard deleted. The needle is assembled at run time
now. Mutation: removing the guard is KILLED by `guard/the file has a __main__ guard`.

REBASED ONTO CURRENT MAIN, 210 commits on. NandhaKishorM#726 reverted the declaration to `dynamic_axes` with a
compatibility reason this branch did not have: `dynamic_shapes` raises under the TorchScript exporter,
the default through torch 2.8, and does not exist before 2.5. So the `_dynamic_shapes` helper this
branch added is REMOVED and upstream's `dynamic_axes` is what ships, with that reasoning carried onto
the constant. The test that asserted `torch.export.Dim` object identity goes with it; the literal
axis-name table is what carries the shared-symbol property now. NandhaKishorM#790's `--per-channel` flag is kept
alongside `--no-verify`.

INT8_ATOL STAYS AT 1e-1, and the docstring now says why in a form that should stop the next reviewer
tightening it. A healthy export's drift, by platform:

    macOS / arm64, per-tensor (the default since NandhaKishorM#790)   1.65e-03
    macOS / arm64, per-channel                           9.95e-04
    Linux x86 (CI)                                       2.53e-02   <- 2e-02 failed this job
    Windows                                              between 5e-02 and 7e-02: a good export
                                                         FAILS at 5e-02, passes at 7e-02

Two orders of magnitude across three platforms, and 1e-1 is the only value that accepts all of them --
on Windows the headroom is under 2x, not 100x. The Windows figures are @Bruce-Yii's, the macOS ones are
mine, the Linux one is this PR's own failing log. Note the per-tensor row: NandhaKishorM#790 changed the default, so
the number this tolerance has to accept is no longer the per-channel one it was first fitted to.

Verified on the new base: 91 passed, 0 failed in script mode and 1 passed under pytest; mutation table
21 of 22 killed, the survivor being the session release that cannot be observed from Python;
test_onnx_quantize 16, test_onnx_batch 59, test_onnx_sort 18, test_onnx_lang_parity 20, test_evals 111,
the ci.yml pytest lane 273 passed / 2 skipped, 75 script suites, ruff and compileall clean.
@keemsisi
keemsisi force-pushed the fix/onnx-export-dynamic-batch branch from bc6d36d to 1928fc4 Compare October 1, 2026 22:28
…ation NandhaKishorM#726 reverted

Reverting to `dynamic_axes` left a comment two screens down still making the case for
`dynamic_shapes` -- 'What decides between them is which one keeps working' -- which is now the
opposite of what the code does. Rewritten to say what ships and why, and to keep the one part of the
old reasoning that survives: torch does call `dynamic_axes` deprecated under `dynamo=True`, and that
is an argument for keeping `verify_batch_dynamic` on by default rather than for moving the declaration
ahead of the torch versions this package supports.
…added to avoid, and four checks could not fail

A second adversarial review of the previous commit found 15 findings. All were reproducible and most
were mine. The headline one means the previous commit traded one defect for another.

THE PYTEST ENTRY POINT WAS WORSE THAN NO ENTRY POINT. `os._exit` is what lets the script path skip the
C++ static destruction that torch + onnxruntime + the quantizer abort in. pytest owns the exit, so it
cannot use that escape -- and with the entry point in place the abort came back, now reported as a green
test followed by a dead process:

    1 passed, 20 warnings in 10.27s
    libc++abi: ... recursive_mutex lock failed: Invalid argument
    EXIT=134

Measured over 40 pytest runs: **9 exited 134 after printing `1 passed`** (22.5%). My own check before
pushing was 8 runs, which saw none of them. Script mode over the same 40: 0 nonzero.

So the file now refuses to run inside pytest, with the reason recorded at the skip. `pytest <file>` is
exit 5 / "1 skipped"; `pytest` over a directory is exit 0 with "1 skipped" visible among the others;
script mode -- what `ci.yml` invokes -- is unchanged. Both modes: 0 nonzero in 20 runs each. That is
not a false success (it says skipped) and it is not flaky. Running the checks under pytest for real
needs a subprocess so the native teardown happens where `os._exit` applies; that is a larger change and
is noted rather than made.

FOUR CHECKS COULD NOT FAIL, each proven by mutation:

* `limitation/... marker width is not dynamic` was satisfied by a HISTORICAL RETRACTION. The string
  "not dynamic" occurs in the docstring only inside "An earlier revision ... said the width was \"not
  dynamic\" ... that is not necessary", so the check passed *because* the docs say its premise was
  wrong -- and asserted the opposite of what the code does. Gutting 735 characters of the limitation
  paragraph was green; rewording the retraction without changing a fact was red. Now inverted: the
  gutting produces 3 FAILs and the rewording is green.
* `graph/every input that shares an axis declares the same symbol name` compared the test's own literal
  against another literal. Desyncing the batch symbols in production left it green (caught by three
  other checks, never by the one that names the property). Now read from `DYNAMIC_AXES`.
* `verify/compares logits against PyTorch` -- "logits" is a substring of "act_logits", so the act rows
  alone satisfied it. My first fix (`" logits "`) also could not fail, because every row carries a
  `(raw logits ...)` suffix. Anchored to the `batch N <field>` column now, and dropping `logits` from
  the comparison fails it by name.
* `cli/the fp32 check sweeps more than batch 1` read the STUB's `batches=(1, 2, 3)` default, so
  shrinking the production default could not fail it. The stub defaults to `None` and the assertion
  reads the real signature.

THE SUMMARY WAS NOT GUARANTEED, which is `attempt()`'s whole promise. Three realistic regressions --
TRACE_BATCH=1, a dropped `dynamic_axes`, INT8_ATOL=1e-9 -- were correctly RED but each ended the file on
a bare traceback at roughly check 12 of 96, with 80+ checks unrun and no summary. Wrapping blocks one at
a time did not fix it (I found four separate propagation sites in turn), so the reporter is registered
with `atexit`: an uncaught exception unwinds to the interpreter, which runs it before exiting non-zero.
Now TRACE_BATCH=1 prints `61 passed, 21 failed`, no dynamic_axes `58 passed, 22 failed`, INT8_ATOL=1e-9
`92 passed, 3 failed`.

Also fixed:

* `INT8_ATOL`'s asserted floor was `5e-2`, BELOW the Windows figure the comment itself cites (a good
  export fails at 5e-2). `6e-2` passed the check while being a value that rejects a healthy Windows
  export. Floor is `7e-2` now, and 6e-2 fails.
* `verify_batch_dynamic` never compared output SHAPES, so a width-divergent graph died on a raw numpy
  broadcast `ValueError` instead of the `SystemExit` the docstring promises. Shapes are checked first.
* A leaked session: `_s3` was never released while the comment claimed all were.
* `probabilities()` was almost unpinned -- replacing it with the identity left 90 of 91 green and the one
  failure pointed at a comment. The documented property is asserted directly now: a constant added to
  every logit in a row must be invisible. With the identity it fails by 1.00e+02.
* Nothing asserted which blocks ran. Simulating the Windows skip path was green at 85 passed with the
  whole `--quantize` wiring and the real INT8 verification unexecuted. The block list and a floor are
  asserted.
* Three false comment claims withdrawn: "16 of the 32 rows come out entirely padding" (a negative slice
  WRAPS -- it is one row, measured); "per-tensor quantization does not complete at all" (it is the
  default since NandhaKishorM#790 and this suite exercises it every run, at 1.65e-03); and `sys.exit` "58 times
  across tests/" (67). Plus the two files contradicting each other about whether live sessions cause the
  abort -- the earlier causation claim is withdrawn.

Verified: 99 passed, 0 failed; 0 of 20 nonzero in script mode and 0 of 20 under pytest; mutation 29 of
30 killed across three specs, the survivor being the session release that cannot be observed from
Python; test_onnx_quantize 16, test_onnx_batch 59, test_onnx_sort 18, test_onnx_lang_parity 20, the
ci.yml pytest lane 273 passed / 2 skipped, 75 script suites, ruff and compileall clean.

Real weights, the 421M english checkpoint exported and run: batch 1/2/3/5/8 at five different sequence
widths and marker counts all run, worst probability difference 1.66e-05 against the 1e-3 tolerance, and
`verify_batch_dynamic` accepts it. Controls on the same real export: the batch axis pinned to 1 is
REFUSED, and a model with one weight scaled 1.25x is REFUSED.
…low Ctrl-C

The five handlers the previous commit added to keep a failing block inside the summary were
`except BaseException`, which catches `KeyboardInterrupt` as well: a Ctrl-C during an export, a
`session.run` or the quantize block would have been recorded as a FAIL and the suite would have
carried on to the next block.

`SystemExit` has to be named explicitly -- it is what `verify_batch_dynamic` raises and it is not an
`Exception` subclass -- so the handlers are `(SystemExit, Exception)`. That keeps the guarantee the
previous commit was for and leaves an interrupt alone.

Verified: a `KeyboardInterrupt` raised inside a wrapped block exits 130 with the interrupt visible and
0 occurrences of the block's FAIL line, where before it was swallowed. Suite unchanged at 99 passed,
0 failed; ruff clean.

Found while answering "did we introduce anything", not by a reviewer -- which is the right way round
for once.
@keemsisi

keemsisi commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Holding on the os._exit blocker, which has not been addressed on the current head.

The finding is the kind I will not merge past, because of what it does rather than what it is: tests/test_onnx_export_batch.py ends in a module-level os._exit(1 if FAIL else 0) with no __main__ guard, so under pytest the process exits during collection and the run reports success having asserted nothing. A test file that can make the suite green by leaving is worse than a missing test, and this one is the only thing standing behind the export.

The change itself is worth having and your framing of it is honest: the original bug is fixed on main by #717, you rebased onto it and adopted its API, and what is left is the exporter verifying the graph it just wrote. That is a real gap, and #726 landed in 0.3.23 moving the declaration back to dynamic_axes with an assertion that the symbolic axes are declared by name, so there is now some overlap to reconcile too.

Add the if __name__ == "__main__": guard so pytest collects and runs the file normally, rebase onto current main so the scripts/export_onnx.py overlap with #726 is resolved in one place, and I will take it.

@NandhaKishorM Both conditions are addressed on 713f748, and the guard needed a different shape than first expected — detail below, since it changes what the file does.

1. The guard

Added, and the file no longer executes inside pytest. The reason is worth recording: os._exit is what lets the script path skip the C++ static destruction that torch + onnxruntime + the quantizer abort in on macOS. pytest owns the exit, so it cannot use that escape — and with a pytest entry point in place the abort returns as a green test followed by a dead process:

1 passed, 20 warnings in 10.27s
libc++abi: ... recursive_mutex lock failed: Invalid argument
EXIT=134

Measured over 40 pytest runs: 9 exited 134 after printing 1 passed (22.5%). Script mode over the same 40: 0 nonzero.

The file therefore declines to run in pytest's process, with that measurement recorded at the skip:

invocation result
python tests/test_onnx_export_batch.py (what ci.yml runs) exit 0, 99 passed, 0 failed — 0 nonzero in 20 runs
pytest over a directory exit 0, 1 skipped visible among the rest — 0 nonzero in 20 runs
pytest <this file> alone exit 5, 1 skipped

That removes the failure mode the review identified — pytest can no longer report anything about this file without saying what it did — and it is deterministic rather than flaky.

Running the checks under pytest for real is possible via a subprocess, so the native teardown happens where os._exit applies. That needs the module body behind a guard, which is a ~700-line reindent, so it is noted rather than done. Happy to do it if you would prefer that shape.

2. Rebased onto current main, #726 resolved in one place

Base is 4aa6761. #726 carries a compatibility reason this branch predates — dynamic_shapes raises under the TorchScript exporter, the default through torch 2.8, and does not exist before 2.5 — so the _dynamic_shapes helper is removed and dynamic_axes=DYNAMIC_AXES is what ships, with that reasoning on the constant. The torch.export.Dim identity test is removed with it; the literal axis-name table carries the shared-symbol property. #790's --per-channel is kept alongside --no-verify.

Also in this round, from @Bruce-Yii's second pass

Four checks in the new suite were not falsifiable and are rewritten, each now verified by mutation. Two gaps in the shipped code are closed: verify_batch_dynamic now compares output shapes before subtracting, so a width-divergent graph raises the documented SystemExit rather than a raw numpy broadcast error; and INT8_ATOL's asserted floor was below the Windows figure its own comment cites, which is corrected. The suite's summary is now guaranteed by an atexit-registered reporter, so a regression that aborts a block still prints N passed, M failed instead of a bare traceback. Four comment figures are corrected against measurement.

What remains that main does not have

The verification. main's exporter never runs the graph it writes — zero InferenceSession( or .run( calls, its only onnxruntime import being quantize_dynamic — so a batch-1-only graph and a working one declare the same thing and nothing at export time distinguishes them. Alongside it, example_inputs() carries a padded row, a masked-off trailing marker and a varying qtype, which torch.ones/torch.zeros do not, so src_key_padding_mask, the masked_fill on option logits and two of the three question types are absent from the traced graph.

Verification

tests/test_onnx_export_batch.py      99 passed, 0 failed   (0 of 20 nonzero, both invocations)
mutation table                       29 of 30 killed across three specs
test_onnx_quantize 16 · test_onnx_batch 59 · test_onnx_sort 18 · test_onnx_lang_parity 20
the ci.yml pytest lane              273 passed, 2 skipped
script-style suites                  75 ok, 0 failed
ruff + compileall                    clean
suite runtime                        10s

The surviving mutant is the ONNX Runtime session release in verify_batch_dynamic's finally, which is not observable from Python — the local is released on return either way, and on the raising path the traceback is cleared before any observer can inspect it. It is kept as hygiene and is not credited with fixing the abort.

Real weights, the 421M english checkpoint exported and run: batch 1/2/3/5/8 at five different sequence widths and marker counts all run, worst probability difference 1.66e-05 against the 1e-3 tolerance, and verify_batch_dynamic accepts it. Controls on the same export: the batch axis pinned to 1 is refused, and a model with one weight scaled 1.25x is refused. Worth noting from that run — raw act_logits drift up to 3.22e-02 while the probabilities differ by exactly 0.00e+00, which is why the comparison is in probability space.

@keemsisi

keemsisi commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

The load-bearing claim holds up — I reproduced it by mutation — but there is one blocker, and it is the last line of the new test file.

The blocker: os._exit is unguarded, and pytest can be made to report success having run nothing

tests/test_onnx_export_batch.py:724 is os._exit(1 if FAIL else 0), at module level. There is no if __name__ == "__main__": guard anywhere in the file — the three __main__ hits are all inside comments, one of which (:666) explains that a CLI body moved into a function precisely so a check can read main's source rather than the guard.

The file matches pytest's test_*.py glob, so a plain pytest run imports it and the os._exit fires during collection. Measured on this branch, same interpreter, same tree:

command output exit
python -m pytest tests/test_runtime_fixes.py -q 9007 bytes — 60 passed, then INTERNALERROR> Traceback 3
python -m pytest tests/test_onnx_export_batch.py -q 0 bytes 0
The control is the comparison that matters. test_runtime_fixes.py ends in sys.exit(...), and pytest turns that into a loud ~30-line INTERNALERROR with exit 3. os._exit(0) produces no output at all and a success status — the file's own 83 checks never print, and pytest reports green having executed none of them.

This would be the only such suite in the repository: sys.exit appears 58 times across tests/*.py, os._exit zero.

It does not currently break CI, and I want to be precise about why, because it changes how urgent this is: ci.yml:232 runs it as python tests/test_onnx_export_batch.py (script mode, where the guard would not matter), and every pytest invocation in ci.yml:193/333/387 and release.yml names its files explicitly, with ci.yml:335-337 citing #374 for that choice. No bare pytest, no testpaths/addopts, no conftest.py. So it is a latent trap, not an active break. The exposure is a contributor running pytest tests/ locally to check their own change, who gets a green that means nothing — and this repo already has form here.

Minimal fix, which keeps the macOS workaround for the invocation CI actually uses:

if __name__ == "__main__":
    sys.stdout.flush()
    sys.stderr.flush()
    os._exit(1 if FAIL else 0)

The gate itself works

I mutated TRACE_BATCH = 2 to 1 — the original #695 bug — and the suite goes red with the consequence named:

ONNXRuntimeError: INVALID_ARGUMENT : Got invalid dimensions for input: marker_pos
  index: 0 Got: 3 Expected: 1     (exit 1)

Restored, the same tree is 83 passed, 0 failed, exit 0, summary printed. Separately, the SystemExit path you added is covered on its own rather than only as a side effect: :472-513 asserts verify_batch_dynamic raises SystemExit in three separate cases, and :546 pins the shipped default via inspect.signature(...).parameters["atol"].default == 1e-3 so the expectation cannot drift with the constant it is checking.

On INT8_ATOL = 1e-1 — a guardrail, because the tightening looks tempting and is wrong

I spent time trying to show this was too loose, and it is not, at least not for the reason it first appears. A good int8 graph on this machine already drifts further than the PR's own CI log suggests. Tightening the constant until the healthy export fails:

INT8_ATOL result on this tree
1e-2 exit 1 — a good graph fails
3e-2 exit 1
5e-2 exit 1
7e-2 exit 0
1e-1 (shipped) exit 0 — 83 passed, 0 failed
So on Windows here a healthy export needs somewhere between 5e-2 and 7e-2, against the 9.95e-04 you measured on macOS arm64 and the 2.53e-02 that failed your CI on Linux x86. That is a spread of roughly two orders of magnitude across three platforms, and 1e-1 is the only value in the set that covers all of them. The headroom is not 100x — on the platform that matters most here it is under 2x.

Worth stating in the docstring, because the obvious "reviewer move" here is to tighten it: any value at or below 5e-2 will reject valid exports on at least one of the three platforms, and the resulting failure would look like a broken quantizer rather than a too-tight tolerance. If you want the gate to carry more numerical weight than "does it still run at batch > 1", the place to add it is a per-platform floor recorded in the test, not a lower constant.

Requesting changes for the os._exit guard only. Everything else I checked holds.

@Bruce-Yii Thank you — a thorough review. All 15 findings reproduced; 14 are fixed on 713f748 and one is declined with a reason. Taking them in order of consequence.

F1 — the entry point could not be kept

Reproduced at a higher rate than your sample: 9 of 40 pytest runs exited 134 after 1 passed, against your 2 of 51. Script mode over the same 40: 0 nonzero.

The conclusion is that no in-process pytest run of this file is safe. os._exit is the escape and pytest owns the exit, so the file now declines to run in pytest's process:

invocation result
python tests/test_onnx_export_batch.py exit 0, 99 passed, 0 failed — 0 of 20 nonzero
pytest over a directory exit 0, 1 skipped visible — 0 of 20 nonzero
pytest <this file> alone exit 5, 1 skipped

Deterministic, and it states what it did. Your note that pytest tests/ -k onnx_export already dies at exit 3 on test_agent_gate.py is recorded too — confirmed pre-existing and untouched by this branch.

The subprocess route is not taken here: it needs the module body behind a guard so pytest's process never opens a session, which is a ~700-line reindent for the same guarantee the skip provides in one block. Say if you would rather have it.

F2 — the strongest finding in the review

The check was satisfied by a historical retraction of its own premise, and asserted the opposite of the code. Both of your directions now invert:

  • gutting the 735-character limitation paragraph → 3 FAILs (was green)
  • rewording the retraction without changing a fact → green (was a FAIL)

It now asserts what the docstring states: dynamic from two markers up, baked at exactly one, and the TopK error a width-1 feed produces.

F3, F5, F13 — the dead checks

  • F3 reads DYNAMIC_AXES rather than comparing two literals, with a second check that every input's leading axis is the same symbol. Desyncing the batch symbols now fails it by name.
  • F5 needed two passes. " %s " is also insufficient: every report row carries a (raw logits ...) suffix, so the padded form matches on act rows too. It is anchored to the batch N <field> column now, and dropping logits from the comparison fails it by name with a diagnostic.
  • F13 implemented as you suggested — the stub defaults to batches=None and the assertion reads the real signature, so "production omitted the argument" is distinguishable from "production passed (1, 2, 3)".

F4 — wrapping the two blocks was not sufficient

There are four propagation sites, not two: the names/ block's session.run, the report-capture redirect_stdout, a try/finally with no except, and the quantize/cli pair. Rather than enumerate them, the reporter is registered with atexit, so an uncaught exception unwinds to the interpreter, which runs it before exiting non-zero:

TRACE_BATCH=1      exit=1   61 passed, 21 failed
no dynamic_axes    exit=1   58 passed, 22 failed
INT8_ATOL=1e-9     exit=1   92 passed,  3 failed
clean              exit=0   99 passed,  0 failed

F6 — the bound contradicted the table it was derived from

Verified your case: 6e-2 passed the check while being a value your Windows data says rejects a healthy export. The floor is 7e-2 now — at or above the widest healthy drift recorded — and 6e-2 fails. Your Windows figures are the binding row in that table and are credited as yours in the docstring.

F7 — fixed as diagnosed

Shapes are compared before the subtraction, with a message naming both shapes and the batch, so a width-divergent graph raises the documented SystemExit instead of a numpy broadcast error.

F8, F11, F12, F15 — the comment claims

  • _s3 is released; it was the one session the file left alive.
  • F11: the mechanism is corrected — a negative slice wraps, so it is one degenerate row rather than 16. Your numbers reproduce; the comment now records one row plus the measured drift (6.35e-02 unclamped against 8.94e-08 clamped), with the raw figures marked order-of-magnitude.
  • F12: withdrawn. Per-tensor is the default since export_onnx --quantize: per-channel dynamic INT8 collapses the model (32% agreement with eager); per-tensor is far better but int8 is lossy overall #790, completes on every run of this suite, and drifts 1.65e-03 — the figure the INT8_ATOL table records. The InferenceError quoted there is what occurs when del model.graph.value_info[:] is removed, which is documented where that line lives.
  • F15: corrected to 67.
  • On the two files disagreeing about whether live sessions cause the abort: the earlier causation claim is withdrawn rather than reconciled. Your point that separating the hypotheses needs hundreds of runs at a ~4% base rate is well taken, and it is moot now that the suite does not run in a process that aborts.

F9, F14 — both fixed, F14 partially

  • F9: the block list and a floor are asserted. Your simulated Windows path, green at 85 with the --quantize wiring unexecuted, now fails.

  • F14: the documented property is asserted directly — a constant added to every logit in a row must be invisible. With probabilities() as the identity it fails by 1.00e+02, pointing at the softmax rather than at a comment.

    Declined: the logits * 3.0 half. Measured on this fixture, a 3x scale stays inside TINY_ATOL because its two logits sit close together, so asserting it would pin the fixture rather than the property. The scale direction is already covered at the shipping tolerance by the existing 1.25x check.

F10 — and one adjacent to it

The entry point is removed, so there is nothing left to pin there. Related: the five handlers F4 required were initially except BaseException, which also catches KeyboardInterrupt — a Ctrl-C would have been recorded as a FAIL with the suite continuing. They are (SystemExit, Exception) now, since SystemExit must be named but an interrupt should propagate. Verified: a raised KeyboardInterrupt exits 130 and is not recorded.

Where it stands

99 passed, 0 failed · 0 of 20 nonzero in both invocations
mutation 29 of 30 killed across three specs
suite runtime 12s -> 10s while going from 83 to 99 checks

The survivor is the session release in verify_batch_dynamic's finally, for the reason your F8 identifies: it is not observable from Python. It is kept as hygiene and no longer credited with fixing the abort.

Real weights, the 421M checkpoint: batch 1/2/3/5/8 at five sequence widths and marker counts all run, worst probability difference 1.66e-05 against the 1e-3 tolerance. Controls on that export — batch axis pinned to 1 refused, one weight scaled 1.25x refused. Supporting the probability-space choice: raw act_logits drift up to 3.22e-02 there while the probabilities differ by exactly 0.00e+00.

The common thread across F2, F3, F5 and F13 is a check reading its expectation from the thing under test. Each rewritten check is now verified against a mutation of the property it names, and the mutation specs are in the repo history alongside the fixes.

@keemsisi

keemsisi commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@NandhaKishorM PR pending your review.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rechecked 713f748 against the blocker I raised earlier. The original os._exit false-green problem is resolved: the exit is now under the __main__ guard, so pytest collection can no longer terminate the process successfully before the checks run. The exporter verification itself is also now doing the load-bearing thing: it runs the written graph in ONNX Runtime at batch > 1 and compares the outputs in probability space, while keeping #726's dynamic_axes direction.

I would consider my original blocker closed on this head.

I am not marking the PR approved yet for one separate reason: Nandha's Oct 1 merge condition also asked for a rebase onto current main / fresh overlap reconciliation. This branch still has merge base 4aa6761, while current main is a4a8921 and has moved substantially since then, including CI/test-suite topology changes. After that rebase, this needs a fresh exact-head integration check rather than carrying this old-head closure forward.

@NandhaKishorM
NandhaKishorM merged commit 9682480 into NandhaKishorM:main Oct 17, 2026
25 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 17, 2026
tests/test_portability.py requires every repo read under tests/ to name its
codec, so the locale cannot decide it on Windows. The new suite's one read of
laya/onnx_agent.py was the only unpinned one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants