Repository navigation
feat(evals): score an ONNX export with laya-evals run --onnx - #506
Merged
Merged
Conversation
The harness could only evaluate the torch Router, so an ONNX deployment -- including a quantized export -- had no way into the same thresholds and baseline gates. Add OnnxRunner, the single-checkpoint adapter the harness contract expects, and `run --onnx PATH`: --model names the checkpoint the export came from, a per-example model the agent does not serve is a clear error rather than a wrong-model no-op, and --batch-size delegates to the agent's batch API when it has one. Measured on the fixture (English checkpoint, CPU): the fp32 export reproduces the torch metrics exactly (choice_acc 0.75, score_mae 1.3418, ece 0.1596) at p50 66 ms vs 117 ms; int8 moves score_mae by 0.009 and ece by 0.006. Weight-free suite fakes the agent and monkeypatches ONNXAgent, so the CLI wiring runs in the existing weight-free evals job without onnxruntime.
This was referenced Sep 26, 2026
Owner
|
Thanks @aashish254, merged. It sits next to #588's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya-evals runcan only evaluate the torchRouter. An ONNX deployment — which the repo shipsas a first-class backend (
ONNXAgent,scripts/export_onnx.py, and now the INT8 sidecar from#498) — has no way into the same thresholds, slices and baseline gates. This adds it:
OnnxRunnerinlaya/evals_cli.py, besideRouterRunner: the harness contract(
predict(state, questions, model=)/ optionalpredict_batch) adapted to a single-checkpointONNXAgent.modelnames a checkpoint the agent does not serve raises a clearEvalErrorinstead of silently answering from the wrong model — the report would otherwiseclaim to score a fleet it never ran.
--batch-sizedelegates to the agent's batch API when it has one and falls back to one callper state otherwise, so this works on today's
ONNXAgentand gets shared forward passes forfree if feat(onnx): add ONNXAgent.predict_batch, the batch API the torch Agent has (#323) #489 lands.
configblock records theonnxpath (only when--onnxis used — torch-pathreports are byte-identical).
Measured (research/evals/fixture.jsonl, 12 labelled rows, English checkpoint, CPU)
--onnxfp32--onnxint8The fp32 export reproduces the torch metrics exactly on the labelled set, and the quantized copy
moves
score_maeby 0.009 andeceby 0.006 — the kind of driftcompare --toleranceexists togate. The table is in
docs/evals.mdnext to the new section.Tests + CI
tests/test_evals_onnx.py(20 checks, standalone PASS/FAIL + exit code) is weight-free andneeds no
onnxruntime: the runner unit tests use a fake agent, and the CLI end-to-end testsmonkeypatch
laya.onnx_agent.ONNXAgent— that module imports only numpy at import time, sinceonnxruntimeis imported inside the constructor. Covered: predict forwarding, the foreign-modelerror naming both checkpoints, batch fallback vs delegation, parser defaults,
--jsonreportconfig.onnx, exit codes, and the default-checkpoint path. Wired into the existing weight-freeevalsjob inci.yml.Existing suites re-run unchanged:
test_evals.py(12 + 18 pytest),test_evals_api.py,test_doc_tables.py; gates:ruff(repo select set),compileall,zensical build --strictall pass.
Not touched
laya/evals.py(the metric math andevaluateloop are unchanged — this is purely a runneradapter plus CLI wiring), the torch path, and no new dependencies.