Skip to content

feat(evals): score an ONNX export with laya-evals run --onnx - #506

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-onnx
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-onnx

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya-evals run can only evaluate the torch Router. An ONNX deployment — which the repo ships
as a first-class backend (ONNXAgent, scripts/export_onnx.py, and now the INT8 sidecar from
#498) — has no way into the same thresholds, slices and baseline gates. This adds it:

laya-evals run data.jsonl --onnx laya.int8.onnx --max-ece 0.05
laya eval run data.jsonl --onnx laya.onnx --batch-size 8 --json report.json   # via the main CLI too
  • OnnxRunner in laya/evals_cli.py, beside RouterRunner: the harness contract
    (predict(state, questions, model=) / optional predict_batch) adapted to a single-checkpoint
    ONNXAgent.
  • A dataset row whose model names a checkpoint the agent does not serve raises a clear
    EvalError instead of silently answering from the wrong model — the report would otherwise
    claim to score a fleet it never ran.
  • --batch-size delegates to the agent's batch API when it has one and falls back to one call
    per state otherwise, so this works on today's ONNXAgent and gets shared forward passes for
    free if feat(onnx): add ONNXAgent.predict_batch, the batch API the torch Agent has (#323) #489 lands.
  • The report's config block records the onnx path (only when --onnx is used — torch-path
    reports are byte-identical).

Measured (research/evals/fixture.jsonl, 12 labelled rows, English checkpoint, CPU)

runner choice_acc noul_acc score_mae ece mean_conf p50 ms
torch Router 0.75 1.00 1.3418 0.1596 0.7304 116.8
--onnx fp32 0.75 1.00 1.3418 0.1596 0.7304 66.3
--onnx int8 0.75 1.00 1.3512 0.1658 0.7304 46.3

The fp32 export reproduces the torch metrics exactly on the labelled set, and the quantized copy
moves score_mae by 0.009 and ece by 0.006 — the kind of drift compare --tolerance exists to
gate. The table is in docs/evals.md next to the new section.

Tests + CI

tests/test_evals_onnx.py (20 checks, standalone PASS/FAIL + exit code) is weight-free and
needs no onnxruntime: the runner unit tests use a fake agent, and the CLI end-to-end tests
monkeypatch laya.onnx_agent.ONNXAgent — that module imports only numpy at import time, since
onnxruntime is imported inside the constructor. Covered: predict forwarding, the foreign-model
error naming both checkpoints, batch fallback vs delegation, parser defaults, --json report
config.onnx, exit codes, and the default-checkpoint path. Wired into the existing weight-free
evals job in ci.yml.

Existing suites re-run unchanged: test_evals.py (12 + 18 pytest), test_evals_api.py,
test_doc_tables.py; gates: ruff (repo select set), compileall, zensical build --strict
all pass.

Not touched

laya/evals.py (the metric math and evaluate loop are unchanged — this is purely a runner
adapter plus CLI wiring), the torch path, and no new dependencies.

The harness could only evaluate the torch Router, so an ONNX deployment
-- including a quantized export -- had no way into the same thresholds
and baseline gates. Add OnnxRunner, the single-checkpoint adapter the
harness contract expects, and `run --onnx PATH`: --model names the
checkpoint the export came from, a per-example model the agent does not
serve is a clear error rather than a wrong-model no-op, and
--batch-size delegates to the agent's batch API when it has one.
Measured on the fixture (English checkpoint, CPU): the fp32 export
reproduces the torch metrics exactly (choice_acc 0.75, score_mae
1.3418, ece 0.1596) at p50 66 ms vs 117 ms; int8 moves score_mae by
0.009 and ece by 0.006. Weight-free suite fakes the agent and
monkeypatches ONNXAgent, so the CLI wiring runs in the existing
weight-free evals job without onnxruntime.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. It sits next to #588's --revision: with --onnx only a bare SHA applies (to the config and tokenizer download), and NAME=SHA is refused as a usage error. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants