Skip to content

feat(onnx): opt-in INT8 quantized sidecar in the ONNX export - #498

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-onnx-quantized-export
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-onnx-quantized-export

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Adds an opt-in --quantize flag to scripts/export_onnx.py. After the fp32 export it writes a
dynamic per-channel INT8 weight-only quantized sidecar next to it: laya.onnx also produces
laya.int8.onnx. ONNXAgent loads it unchanged — the graph keeps the fp32 export's input/output
names, so it is just ONNXAgent(ckpt, onnx_path="laya.int8.onnx").

python scripts/export_onnx.py --model convaiinnovations/laya --output laya.onnx --quantize

Nothing changes unless the flag is passed, and the fp32 model is untouched, so A/B stays possible.

Measured on the English checkpoint (CPU, M-series, 20 realistic support-ticket states x

choice/noul/score through ONNXAgent.system_one)

fp32 int8 (this PR)
model file 1.6 GB 581 MB (~2.9x smaller)
p50 per-state latency ~340 ms ~250 ms (~1.35x faster)
p95 ~400 ms ~280 ms
decision changes vs fp32 — 0 / 20 states (choice labels, noul booleans, score levels)
max probability drift — 0.0901

Dynamic quantization needs no calibration dataset. Per-channel scales are load-bearing:
per-tensor int8 on the same graph flipped 3 of 20 states (noul booleans crossing 0.5 and a score
level) with drift up to 0.29 — parity-unusable; per-channel flipped none at the same file size and
speed. That comparison is why the exporter hard-codes per_channel=True, with the numbers in a
code comment.

Two implementation notes from the real graph

  • The torch exporter leaves intermediate value_info shapes that disagree with the quantizer's
    own shape-inference re-derivation (ShapeInferenceError: (1028) vs (256) on the real export).
    The declarations are informational, so quantize_model drops them before quantizing.
  • The int8 graph is CPU-only by nature: ONNX Runtime has no INT8 MatMul kernel on
    CUDAExecutionProvider. Documented in the flag help and the reference page.

Tests + CI

  • tests/test_onnx_quantize.py (14 checks, standalone assert-style PASS/FAIL + exit code) is
    weight-free: it exercises quantize_model on a tiny hand-built MatMul graph — INT8 initializer
    present, per-channel scales asserted, I/O names preserved, outputs close to fp32 through ORT,
    weight bytes shrink, fp32 sidecar untouched, sidecar naming, and the --quantize CLI surface
    through --help. No checkpoint download.
  • New CI lane onnx export (quantization, weight-free) in ci.yml. The existing test job's
    comment says onnx-dependent tests belong "in a lane that installs onnx" rather than
    skip-guarding into a trivial pass, so this adds that lane instead of touching the shared
    install lists.
  • docs/reference/agent.md: a short "Quantized export" section with the command and the measured
    table.

Gates run locally: ruff (repo's select set) clean on the changed files, python -m compileall
clean, zensical build --strict --clean passes.

laya[onnx] already ships onnx + onnxruntime, so no dependency changes.

Add --quantize to scripts/export_onnx.py: dynamic per-channel INT8
weight-only quantization of the MatMul layers, written as a sidecar
(<output>.int8.onnx) next to the fp32 export so A/B stays possible.
Measured on the English checkpoint (CPU, 20 ticket states x
choice/noul/score): 1.6 GB -> 581 MB, p50 ~340 ms -> ~250 ms, zero
decision changes (max probability drift 0.09); per-tensor scales
flipped 3 of 20 states, so per-channel is what ships. Weight-free
suite on a synthetic MatMul graph, run in a new CI lane that installs
onnx.
@NandhaKishorM
NandhaKishorM merged commit 30ec6fa into NandhaKishorM:main Sep 27, 2026
23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. The per-channel vs per-tensor numbers are exactly the kind of measurement that makes this easy to accept. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants