Skip to content

feat(rust): add Rust SDK (laya-onnx) - #672

Merged
NandhaKishorM merged 26 commits into
NandhaKishorM:mainfrom
kevin-gatimu:feat/rust-sdk
Oct 17, 2026
Merged

NandhaKishorM merged 26 commits into
NandhaKishorM:mainfrom
kevin-gatimu:feat/rust-sdk

Conversation

@kevin-gatimu

@kevin-gatimu kevin-gatimu commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Closes #674

What is this

A Rust SDK in laya-rust/: a port of the Python SDK's inference path (tokenization, sequence building, calibration, answers) running the official checkpoints exported to ONNX, on ONNX Runtime. No Python process runs at runtime.

What it does

  • choice / score / noul answers, with confidence and answer_confidence, for all three checkpoints (multilingual, english, typed-decisions). It supports both the single-file and the split ONNX layouts.
  • It also includes the router, language detection, email, and shortlist, plus opt-in download from HF.
  • Package laya-onnx 0.1.0 on crates.io, versioned on its own. It's ready to publish, but I haven't published anything; that's yours to do.
  • Three samples: quickstart, routing, and benchmark.

What it unlocks

Rust services can call Laya in-process, with no Python sidecar.

Verification

  • The golden outputs were recorded from laya (Python) 0.4.0 and are checked in under tests/golden/. CI re-records them from the Python code at each commit with laya-rust/tools/regen_golden.py (it uses dump_golden.py and dump_routing_golden.py), so the checked-in copies are only a starting point. There are 17 cases per checkpoint, plus tokenizer and JSON probes and routing probes.
  • Parity: probabilities and confidences are within 2e-4 of Python, and scores within 2e-3. The recorded values are rounded to 4 dp. Token ids and marker positions match exactly.
  • Full suite: 208 passed, 0 failed, 0 skipped, across all three checkpoints and both layouts. In this suite each #[test] loops over all golden cases and checkpoints.
  • One-command parity check:
    cd laya-rust && LAYA_ONNX_ROOT=/path/to/onnx cargo test --release -- --test-threads=1
    Without the exported models, the model tiers print skip: and the stub and math tiers still run.

Notes for reviewers

  • Scope: laya-rust/ plus one new workflow, .github/workflows/rust.yml. No PyPI or Python package changes. The workflow is advisory (continue-on-error), runs on every push and pull request with no path filter, re-records the goldens from Python, and fails the run if a model-backed test skips. It never publishes.
  • Known gaps:
    • lang and email were brought in line with Python 0.4.0 where the regenerated goldens showed drift.
    • The opt-in min_confidence / abstention, predict_long and lang_temperatures features aren't ported yet.
  • Proposal: publish the split-layout ONNX exports on HF next to the PyTorch weights. Today users run tools/export_onnx.py themselves, and one set of exports would serve every ONNX-based SDK.
  • Follow-ups:
    • the ONNX weights on HF;
    • a release workflow (tag laya-rust-v*) and publishing to crates.io;
    • the opt-in features above.
  • CI: the red dependency CVEs check was already failing on main before this PR. It's tracked in [bug]: dependency CVEs job in security.yml fails on main: 7 advisories in dependencies of optional extras #645 and unrelated to this change.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding this for the same maintainer decision as #671, not on the code.

8,627 lines of Rust with 44,172 lines of golden fixtures, all three checkpoints, both ONNX layouts, router, language detection, email and shortlist, running on ONNX Runtime with no Python at runtime. The approach is right and the verification strategy is the one I would ask for.

The blocker is the same and it is mine: there is no CI lane that builds or tests laya-rust, so the golden fixtures stop being checked the moment this lands, and the Python side moved by 61 PRs in this release alone. Add a crates.io release line and a vendored Cargo.lock and that is a second toolchain to maintain with no review coverage.

The highest-value change is the same: a workflow that runs cargo test on every push, with the golden fixtures regenerated by a committed script so a Python-side change that breaks parity shows up red. That turns the drift objection into a check and makes this a much easier yes.

Leaving it open rather than closing. Thanks for the work, and for proposing it in #674 first rather than dropping it unannounced.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Still holding, and now there is a template for getting it in.

#671, the .NET SDK from the same author, merged for 0.3.26. The thing that moved it was the parity lane: .github/workflows/dotnet.yml re-records the golden fixtures from the Python package at each commit, never caches them, and fails the run if a model-backed test skips so missing artifacts cannot pass as green. The author also demonstrated the gate going red on four deliberate breaks, and its first real run found genuine drift in language detection, router reasons and email cleaning, which they ported from Python without loosening a tolerance.

That is the whole difference. This branch still has no .github workflow and its last commit is 2026-09-28, so my original objection is untouched: 44,172 lines of golden fixtures that nothing re-checks become decorative the moment they land, and the Rust port silently diverges as the Python side moves. It has moved by about 70 pull requests since you last pushed.

So the ask is unchanged and now has a worked example to copy:

  1. A workflow that runs cargo test on every push and pull request, no path filter.
  2. The goldens regenerated by a committed script from the Python laya/ at that commit rather than read from the repository, so a Python-side change that moves an answer turns the lane red.
  3. A guard that fails the run when a model-backed test skips.

One thing I would do differently from #671 if you follow it: I merged that lane as advisory (continue-on-error), not blocking, and I would do the same here. A blocking lane means a Python improvement is held up until someone writes Rust, which puts the cost on the wrong person. Advisory still surfaces drift and attributes it to the pull request that caused it. So you do not need to make the lane green forever, only honest.

Also worth a rebase when you come back to it: laya/ has changed a lot since September, including DecisionModel.forward's dtype handling (#897), per-bucket abstention thresholds (#853) and histogram binning (#871, #890), so some of your goldens will legitimately need re-recording.

The approach is right and the verification strategy is right. It is the maintenance seam that is still open.

Copilot AI balanced review requested due to automatic review settings October 8, 2026 11:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Python float serialization is not byte-compatible as claimed, and the stated parity baseline and CI scope conflict with the implementation.

4 open findings
What changed in this PR

Adds the laya-onnx Rust SDK for local ONNX inference, with Python-parity tooling, fixtures, tests, samples, and advisory CI.

Changes:

  • Implements core serialization, calibration, tokenization, configuration, email, and batching APIs.
  • Adds golden-vector, property, routing, shortlist, and model-backed tests.
  • Adds regeneration tools, sample crates, packaging metadata, and Rust CI.
File Description
laya-rust/​tools/​test_ci_tools.py Tests CI utilities.
laya-rust/​tools/​sample_cases.py Defines shared sample fixtures.
laya-rust/​tools/​requirements-regen.txt Pins regeneration dependencies.
laya-rust/​tools/​regen_golden.py Regenerates exports and goldens.
laya-rust/​tools/​hashing_embedder.py Provides deterministic demo embeddings.
laya-rust/​tools/​check_test_skips.py Rejects skipped or incomplete tests.
laya-rust/​tests/​shortlist_end_to_end_tests.rs Tests shortlist inference parity.
laya-rust/​tests/​router_end_to_end_tests.rs Tests router inference and concurrency.
laya-rust/​tests/​python_json_tests.rs Tests Python-compatible serialization.
laya-rust/​tests/​proptest_tests.rs Adds property-based checks.
laya-rust/​tests/​laya_email_tests.rs Tests email cleaning parity.
laya-rust/​tests/​hashing_embedder_tests.rs Tests embedder vectors.
laya-rust/​tests/​golden/​typed-decisions/​meta.json Records typed-decisions metadata.
laya-rust/​tests/​golden/​typed-decisions/​index.json Indexes typed-decisions cases.
laya-rust/​tests/​golden/​typed-decisions/​case_single_option.json Adds single-option golden.
laya-rust/​tests/​golden/​typed-decisions/​case_mask_injection.json Adds mask-injection golden.
laya-rust/​tests/​golden/​typed-decisions/​case_list_state.json Adds list-state golden.
laya-rust/​tests/​golden/​typed-decisions/​case_choice_criteria_list.json Adds list-criteria golden.
laya-rust/​tests/​golden/​routing/​meta.json Records routing fixture metadata.
laya-rust/​tests/​golden/​multilingual/​meta.json Records multilingual metadata.
laya-rust/​tests/​golden/​multilingual/​index.json Indexes multilingual cases.
laya-rust/​tests/​golden/​multilingual/​case_single_option.json Adds single-option golden.
laya-rust/​tests/​golden/​multilingual/​case_mask_injection.json Adds mask-injection golden.
laya-rust/​tests/​golden/​multilingual/​case_list_state.json Adds list-state golden.
laya-rust/​tests/​golden/​multilingual/​case_choice_criteria_list.json Adds list-criteria golden.
laya-rust/​tests/​golden/​english/​meta.json Records English metadata.
laya-rust/​tests/​golden/​english/​index.json Indexes English cases.
laya-rust/​tests/​golden/​english/​case_single_option.json Adds single-option golden.
laya-rust/​tests/​golden/​english/​case_mask_injection.json Adds mask-injection golden.
laya-rust/​tests/​golden/​english/​case_list_state.json Adds list-state golden.
laya-rust/​tests/​golden/​english/​case_choice_criteria_list.json Adds list-criteria golden.
laya-rust/​tests/​golden_questions_tests.rs Tests question rendering parity.
laya-rust/​tests/​common/​test_artifacts.rs Locates test model artifacts.
laya-rust/​tests/​common/​stub_tokenizer.rs Provides a deterministic tokenizer stub.
laya-rust/​tests/​common/​routing_golden.rs Loads routing fixtures.
laya-rust/​tests/​common/​result_compare.rs Centralizes golden result comparison.
laya-rust/​tests/​common/​mod.rs Exposes shared test helpers.
laya-rust/​tests/​common/​hashing_embedder.rs Ports the demo embedder to Rust.
laya-rust/​tests/​common/​engines.rs Loads test engines and layouts.
laya-rust/​tests/​collator_tests.rs Tests batch collation.
laya-rust/​src/​tokenization/​mod.rs Exposes tokenizer APIs.
laya-rust/​src/​tokenization/​laya_tokenizer.rs Defines tokenizer contracts.
laya-rust/​src/​python_json.rs Implements Python-style JSON formatting.
laya-rust/​src/​lib.rs Defines the crate’s public surface.
laya-rust/​src/​laya_options.rs Defines checkpoints and engine options.
laya-rust/​src/​laya_email.rs Ports email utilities.
laya-rust/​src/​laya_config.rs Parses model configuration.
laya-rust/​src/​error.rs Defines crate-wide errors.
laya-rust/​src/​collator.rs Builds padded ONNX batches.
laya-rust/​src/​calibration.rs Implements probability calibration.
laya-rust/​samples/​laya-sample/​Cargo.toml Configures the quickstart sample.
laya-rust/​samples/​laya-sample/​.gitignore Ignores sample artifacts.
laya-rust/​samples/​laya-routing/​src/​hashing_embedder.rs Adds routing demo embeddings.
laya-rust/​samples/​laya-routing/​Cargo.toml Configures the routing sample.
laya-rust/​samples/​laya-benchmark/​README.md Documents benchmarking.
laya-rust/​samples/​laya-benchmark/​Cargo.toml Configures the benchmark sample.
laya-rust/​samples/​laya-benchmark/​.gitignore Ignores benchmark artifacts.
laya-rust/​CHANGELOG.md Documents the initial release.
laya-rust/​Cargo.toml Configures the workspace and crate.
laya-rust/​.gitignore Ignores Rust build and model artifacts.
.github/​workflows/​rust.yml Adds advisory Rust parity CI.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread laya-rust/src/python_json.rs Resolved
Comment thread .github/workflows/rust.yml Resolved
Comment thread laya-rust/CHANGELOG.md Resolved
@kevin-gatimu

Copy link
Copy Markdown
Contributor Author

Hi @NandhaKishorM,

I pushed the parity lane you described in this comment, modelled on the one in #671. The branch is rebased onto current main (3cf26cb).

What was asked for

  1. rust.yml runs cargo test on every push and PR with no path filter. It is advisory (continue-on-error).
  2. regen_golden.py re-records all goldens from the Python laya/ at that commit. They are never cached.
  3. check_test_skips.py fails the run if a model-backed test skips, or if fewer than 200 tests pass. It has its own tests in test_ci_tools.py.

Drift found and ported (no tolerance loosened, recorded numeric answers unchanged): routing reasons, email cleaning, script counting, score legends, usage and option diagnostics, error messages, clipped option-span collisions, and float repr on exact ties (962380e). The goldens are re-recorded for Python 0.4.0, and the PR and #674 text now say so.

Verification

  • Local: 208 passed, 0 failed, 0 skipped. The 14 CI-script tests pass.
  • Hosted Linux, clean run on the pushed head (9d591bf): green. Goldens regenerated from Python, cargo tests passed, and the skip guard passed.
  • Hosted, deliberate breaks. Each went red on the cargo step and the skip guard:
    • Rust-side, TEMP_MAX 5.0→4.0: run. clamp_temperature_matches_python failed.
    • Python-side, quoted-line stripping disabled in laya/email.py: run. email_probe_clean_matches_every_entry failed.
  • A differential sweep of 2,261 float values against Python went from 33 mismatches per checkpoint to 0. Python now records those probes on every run.
  • One gap: changing a Python default that the golden cases always pass explicitly (clean_email_body's max_chars) is not detected.

Not verified: GPU paths and optional features. Nothing in the workflow publishes, and Cargo.lock and the crate version are unchanged.

@NandhaKishorM
NandhaKishorM merged commit 6c08553 into NandhaKishorM:main Oct 17, 2026
30 of 35 checks passed
@kevin-gatimu
kevin-gatimu deleted the feat/rust-sdk branch October 20, 2026 14:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature]: Rust SDK (​laya-onnx​), an ONNX Runtime port of the Python SDK

3 participants