Skip to content

test(dotnet,rust): run each identical forward pass once in the batch replays - #1381

Open
kevin-gatimu wants to merge 1 commit into
NandhaKishorM:mainfrom
kevin-gatimu:test/replay-identical-once
Open

kevin-gatimu wants to merge 1 commit into
NandhaKishorM:mainfrom
kevin-gatimu:test/replay-identical-once

Conversation

@kevin-gatimu

Copy link
Copy Markdown
Contributor

Runs each byte-identical forward pass once in the batch replays, in both SDKs, with no loss of coverage.

  • A test-only memoiser shares the ONNX run between batch cases whose collated input tensors are identical (all inputs compared element by element, one memo per checkpoint and layout). Every case still tokenizes, plans its chunks, collates, gates and decodes on its own, and is compared with its own golden.
  • It needs one small internal hook on each engine, off by default (.NET: an internal ForwardInterceptor; Rust: a #[doc(hidden)] set_forward_hook). Run / run are unchanged.
  • It removes 8 of 107 batch passes per checkpoint and layout, about a third of the batch replay's padded tokens.

Tested on our fork, on top of #1373 and #1375: the longest .NET cell went from 45 to 37 min and the longest Rust shard from 69 to 55 min; every test and skip-guard step passed. Model-free: .NET 2177, Rust 377 passed.

Copilot AI balanced review requested due to automatic review settings October 26, 2026 04:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Exact-input matching, isolated caches, copied outputs, and scoped hooks preserve independent per-case assertions with no blocking issues identified.

No open issues 🎉

What changed in this PR

This PR reduces repeated ONNX inference in the .NET and Rust batch parity suites while preserving each case’s preprocessing, decoding, and golden assertions.

Changes:

  • Adds test-only memoization keyed by all collated input tensors and dimensions, isolated by checkpoint and layout.
  • Adds opt-in engine hooks around prediction forward passes without changing Run / run.
  • Adds model-free tests for cache hits, differing inputs, output-copy isolation, and checkpoint/layout separation.
File Description
laya-rust/​tests/​predict_batch_tests.rs Installs memoization during batch parity replays.
laya-rust/​tests/​forward_memo_tests.rs Tests memoization correctness and error handling.
laya-rust/​tests/​common/​mod.rs Exposes the shared memoization test helper.
laya-rust/​tests/​common/​forward_memo.rs Implements exact-input caching and scoped hook cleanup.
laya-rust/​src/​laya_engine.rs Adds an optional prediction forward hook.
laya-dotnet/​tests/​Laya.Tests/​PredictParityTests.Batch.cs Installs memoization during batch parity replays.
laya-dotnet/​tests/​Laya.Tests/​ForwardMemoTests.cs Tests input matching, output copies, and cache isolation.
laya-dotnet/​tests/​Laya.Tests/​ForwardMemo.cs Implements exact-input caching and scoped interceptor cleanup.
laya-dotnet/​src/​Laya/​LayaEngine.cs Adds an internal prediction forward interceptor.

Review effort: Balanced

NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding this one, and I want to be clear that it is not a rejection of the idea. The measurement is good and the win is real: removing 8 of 107 batch passes and about a third of the padded tokens, with the Rust lane going from roughly 69 to 55 minutes and .NET from 45 to 37. Those lanes are the slowest thing in the release and this is the first change to move them for the right reason, by doing less work rather than by raising a budget.

Two things need to change before I merge it.

First, the Rust seam. pub type ForwardHook and pub fn set_forward_hook are marked #[doc(hidden)], which hides them from rustdoc but does not make them private. laya-onnx is not on crates.io yet and the crate does not set publish = false, so the first publish makes this public API, and it is public API whose purpose is to replace a production forward pass. Once it is published it cannot be removed without a breaking release, and #[doc(hidden)] will not stop anyone who reads the source or an error message from finding it. The .NET side already has the shape I would want, internal Func<...>? ForwardInterceptor, which is invisible outside the assembly.

What I am asking for is a test-hooks cargo feature gating the hook, with --features test-hooks added to the lane that needs it. That keeps your speedup in CI and keeps the seam out of the published surface. A #[cfg(any(test, feature = "test-hooks"))] on the type, the setter and the call site should be the whole change.

Second, the RwLock::read() on every prediction. Even uncontended that is real work on the hot path, and it is paid by every user to support a test seam. Behind the feature gate it disappears from a default build entirely, so the same change answers both points.

One more thing worth confirming while you are in there: with the hook behind a feature, please check that the lane still reports the same passed count rather than silently skipping the tests that need it. check_test_skips.py --min-passed should catch it, but a feature flag that is quietly absent is exactly the failure that tool exists for.

Happy to merge as soon as the seam is gated. The performance work itself I have no notes on.

…replays

A test-only memoiser around the engine's forward pass (an internal hook, off by default) shares the ONNX run between batch cases whose collated input tensors are byte-identical. Every case still tokenizes, plans, collates, gates and decodes on its own and is compared with its own golden.
@kevin-gatimu
kevin-gatimu force-pushed the test/replay-identical-once branch from 5a086b1 to 377fa81 Compare November 1, 2026 20:17

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants