Repository navigation
test(dotnet,rust): run each identical forward pass once in the batch replays - #1381
kevin-gatimu wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟢 Approval recommended
Exact-input matching, isolated caches, copied outputs, and scoped hooks preserve independent per-case assertions with no blocking issues identified.
No open issues 🎉
What changed in this PR
This PR reduces repeated ONNX inference in the .NET and Rust batch parity suites while preserving each case’s preprocessing, decoding, and golden assertions.
Changes:
- Adds test-only memoization keyed by all collated input tensors and dimensions, isolated by checkpoint and layout.
- Adds opt-in engine hooks around prediction forward passes without changing
Run/run. - Adds model-free tests for cache hits, differing inputs, output-copy isolation, and checkpoint/layout separation.
| File | Description |
|---|---|
laya-rust/tests/predict_batch_tests.rs |
Installs memoization during batch parity replays. |
laya-rust/tests/forward_memo_tests.rs |
Tests memoization correctness and error handling. |
laya-rust/tests/common/mod.rs |
Exposes the shared memoization test helper. |
laya-rust/tests/common/forward_memo.rs |
Implements exact-input caching and scoped hook cleanup. |
laya-rust/src/laya_engine.rs |
Adds an optional prediction forward hook. |
laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs |
Installs memoization during batch parity replays. |
laya-dotnet/tests/Laya.Tests/ForwardMemoTests.cs |
Tests input matching, output copies, and cache isolation. |
laya-dotnet/tests/Laya.Tests/ForwardMemo.cs |
Implements exact-input caching and scoped interceptor cleanup. |
laya-dotnet/src/Laya/LayaEngine.cs |
Adds an internal prediction forward interceptor. |
Review effort: Balanced
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window attribution preserved through end hooks (#1369), RemoteRouter identifying itself with a configurable User-Agent (#1396), ten more inputs refused where they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404, #1405, #1406), per-language temperatures and per-call budgets in .NET and Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle, SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation (#1399, #1407), and parity-lane sharding that replaces the budget headroom added for 0.4.16 (#1373, #1375). Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean. Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0: Java 20 families match laya (9 unverified), UnicodeTables matches CPython 15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match, both laya-cpp .inc files match. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.17 only. Security audit of the merge diff, 362766 added lines over 480 files: zero obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm install hooks unchanged, and no third-party host in any added line (example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12 dynamic-exec hits are the C++ test harness invoking its own checker through sys.executable, the Go sample generator reading laya's own example by AST at development time, and one comment in rust.yml. Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)] but pub Rust hook that would ship as public API able to replace the forward pass), #1386 (route_batch switched from self.route to self._route, silently bypassing subclass overrides the docstring promises), and nine stacked port PRs that conflict with each other and need linearizing (#1377, #1378, #1380, #1382, #1383, #1385, #1387, #1388, #1389, #1390). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Holding this one, and I want to be clear that it is not a rejection of the idea. The measurement is good and the win is real: removing 8 of 107 batch passes and about a third of the padded tokens, with the Rust lane going from roughly 69 to 55 minutes and .NET from 45 to 37. Those lanes are the slowest thing in the release and this is the first change to move them for the right reason, by doing less work rather than by raising a budget. Two things need to change before I merge it. First, the Rust seam. What I am asking for is a Second, the One more thing worth confirming while you are in there: with the hook behind a feature, please check that the lane still reports the same passed count rather than silently skipping the tests that need it. Happy to merge as soon as the seam is gated. The performance work itself I have no notes on. |
fed15d1 to
5a086b1
Compare
…replays A test-only memoiser around the engine's forward pass (an internal hook, off by default) shares the ONNX run between batch cases whose collated input tensors are byte-identical. Every case still tokenizes, plans, collates, gates and decodes on its own and is compared with its own golden.
5a086b1 to
377fa81
Compare
Runs each byte-identical forward pass once in the batch replays, in both SDKs, with no loss of coverage.
.NET: an internalForwardInterceptor; Rust: a#[doc(hidden)]set_forward_hook).Run/runare unchanged.Tested on our fork, on top of #1373 and #1375: the longest .NET cell went from 45 to 37 min and the longest Rust shard from 69 to 55 min; every test and skip-guard step passed. Model-free: .NET 2177, Rust 377 passed.