Skip to content

ci(rust): shard the parity lane per checkpoint - #1375

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
kevin-gatimu:ci/rust-shard-parity
Oct 28, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
kevin-gatimu:ci/rust-shard-parity

Conversation

@kevin-gatimu

Copy link
Copy Markdown
Contributor

Shards the Rust parity lane per checkpoint, as suggested on #1359.

  • One job per checkpoint (english, multilingual, typed-decisions) plus a router cell, selected with LAYA_TEST_CHECKPOINTS. A deselected checkpoint is skipped silently; a selected one with missing artifacts or goldens still prints skip:, which the guard rejects, and an unknown name fails.
  • Every cell regenerates its own goldens from Python at this commit and has its own --min-passed floor and timeout. Exports are cached per checkpoint, and the runner's CPU is logged.
  • Debug profile kept: release measured about 2% faster, since the time is ONNX Runtime inference.

Tested on our fork: english 26 min, multilingual 25, typed-decisions 69, router 10; 371, 371, 371 and 3 passed with no skips; 95 min wall at max-parallel 2. The unsharded lane took 143 min on the same runners.

If you'd also like it, the split layout could replay a named subset (every question type, padded batches, max-length and long-document cases) instead of every case. That cut the Rust lane from 143 to 74 minutes on our fork. It's on our fork as test/split-layout-subset; I'll open it only if you want it.

One job per checkpoint (english, multilingual, typed-decisions) plus a router cell, selected with LAYA_TEST_CHECKPOINTS. A deselected checkpoint is skipped silently; a selected one with missing artifacts or goldens still prints skip:, which the guard rejects, and an unknown name fails. Every cell regenerates its own goldens from Python at this commit and has its own --min-passed floor and timeout. Exports are cached per checkpoint, and a runner hardware step is logged. Requested on NandhaKishorM#1359.
Copilot AI balanced review requested due to automatic review settings October 26, 2026 02:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A router fixture check still reads deselected checkpoints, and library-only runs silently accept unknown checkpoint names.

2 open issues
What changed in this PR

This PR splits the Rust SDK’s Python-parity CI lane into checkpoint-specific jobs and a router job, reducing wall time while retaining full parity coverage.

Changes:

  • Adds per-job timeouts, pass-count floors, export caches, and runner diagnostics.
  • Introduces LAYA_TEST_CHECKPOINTS filtering across model-backed and golden-backed tests.
  • Documents shard behavior and adds skip-guard regression coverage.
File Description
laya-rust/​tools/​test_ci_tools.py Tests shard-specific floors and missing-artifact rejection.
laya-rust/​tests/​tokenizer_parity_tests.rs Filters tokenizer checks by checkpoint selection.
laya-rust/​tests/​shortlist_end_to_end_tests.rs Limits shortlist checks to selected checkpoints.
laya-rust/​tests/​shape_sweep_tests.rs Filters fused and split-layout shape sweeps.
laya-rust/​tests/​router_end_to_end_tests.rs Gates model-backed router tests on both checkpoints.
laya-rust/​tests/​python_json_tests.rs Filters golden checks and adjusts comparison counts.
laya-rust/​tests/​predict_parity_tests.rs Applies selection to prediction parity tests.
laya-rust/​tests/​predict_long_tests.rs Applies selection to long-request tests.
laya-rust/​tests/​predict_batch_tests.rs Filters batch parity checks by checkpoint.
laya-rust/​tests/​min_confidence_tests.rs Filters recorded confidence cases.
laya-rust/​tests/​golden_questions_tests.rs Filters question-rendering goldens.
laya-rust/​tests/​common/​test_artifacts.rs Adds checkpoint selection parsing and validation.
laya-rust/​tests/​common/​engines.rs Adds selection-aware iteration and loading.
laya-rust/​tests/​collator_tests.rs Gates multilingual tensor fixtures.
laya-rust/​tests/​calibration_tests.rs Gates multilingual configuration fixtures.
laya-rust/​src/​batch_prediction.rs Filters library parity cases by checkpoint.
laya-rust/​README.md Documents sharding, caching, and checkpoint selection.
.github/​workflows/​rust.yml Defines four parity jobs with individual budgets and caches.

Review effort: Balanced

Comment thread laya-rust/src/batch_prediction.rs Resolved
…keep the sample-input check in the router cell

The in-crate planning test now fails on an unknown LAYA_TEST_CHECKPOINTS name, as tests/common does, so a library-only run cannot pass by ignoring a typo. sample_inputs_match_recorded_model_case_inputs reads the english and multilingual goldens, so it runs only where both are selected and regenerated.
@NandhaKishorM
NandhaKishorM merged commit f933a55 into NandhaKishorM:main Oct 28, 2026
53 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. Sharding per checkpoint is the right call, and the reasoning in the matrix comment is the part I appreciated most: the serial constraint is per checkpoint rather than across them, so one job per checkpoint is correct rather than merely faster. The per-cell min-passed floor closes the hole where a cell that loads nothing would pass by scoring nothing.

I had raised the single lane budget to 240 minutes shortly before this landed, after the old lane timed out at 121 minutes against a 120 minute cap. Your version supersedes that, and the per-cell budgets are better than one pot the next parity PR spends.

One cell needed a correction, now in dc0916c. The router cell selects two checkpoints, so on a cold export cache it pays for both before a single test runs, and 30 minutes does not cover that. It was CANCELLED 29m22s into the export step. Measured cold on that same run, from the sibling cells that did finish: english 23m58s and multilingual 8m50s, so about 33 minutes of export alone plus about a minute of dependency install. I moved it to 60.

The reason this is not a one time cost is worth noting in case it bites again. A cache is only saved by a job that succeeds, and the export cache key includes laya/common.py, so any change to the model definition sends every cell back to the cold path. The cell has to be able to finish from cold, not just from warm.

Your comment about a hit limit being recorded as CANCELLED is exactly right and is the trap I keep having to re-derive. The word alone cannot distinguish a superseded run from an exhausted budget; only the arithmetic on started_at against the cell's limit can. This cell is the clearest example yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants