Repository navigation
ci(rust): shard the parity lane per checkpoint - #1375
Conversation
One job per checkpoint (english, multilingual, typed-decisions) plus a router cell, selected with LAYA_TEST_CHECKPOINTS. A deselected checkpoint is skipped silently; a selected one with missing artifacts or goldens still prints skip:, which the guard rejects, and an unknown name fails. Every cell regenerates its own goldens from Python at this commit and has its own --min-passed floor and timeout. Exports are cached per checkpoint, and a runner hardware step is logged. Requested on NandhaKishorM#1359.
There was a problem hiding this comment.
🟡 Changes recommended
A router fixture check still reads deselected checkpoints, and library-only runs silently accept unknown checkpoint names.
2 open issues
What changed in this PR
This PR splits the Rust SDK’s Python-parity CI lane into checkpoint-specific jobs and a router job, reducing wall time while retaining full parity coverage.
Changes:
- Adds per-job timeouts, pass-count floors, export caches, and runner diagnostics.
- Introduces
LAYA_TEST_CHECKPOINTSfiltering across model-backed and golden-backed tests. - Documents shard behavior and adds skip-guard regression coverage.
| File | Description |
|---|---|
laya-rust/tools/test_ci_tools.py |
Tests shard-specific floors and missing-artifact rejection. |
laya-rust/tests/tokenizer_parity_tests.rs |
Filters tokenizer checks by checkpoint selection. |
laya-rust/tests/shortlist_end_to_end_tests.rs |
Limits shortlist checks to selected checkpoints. |
laya-rust/tests/shape_sweep_tests.rs |
Filters fused and split-layout shape sweeps. |
laya-rust/tests/router_end_to_end_tests.rs |
Gates model-backed router tests on both checkpoints. |
laya-rust/tests/python_json_tests.rs |
Filters golden checks and adjusts comparison counts. |
laya-rust/tests/predict_parity_tests.rs |
Applies selection to prediction parity tests. |
laya-rust/tests/predict_long_tests.rs |
Applies selection to long-request tests. |
laya-rust/tests/predict_batch_tests.rs |
Filters batch parity checks by checkpoint. |
laya-rust/tests/min_confidence_tests.rs |
Filters recorded confidence cases. |
laya-rust/tests/golden_questions_tests.rs |
Filters question-rendering goldens. |
laya-rust/tests/common/test_artifacts.rs |
Adds checkpoint selection parsing and validation. |
laya-rust/tests/common/engines.rs |
Adds selection-aware iteration and loading. |
laya-rust/tests/collator_tests.rs |
Gates multilingual tensor fixtures. |
laya-rust/tests/calibration_tests.rs |
Gates multilingual configuration fixtures. |
laya-rust/src/batch_prediction.rs |
Filters library parity cases by checkpoint. |
laya-rust/README.md |
Documents sharding, caching, and checkpoint selection. |
.github/workflows/rust.yml |
Defines four parity jobs with individual budgets and caches. |
Review effort: Balanced
…keep the sample-input check in the router cell The in-crate planning test now fails on an unknown LAYA_TEST_CHECKPOINTS name, as tests/common does, so a library-only run cannot pass by ignoring a typo. sample_inputs_match_recorded_model_case_inputs reads the english and multilingual goldens, so it runs only where both are selected and regenerated.
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window attribution preserved through end hooks (#1369), RemoteRouter identifying itself with a configurable User-Agent (#1396), ten more inputs refused where they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404, #1405, #1406), per-language temperatures and per-call budgets in .NET and Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle, SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation (#1399, #1407), and parity-lane sharding that replaces the budget headroom added for 0.4.16 (#1373, #1375). Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean. Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0: Java 20 families match laya (9 unverified), UnicodeTables matches CPython 15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match, both laya-cpp .inc files match. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.17 only. Security audit of the merge diff, 362766 added lines over 480 files: zero obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm install hooks unchanged, and no third-party host in any added line (example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12 dynamic-exec hits are the C++ test harness invoking its own checker through sys.executable, the Go sample generator reading laya's own example by AST at development time, and one comment in rust.yml. Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)] but pub Rust hook that would ship as public API able to replace the forward pass), #1386 (route_batch switched from self.route to self._route, silently bypassing subclass overrides the docstring promises), and nine stacked port PRs that conflict with each other and need linearizing (#1377, #1378, #1380, #1382, #1383, #1385, #1387, #1388, #1389, #1390). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged. Sharding per checkpoint is the right call, and the reasoning in the matrix comment is the part I appreciated most: the serial constraint is per checkpoint rather than across them, so one job per checkpoint is correct rather than merely faster. The per-cell I had raised the single lane budget to 240 minutes shortly before this landed, after the old lane timed out at 121 minutes against a 120 minute cap. Your version supersedes that, and the per-cell budgets are better than one pot the next parity PR spends. One cell needed a correction, now in dc0916c. The The reason this is not a one time cost is worth noting in case it bites again. A cache is only saved by a job that succeeds, and the export cache key includes Your comment about a hit limit being recorded as CANCELLED is exactly right and is the trap I keep having to re-derive. The word alone cannot distinguish a superseded run from an exhausted budget; only the arithmetic on |

Shards the Rust parity lane per checkpoint, as suggested on #1359.
LAYA_TEST_CHECKPOINTS. A deselected checkpoint is skipped silently; a selected one with missing artifacts or goldens still printsskip:, which the guard rejects, and an unknown name fails.--min-passedfloor and timeout. Exports are cached per checkpoint, and the runner's CPU is logged.Tested on our fork: english 26 min, multilingual 25, typed-decisions 69, router 10; 371, 371, 371 and 3 passed with no skips; 95 min wall at
max-parallel2. The unsharded lane took 143 min on the same runners.If you'd also like it, the split layout could replay a named subset (every question type, padded batches, max-length and long-document cases) instead of every case. That cut the Rust lane from 143 to 74 minutes on our fork. It's on our fork as
test/split-layout-subset; I'll open it only if you want it.