Repository navigation
feat(dotnet): port predict_batch and router batch APIs - #1358
Conversation
Add LayaEngine.PredictBatch (ILayaBatchPredictor, LayaBatchOptions with BatchSize and SortByLength), LayaRouter.PredictBatch and LayaRouter.RouteBatch over LayaBatchRequest, ported from Python ONNXAgent.predict_batch and Router.predict_batch / route_batch: batch_size chunking, sort_by_length windows of eight batches, grouping by checkpoint then question schema, input order restored, per-request lang_guess. Plain Predict now behaves as Python's system_one (predict_batch([state])): a null state is refused with Python's TypeError text, an empty question set returns empty answers with zero usage, and the usage carries the truncation diagnostics. The unknown-model error carries Python's text. Goldens come from tools/regen_golden.py at Python 0.4.14: 29 batch cases, 2 batch error cases, 10 route_batch cases and 6 error cases, shared verbatim with the Rust SDK.
There was a problem hiding this comment.
🔵 Needs a closer look
The shared inference refactor and checkpoint lifetime handling warrant human confirmation of model-backed parity across fused and split layouts.
No open issues 🎉
What changed in this PR
Ports Python’s batch prediction and routing APIs to the .NET SDK, enabling shared local inference while preserving request order.
Changes:
- Adds engine and router batch APIs with batch-size limits, length-based sorting, and question-schema grouping.
- Aligns single-state prediction with Python’s null-state validation, empty-question behavior, and usage diagnostics.
- Adds parity tests and Python-generated batch fixtures, and documents the APIs and remaining limitations.
| File | Description |
|---|---|
| laya-dotnet/tools/routing_cases.py | Defines batch routing and validation cases. |
| laya-dotnet/tools/golden_cases.py | Defines batch inference scenarios and edge cases. |
| laya-dotnet/tools/dump_routing_golden.py | Generates batch routing reference data. |
| laya-dotnet/tools/dump_golden.py | Records batch predictions, token widths, and run shapes. |
| laya-dotnet/tests/Laya.Tests/PredictParityTests.cs | Checks ordinary prediction usage diagnostics. |
| laya-dotnet/tests/Laya.Tests/PredictParityTests.Batch.cs | Tests batch parity, planning, errors, and single-state behavior. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/meta.json | Updates reference generator version. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/index.json | Registers batch reference cases. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_sort_partial_window.json | Records partial-window sorting results. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_single_question.json | Records single-question batching results. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_options_exceed_head.json | Records oversized-question rejection. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_one_state.json | Records a single-state batch. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_none_state.json | Records null-state rejection. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_empty_states.json | Records empty-state-list behavior. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_empty_states_invalid_questions.json | Records empty-batch validation bypass. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_empty_questions.json | Records empty answers and zero usage. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_empty_questions_with_options.json | Records empty questions with batching options. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_empty_both.json | Records empty states and questions. |
| laya-dotnet/tests/Laya.Tests/golden/typed-decisions/case_batch_duplicate_states.json | Records repeated-state results. |
| laya-dotnet/tests/Laya.Tests/golden/routing/meta.json | Updates version and batch routing counts. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/meta.json | Updates reference generator version. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/index.json | Registers batch reference cases. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_sort_partial_window.json | Records partial-window sorting results. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_single_question.json | Records single-question batching results. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_options_exceed_head.json | Records oversized-question rejection. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_one_state.json | Records a single-state batch. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_none_state.json | Records null-state rejection. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_empty_states.json | Records empty-state-list behavior. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_empty_states_invalid_questions.json | Records empty-batch validation bypass. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_empty_questions.json | Records empty answers and zero usage. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_empty_questions_with_options.json | Records empty questions with batching options. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_empty_both.json | Records empty states and questions. |
| laya-dotnet/tests/Laya.Tests/golden/multilingual/case_batch_duplicate_states.json | Records repeated-state results. |
| laya-dotnet/tests/Laya.Tests/golden/english/meta.json | Updates reference generator version. |
| laya-dotnet/tests/Laya.Tests/golden/english/index.json | Registers batch reference cases. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_sort_partial_window.json | Records partial-window sorting results. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_single_question.json | Records single-question batching results. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_options_exceed_head.json | Records oversized-question rejection. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_one_state.json | Records a single-state batch. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_none_state.json | Records null-state rejection. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_mixed.json | Records mixed state types and lengths. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_empty_states.json | Records empty-state-list behavior. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_empty_states_invalid_questions.json | Records empty-batch validation bypass. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_empty_questions.json | Records empty answers and zero usage. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_empty_questions_with_options.json | Records empty questions with batching options. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_empty_both.json | Records empty states and questions. |
| laya-dotnet/tests/Laya.Tests/golden/english/case_batch_duplicate_states.json | Records repeated-state results. |
| laya-dotnet/src/Laya/LayaRouter.cs | Adds batch routing, schema grouping, and shared checkpoint leases. |
| laya-dotnet/src/Laya/LayaEngine.cs | Adds batch inference and aligns single-state behavior. |
| laya-dotnet/src/Laya/LayaBatch.cs | Defines batch APIs, validation, and chunk planning. |
| laya-dotnet/README.md | Documents batch usage, behavior, and limitations. |
| laya-dotnet/CHANGELOG.md | Records new APIs and compatibility changes. |
Review effort: Balanced
8 PRs. Rust and .NET batch prediction and router batch APIs (#1358, #1359), the min_confidence abstention gate and jev_confidence in both ports (#1362, #1363), and four inputs refused where they enter instead of failing later (#1365, #1366, #1367, #1368). Verified before tagging: all 92 suites pass, 612 tests. ruff clean on both lists. Java fixture, Unicode table and language table gates clean; three Go tables_gen.go match. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.16 only. Security audit of the merge diff, 180330 added lines over 349 files: zero dynamic exec, zero network calls, zero URLs, zero base64 or obfuscation, zero secrets in the diff or the tracked tree, zero laya-max boundary hits, no writes outside the test temp dir, the dependency surface untouched, no npm install hooks, no binaries, every new file 100644, and 20 of 20 actions SHA-pinned with codeql-action init, autobuild and analyze all on 4.39.1. Held from this release: #1364 (correct in Python but it moves a cross-port contract Decisions.java reproduces deliberately and structured.ts shares), #1324 (real table regression), and seven dependency and action bumps for a pass of their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged, in 0.4.16. Grouping by checkpoint and question schema and then restoring input order is the right shape, and it is the half of a batch port that is easy to get subtly wrong: a port that groups but returns results in group order looks correct on a single-checkpoint batch and silently mismatches on a mixed one. Reporting a predictor that has no The goldens being recorded from the Python package rather than written by hand is what makes this reviewable at all, given there are now around 50 batch cases per checkpoint. One consequence worth knowing, which is not a defect in this PR: the replay suites are expensive, and on the Rust side the equivalent tests pushed that lane past its 120-minute budget. I have raised it and written up the real fix on #1359. The .NET lane came back clean at 5 of 5 jobs, so nothing is needed here. |
Same class of failure rust.yml just had, on the other port. The typed-decisions cell hit its 60-minute wall on a478221 and GitHub recorded it as CANCELLED, which reads like a superseded run and not a budget failure. Its test step had been running 34 minutes and printed no result summary, so the cell verified nothing while the lane stayed green on continue-on-error. Measured on a478221, where #1358 added about 50 batch cases per checkpoint: english 52m passed, typed-decisions 60m killed mid-run, multilingual 13m passed, router 26m (failed on a transient setup-dotnet feed error resolving the global.json pin, not on time: the same pin resolved four times earlier in that run). english landing 8 minutes under the wall is the same warning typed-decisions already acted on, so this raises the cell rather than only the one that fell over. Per-checkpoint sharding is already in place, so the next lever is splitting the replay classes off the tokenizer and shape sweeps rather than more headroom. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ports Python's
ONNXAgent.predict_batchandRouter.predict_batch/route_batchto the .NET SDK.LayaEngine.PredictBatch(LayaBatchOptions:BatchSize,SortByLength),LayaRouter.PredictBatch,LayaRouter.RouteBatch. Additive API.system_one = predict_batch([state]):Predictrefuses a null state with Python's text, returns empty answers for an empty question set, and reports the usage diagnostics.Tests: model-free 1935 passed; lanes english 434, multilingual 446, typed-decisions 446, router 4; 0 failed, 0 skipped.