Skip to content

feat(dotnet): port predict_batch and router batch APIs - #1358

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
kevin-gatimu:feat/dotnet-predict-batch
Oct 25, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
kevin-gatimu:feat/dotnet-predict-batch

Conversation

@kevin-gatimu

Copy link
Copy Markdown
Contributor

Ports Python's ONNXAgent.predict_batch and Router.predict_batch / route_batch to the .NET SDK.

  • LayaEngine.PredictBatch (LayaBatchOptions: BatchSize, SortByLength), LayaRouter.PredictBatch, LayaRouter.RouteBatch. Additive API.
  • Behaviour change, matching system_one = predict_batch([state]): Predict refuses a null state with Python's text, returns empty answers for an empty question set, and reports the usage diagnostics.
  • Goldens regenerated at 0.4.14 (29 batch cases, 10 route_batch cases), shared with the Rust PR.

Tests: model-free 1935 passed; lanes english 434, multilingual 446, typed-decisions 446, router 4; 0 failed, 0 skipped.

Add LayaEngine.PredictBatch (ILayaBatchPredictor, LayaBatchOptions with
BatchSize and SortByLength), LayaRouter.PredictBatch and
LayaRouter.RouteBatch over LayaBatchRequest, ported from Python
ONNXAgent.predict_batch and Router.predict_batch / route_batch: batch_size
chunking, sort_by_length windows of eight batches, grouping by checkpoint
then question schema, input order restored, per-request lang_guess.

Plain Predict now behaves as Python's system_one (predict_batch([state])):
a null state is refused with Python's TypeError text, an empty question
set returns empty answers with zero usage, and the usage carries the
truncation diagnostics. The unknown-model error carries Python's text.

Goldens come from tools/regen_golden.py at Python 0.4.14: 29 batch
cases, 2 batch error cases, 10 route_batch cases and 6 error cases, shared
verbatim with the Rust SDK.
Copilot AI balanced review requested due to automatic review settings October 24, 2026 18:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The shared inference refactor and checkpoint lifetime handling warrant human confirmation of model-backed parity across fused and split layouts.

No open issues 🎉

What changed in this PR

Ports Python’s batch prediction and routing APIs to the .NET SDK, enabling shared local inference while preserving request order.

Changes:

  • Adds engine and router batch APIs with batch-size limits, length-based sorting, and question-schema grouping.
  • Aligns single-state prediction with Python’s null-state validation, empty-question behavior, and usage diagnostics.
  • Adds parity tests and Python-generated batch fixtures, and documents the APIs and remaining limitations.
File Description
laya-dotnet/​tools/​routing_cases.py Defines batch routing and validation cases.
laya-dotnet/​tools/​golden_cases.py Defines batch inference scenarios and edge cases.
laya-dotnet/​tools/​dump_routing_golden.py Generates batch routing reference data.
laya-dotnet/​tools/​dump_golden.py Records batch predictions, token widths, and run shapes.
laya-dotnet/​tests/​Laya.Tests/​PredictParityTests.cs Checks ordinary prediction usage diagnostics.
laya-dotnet/​tests/​Laya.Tests/​PredictParityTests.Batch.cs Tests batch parity, planning, errors, and single-state behavior.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​meta.json Updates reference generator version.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​index.json Registers batch reference cases.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_sort_partial_window.json Records partial-window sorting results.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_single_question.json Records single-question batching results.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_options_exceed_head.json Records oversized-question rejection.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_one_state.json Records a single-state batch.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_none_state.json Records null-state rejection.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_empty_states.json Records empty-state-list behavior.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_empty_states_invalid_questions.json Records empty-batch validation bypass.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_empty_questions.json Records empty answers and zero usage.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_empty_questions_with_options.json Records empty questions with batching options.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_empty_both.json Records empty states and questions.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_duplicate_states.json Records repeated-state results.
laya-dotnet/​tests/​Laya.Tests/​golden/​routing/​meta.json Updates version and batch routing counts.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​meta.json Updates reference generator version.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​index.json Registers batch reference cases.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_sort_partial_window.json Records partial-window sorting results.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_single_question.json Records single-question batching results.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_options_exceed_head.json Records oversized-question rejection.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_one_state.json Records a single-state batch.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_none_state.json Records null-state rejection.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_empty_states.json Records empty-state-list behavior.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_empty_states_invalid_questions.json Records empty-batch validation bypass.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_empty_questions.json Records empty answers and zero usage.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_empty_questions_with_options.json Records empty questions with batching options.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_empty_both.json Records empty states and questions.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_duplicate_states.json Records repeated-state results.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​meta.json Updates reference generator version.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​index.json Registers batch reference cases.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_sort_partial_window.json Records partial-window sorting results.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_single_question.json Records single-question batching results.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_options_exceed_head.json Records oversized-question rejection.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_one_state.json Records a single-state batch.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_none_state.json Records null-state rejection.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_mixed.json Records mixed state types and lengths.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_empty_states.json Records empty-state-list behavior.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_empty_states_invalid_questions.json Records empty-batch validation bypass.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_empty_questions.json Records empty answers and zero usage.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_empty_questions_with_options.json Records empty questions with batching options.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_empty_both.json Records empty states and questions.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_duplicate_states.json Records repeated-state results.
laya-dotnet/​src/​Laya/​LayaRouter.cs Adds batch routing, schema grouping, and shared checkpoint leases.
laya-dotnet/​src/​Laya/​LayaEngine.cs Adds batch inference and aligns single-state behavior.
laya-dotnet/​src/​Laya/​LayaBatch.cs Defines batch APIs, validation, and chunk planning.
laya-dotnet/​README.md Documents batch usage, behavior, and limitations.
laya-dotnet/​CHANGELOG.md Records new APIs and compatibility changes.

Review effort: Balanced

NandhaKishorM added a commit that referenced this pull request Oct 25, 2026
8 PRs. Rust and .NET batch prediction and router batch APIs (#1358,
#1359), the min_confidence abstention gate and jev_confidence in both
ports (#1362, #1363), and four inputs refused where they enter instead
of failing later (#1365, #1366, #1367, #1368).

Verified before tagging: all 92 suites pass, 612 tests. ruff clean on
both lists. Java fixture, Unicode table and language table gates clean;
three Go tables_gen.go match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.16 only.

Security audit of the merge diff, 180330 added lines over 349 files:
zero dynamic exec, zero network calls, zero URLs, zero base64 or
obfuscation, zero secrets in the diff or the tracked tree, zero
laya-max boundary hits, no writes outside the test temp dir, the
dependency surface untouched, no npm install hooks, no binaries, every
new file 100644, and 20 of 20 actions SHA-pinned with codeql-action
init, autobuild and analyze all on 4.39.1.

Held from this release: #1364 (correct in Python but it moves a
cross-port contract Decisions.java reproduces deliberately and
structured.ts shares), #1324 (real table regression), and seven
dependency and action bumps for a pass of their own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM
NandhaKishorM merged commit 7af2d4c into NandhaKishorM:main Oct 25, 2026
50 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged, in 0.4.16.

Grouping by checkpoint and question schema and then restoring input order is the right shape, and it is the half of a batch port that is easy to get subtly wrong: a port that groups but returns results in group order looks correct on a single-checkpoint batch and silently mismatches on a mixed one. Reporting a predictor that has no predict_batch rather than emulating one with a loop is also correct. An emulated batch would answer, so nobody would notice the sharing was gone.

The goldens being recorded from the Python package rather than written by hand is what makes this reviewable at all, given there are now around 50 batch cases per checkpoint.

One consequence worth knowing, which is not a defect in this PR: the replay suites are expensive, and on the Rust side the equivalent tests pushed that lane past its 120-minute budget. I have raised it and written up the real fix on #1359. The .NET lane came back clean at 5 of 5 jobs, so nothing is needed here.

NandhaKishorM added a commit that referenced this pull request Oct 26, 2026
Same class of failure rust.yml just had, on the other port. The
typed-decisions cell hit its 60-minute wall on a478221 and GitHub
recorded it as CANCELLED, which reads like a superseded run and not a
budget failure. Its test step had been running 34 minutes and printed no
result summary, so the cell verified nothing while the lane stayed green
on continue-on-error.

Measured on a478221, where #1358 added about 50 batch cases per
checkpoint: english 52m passed, typed-decisions 60m killed mid-run,
multilingual 13m passed, router 26m (failed on a transient setup-dotnet
feed error resolving the global.json pin, not on time: the same pin
resolved four times earlier in that run).

english landing 8 minutes under the wall is the same warning
typed-decisions already acted on, so this raises the cell rather than
only the one that fell over.

Per-checkpoint sharding is already in place, so the next lever is
splitting the replay classes off the tokenizer and shape sweeps rather
than more headroom.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants