Skip to content

feat(dotnet): port the custom labels of noul questions - #1382

Open
kevin-gatimu wants to merge 2 commits into
NandhaKishorM:mainfrom
kevin-gatimu:feat/dotnet-noul-labels
Open

kevin-gatimu wants to merge 2 commits into
NandhaKishorM:mainfrom
kevin-gatimu:feat/dotnet-noul-labels

Conversation

@kevin-gatimu

Copy link
Copy Markdown
Contributor

Depends on #1377 (option_order); only the last commit is new.

Ports the custom labels of Python's noul questions to the .NET SDK.

  • NoulQuestion.Labels, WithLabels and WithoutLabels. Additive API.
  • Validated and stripped as _resolve_noul_labels does, with Python's text, at prediction; rendered as <label>: <criterion or default text>; part of the router's batch grouping key. The answer is unchanged (P(true)).
  • Goldens regenerated (labels with and without criteria, padded, with option_order, batch, long and error cases), shared with the Rust PR.

Tests: model-free 2224; lanes english 616, multilingual 628, typed-decisions 628, router 4; 0 failed, 0 skipped.

Copilot AI balanced review requested due to automatic review settings October 26, 2026 07:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The changes require final human review because they are too complex or risky for automated approval.

No open issues 🎉

What changed in this PR

Ports Python’s custom noul labels to the .NET SDK, building on #1377’s option-order support while preserving the answer’s P(true) meaning.

Changes:

  • Adds label and option-order APIs with Python-compatible validation, rendering and probability mapping.
  • Includes labels and ordering in batch grouping, with shortlist handling for narrowed options.
  • Extends tests, golden generation and SDK documentation.

Verification: Checked 153 indexed feature fixtures for consistent names and success/error shapes; no inconsistencies were found. Full SDK and checkpoint-backed tests were not independently run.

File Description
laya-dotnet/​tools/​dump_golden.py Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​TokenizerParityTests.cs Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​ShortlistEndToEndTests.cs Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​PredictParityTests.cs Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​NoulLabelsTests.cs Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​GoldenData.cs Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​meta.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​index.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_score.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_score_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_out_of_range.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_noul_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_negative.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_id_repr.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_first_bad_reported.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_empty.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_duplicate.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_option_order_choice_labels_list.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_order_checked_first.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_non_string_true.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_non_string_false.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_id_repr.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_first_bad_reported.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_equal.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_equal_after_strip.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_blank_false.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_noul_labels_blank_after_strip.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_option_order_short.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_option_order_mixed.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_option_order_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_option_order_before_stride.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_noul_labels_short.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_noul_labels_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_long_noul_labels_before_stride.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_option_order_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_option_order_empty_states_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_noul_labels_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​typed-decisions/​case_batch_noul_labels_empty_states_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​meta.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​index.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_score_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_out_of_range.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_noul_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_negative.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_id_repr.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_first_bad_reported.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_empty.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_duplicate.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_option_order_choice_labels_list.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_order_checked_first.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_non_string_true.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_non_string_false.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_id_repr.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_first_bad_reported.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_equal.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_equal_after_strip.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_blank_false.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_noul_labels_blank_after_strip.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_option_order_short.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_option_order_mixed.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_option_order_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_option_order_before_stride.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_noul_labels_short.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_noul_labels_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_long_noul_labels_before_stride.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_option_order_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_option_order_empty_states_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_noul_labels_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​multilingual/​case_batch_noul_labels_empty_states_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​meta.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​index.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_score.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_score_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_out_of_range.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_noul_wrong_length.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_negative.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_id_repr.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_first_bad_reported.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_empty.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_duplicate.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_option_order_choice_labels_list.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_order_checked_first.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_non_string_true.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_non_string_false.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_id_repr.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_first_bad_reported.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_equal.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_equal_after_strip.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_blank_false.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_noul_labels_blank_after_strip.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_option_order_short.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_option_order_mixed.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_option_order_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_option_order_before_stride.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_noul_labels_short.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_noul_labels_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_long_noul_labels_before_stride.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_option_order_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_option_order_empty_states_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_noul_labels_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​golden/​english/​case_batch_noul_labels_empty_states_invalid.json Updated as part of this pull request.
laya-dotnet/​tests/​Laya.Tests/​BatchPredictionTests.cs Updated as part of this pull request.
laya-dotnet/​src/​Laya/​SequenceBuilder.cs Updated as part of this pull request.
laya-dotnet/​src/​Laya/​Questions.cs Updated as part of this pull request.
laya-dotnet/​src/​Laya/​LayaShortlist.cs Updated as part of this pull request.
laya-dotnet/​src/​Laya/​LayaRouter.cs Updated as part of this pull request.
laya-dotnet/​src/​Laya/​LayaEngine.cs Updated as part of this pull request.
laya-dotnet/​README.md Updated as part of this pull request.
laya-dotnet/​CHANGELOG.md Updated as part of this pull request.

Review effort: Balanced

@NandhaKishorM

Copy link
Copy Markdown
Owner

Held for integration, not on merit. This is one of nine PRs in the same chain that were each opened against main independently, so they conflict with each other as soon as any one lands. I merged #1370, #1371, #1373, #1374, #1375, #1379 and #1384 clean, and then the rest of the stack collided.

The full conflict map and what I need is on #1377, so I am not repeating it here: short version, pick an order, rebase each onto the one before it, and regenerate golden/*/index.json rather than hand-merging the appended case lists.

For this one specifically, the conflicting paths against the integration branch were:

CHANGELOG.md, README.md, src/Laya/LayaEngine.cs, src/Laya/SequenceBuilder.cs,
tools/dump_golden.py, tools/golden_cases.py,
tests/Laya.Tests/golden/{english,multilingual,typed-decisions}/index.json

NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@kevin-gatimu
kevin-gatimu force-pushed the feat/dotnet-noul-labels branch from fb44d28 to a17f70e Compare October 29, 2026 18:26
@kevin-gatimu

Copy link
Copy Markdown
Contributor Author

Rebased onto #1377 (now on 0.4.17) with goldens regenerated; conflicts resolved. The diff will show #1377's commit until that one merges.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Same as #1377, so the short version: both of your red checks are mine, not yours.

check whose
tests (windows) mine, fixed on main in 54b3558
onnx (clang++, tsan) mine, fixed on main in 118d361

The first was a test feeding a 200000 character document through an environment variable, which Windows caps at 32767, so it failed before reaching the code under test. The second had been red since the C++ ONNX backend landed, hidden by continue-on-error: ThreadSanitizer cannot see inside the prebuilt ONNX Runtime and read its thread pool as 342 data races. That job is now 479 of 479 with zero warnings.

Both fixes are on main as of today. This PR touches only laya-dotnet (175 files), so neither failure is attributable to it.

Please rebase on current main and the lane should go green. I will review the port itself then, since I would rather read 175 files once against a clean CI than twice.

One thing worth flagging now rather than after the rebase: your base is fe9a6c35, which is 0.4.17. Main has since taken 0.4.18 and 0.4.19, including changes to the MCP route tools, laya/serve.py and laya/router.py. None of those are in laya-dotnet, so I do not expect conflicts, but the parity goldens your port is checked against have moved, so rebase rather than merge forward if you can, and let the regeneration step tell us the truth.

ChoiceQuestion, ScoreQuestion and NoulQuestion gain OptionOrder and
WithOptionOrder (a permutation of the option slots). Options are encoded
and marked in slot order, the softmax runs in slot order and the
probabilities are mapped back to option order before the answer is
built, as laya/common.py and Agent do. The order is validated with
Python's text where Python validates it, joins the router's batch
grouping key, and is dropped with Python's warning when the shortlist
narrows the labels. Existing signatures are unchanged.

Goldens come from tools/regen_golden.py; the case lists are shared
verbatim with the Rust SDK.
NoulQuestion gains Labels, WithLabels and WithoutLabels: the false and
true option labels Python's noul questions accept. They are validated
and stripped as laya/common.py _resolve_noul_labels does, with Python's
text, at prediction as Agent._check_question does; rendered as
"<label>: <criterion or default text>"; part of the router's batch
grouping key. The answer is unchanged (P(true)). Existing signatures are
unchanged.

Goldens come from tools/regen_golden.py; the case lists are shared
verbatim with the Rust SDK.
@kevin-gatimu
kevin-gatimu force-pushed the feat/dotnet-noul-labels branch from 43ff93f to f8849a7 Compare November 1, 2026 16:34

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants