Skip to content

fix(cli): judge the --sort-by-length note per checkpoint group, not per file (#1111) - #1114

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
akushonkamen:fix/1111-sort-note-per-group
Oct 12, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
akushonkamen:fix/1111-sort-note-per-group

Conversation

@akushonkamen

Copy link
Copy Markdown
Contributor

What

--sort-by-length's no-op note is now judged where the reorder actually happens: per checkpoint/question group, not per file. Router.predict_batch takes an optional keyword-only diagnostics dict (laya/router.py:1602) and, when sort_by_length is set, reports under "sort_by_length" how many question groups reached an agent, how many of those could reorder, and the smallest / largest group size — counted at laya/router.py:1799-1813, where the groups are actually formed. run_batch (laya/cli.py) asks for the report only from routers whose predict_batch names the parameter (_takes_predict_diagnostics, laya/cli.py:310 — exact-name check, so a **kwargs forwarder cannot silently swallow the dict) and prints the note after the run from what actually reordered (_note_sort_report, laya/cli.py:328): the pre-run wording kept, sized by the group, when one group saw the whole file; a per-group accounting (laya/cli.py:345) when several groups could not. RemoteRouter.predict_batch accepts and ignores the same argument in the same position (laya/mcp/remote.py:206), keeping the two signatures identical. The hooks API quick-reference signature list gains the parameter (docs/hooks/api.md:303).

Untouched by design: laya/cli.py:371-373 still forwards sort_by_length only when the user asked for it (the stub/old-agent compatibility contract), and the grouping window semantics in the agents (laya/agent.py:1519, laya/onnx_agent.py:761) are not modified. An agent whose predict_batch predates the knob (#294) reports as unsorted, not as an error.

Why

Fixes #1111. The note's gate read the file: chunk = args.batch_size ... else len(lines) and if not 1 < chunk < len(lines) decided whether the warning printed at all. But the reorder itself switches per checkpoint and question group inside Router.predict_batch — each group calls its agent with the group's own state count against 1 < chunk < len(states) (laya/agent.py:1519, identically laya/onnx_agent.py:761). When auto-routing splits a file across several checkpoints, the two numbers diverge: nothing reorders anywhere while the file-level 1 < chunk < len(lines) holds, so the note stays silent — the exact silent same-speed result the note exists to prevent. The CLI cannot know the post-routing group sizes before the call, so the only honest place to judge is after the run, from the router itself.

How it was verified

Plain-script suites as the repo's CONTRIBUTING/AGENTS.md prescribes (these are not pytest-collectable); every command under timeout 300.

Full suites, final tree (re-run at submission time, all exit 0):

python3 tests/test_cli.py             # 192 passed, 0 failed
python3 tests/test_router.py          # 810 passed, 0 failed
python3 tests/test_hooks_api.py       # 548 passed, 0 failed
python3 tests/test_batch.py           # 78 passed, 0 failed
python3 tests/test_confidence.py      # 214 passed, 0 failed
python3 tests/test_cli_lang_guess.py  # 17 passed, 0 failed
python3 tests/test_blank_lang_routing.py  # 24 passed, 0 failed
python3 tests/test_evals_api.py       # 50 passed, 0 failed

Red → green (new tests only, sources stashed back to 6880462):

  • tests/test_cli.py: 189 passed, 2 failed — exactly the two new no-op-silence assertions
  • tests/test_router.py: TypeError: unexpected keyword argument 'diagnostics' at the new call — crash = red
  • tests/test_hooks_api.py: 547 passed, 1 failed — the signature-contract test (diagnostics parameter missing)
  • with the fix restored: all suites above green.

Repo gates:

uvx ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python3 -m compileall -q laya/ tests/                                     # OK, exit 0

Honest declarations:

  • The workorder's literal command timeout 300 pytest -k "sort_by_length or sort" was not run as written: no pytest in this environment (PEP 668 externally-managed interpreter; declined --break-system-packages), and per the repo's convention these suites are plain scripts — pytest collects no cases from them. The documented python3 tests/test_xxx.py commands above are what was run instead.
  • tests/test_mcp.py = SKIP (mcp extra not installed; the repo's own established skip, exit 0). tests/test_mcp_remote.py not run (needs pytest + mcp extra); instead the RemoteRouter.predict_batch ↔ Router.predict_batch signatures were compared parameter-by-parameter with a direct script, confirming diagnostics is accepted and left untouched (the 548-green test_hooks_api covers the same contract).
  • The issue's reproduction against a real checkpoint is not feasible here (empty HF cache); the red reproduction mirrors Router.predict_batch's grouping contract with a repo-style stub (/tmp/repro1111.py), and an end-to-end script with a real Router + stubbed load (/tmp/e2e1111.py) verifies the full chain prints the per-group note: ALL OK.

Disclosure: this change was prepared by an AI-assisted contributor (Claude Code), consistent with the claim comment on #1111.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes #1111

…er file (NandhaKishorM#1111)

The note's gate read the file: it printed only when --batch-size could not
clear 1 < chunk < len(lines). The reorder itself switches per checkpoint and
question group inside Router.predict_batch, on each group's own state count
(agent.py:1519, identically onnx_agent.py:761), so a file that auto-routes to
several checkpoints reordered nothing while the note stayed silent -- the
silent same-speed result the note exists to prevent (NandhaKishorM#1111).

Router.predict_batch now takes an optional keyword-only diagnostics dict and,
when sort_by_length is set, reports under "sort_by_length" how many question
groups reached an agent, how many could reorder, and the smallest / largest
group size -- counted where the groups are actually formed, so an agent whose
predict_batch predates the knob (NandhaKishorM#294) reports as unsorted, not as an error.
RemoteRouter.predict_batch accepts and ignores the same argument, keeping the
signatures identical (test_hooks_api). run_batch asks for the report only from
routers whose predict_batch names the parameter, notes after the run from what
actually reordered -- keeping the pre-run wording, sized by the group, when one
group saw the whole file -- and leaves routers without the parameter (stubs,
**kwargs forwarders) exactly as they were: same call arguments, same pre-run
note. The grouping semantics in the agents are untouched. The handwritten
signature list on the hooks API quick-reference page gains the parameter too.

Red (this tree's tests against 6880462 sources): tests/test_cli.py 190 passed,
2 failed -- the two no-op-silence assertions; tests/test_router.py died with
TypeError: unexpected keyword argument 'diagnostics'; tests/test_hooks_api.py
547 passed, 1 failed. Green: 192 / 810 / 548 / 78 passed, 0 failed (plus
confidence 214, lang-guess 17, blank-lang 24, evals-api 50), ruff and
compileall clean.
@akushonkamen
akushonkamen deleted the fix/1111-sort-note-per-group branch October 11, 2026 12:58
@akushonkamen
akushonkamen restored the fix/1111-sort-note-per-group branch October 11, 2026 12:58
@akushonkamen akushonkamen reopened this Oct 11, 2026
@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding this out of 0.4.3, not closing. One repository gate catches it, and the fix is small but it is a design decision that should be yours rather than mine.

Measured on the integration branch with this merged:

FAILED tests/test_serve.py::test_batch_call_controls_are_exactly_predict_batch_kwargs_or_refusal
  AssertionError: predict_batch() takes ['batch_size', 'diagnostics', 'hooks', 'hooks_raise',
  'hooks_timeout', 'min_confidence', 'on_predict_end', 'on_predict_start', 'sort_by_length'];
  serve declares ['batch_size', 'hooks', 'hooks_raise', 'hooks_timeout', 'min_confidence',
  'on_predict_end', 'on_predict_start', 'sort_by_length']
  Extra items in the left set: 'diagnostics'

That pin exists for this exact case. Its docstring: "A control added to Router.predict_batch must be placed on one side of the line. Either serve forwards it (BATCH_BODY_CALL_CONTROLS) or it refuses it (hooks_timeout, the one hook-execution arg that predict_batch takes). Silently ignoring a new kwarg is the failure mode this pin exists to catch." Adding the keyword-only diagnostics to predict_batch puts it on neither side.

The awkward part, and the reason I am not fixing it myself: BODY_REFUSALS is shared with the single path, and tests/test_serve.py:2415 pins BODY_CONTROLS | BODY_REFUSALS to cover Router.predict exactly, in both directions. Router.predict does not take diagnostics, so adding it to BODY_REFUSALS breaks that pin instead. It needs a batch-specific refusal set, something like BATCH_BODY_REFUSALS = BODY_REFUSALS + ("diagnostics",) with the batch pin reading that and the single pin unchanged.

Refusing rather than forwarding is clearly right, for what it is worth: diagnostics is an out-parameter a caller passes a dict into to collect a report, which is not a thing an HTTP body can express, so a request carrying diagnostics should get a clear 422 rather than have it quietly ignored.

The feature itself is good and the reasoning behind it is right. Judging the note per checkpoint and question group is where the reorder actually happens, and _takes_predict_diagnostics using an exact-name check so a **kwargs forwarder cannot silently swallow the dict is the detail that makes it trustworthy. Issue #1111 is open and this is the answer to it. Add the batch refusal set, rebase onto 0.4.3, and I will merge it next pass.

@NandhaKishorM
NandhaKishorM merged commit b0c496c into NandhaKishorM:main Oct 12, 2026
NandhaKishorM added a commit that referenced this pull request Oct 12, 2026
…ling and dropping it

#1114 adds `diagnostics` to `Router.predict_batch` so the CLI can say which checkpoint
groups actually reordered (#1111). The HTTP batch endpoint pins its control list against
that signature, and a control has to be placed on one side of the line before the suite
goes green: forwarded, lifted onto each item, or refused.

It is refused, and it is not a `BODY_REFUSALS` member. Those five run inside the server
process. This one is an out-parameter: `predict_batch` writes the report into a dict the
caller already holds a reference to, and an HTTP caller holds none, so a body that sent
`diagnostics` would be answered 200 while the report it asked for was filled in and
discarded with the request. `BODY_REFUSALS` is also pinned against `Router.predict`,
which has no `diagnostics`, so adding it there would red the single-endpoint pin.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged, with one follow-up commit from me that the PR needed and did not have.

The bug reproduces exactly as #1111 describes. Six lines, --batch-size 4, two checkpoints:

models: ['english', 'multilingual', 'english', 'multilingual', 'english', 'multilingual']
diagnostics: {'sort_by_length': {'groups': 2, 'reordered': 0, 'smallest': 3, 'largest': 3}}

The old pre-run note checked 1 < 4 < 6, which holds, so it printed nothing. Both groups hold 3 states and 4 > 3, so neither could reorder. The file-wide denominator was the whole defect, and the per-group report is the only thing that can see it. Also confirmed diagnostics is left untouched when sort_by_length is false.

What I had to add. Router.predict_batch's parameter list is pinned by tests/test_serve.py::test_batch_call_controls_are_exactly_predict_batch_kwargs_or_refusal: every kwarg must be forwarded, lifted onto each item, or refused before the suite goes green. On your branch:

AssertionError: predict_batch() takes ['batch_size', 'diagnostics', 'hooks', ...]; serve declares ['batch_size', 'hooks', ...]

I noted that on the previous round and it is still the reason this could not merge as submitted, so I wrote the serve side myself (9f7a6a61): a new BATCH_BODY_OUT_PARAMETERS = ("diagnostics",) and a 422 for a body that sends it.

The reason it is a separate tuple rather than a BODY_REFUSALS member is worth recording, because it is the thing that makes this not a one-line fix. BODY_REFUSALS is itself pinned against Router.predict, which has no diagnostics, so adding it there reds the single-endpoint pin instead. And the two refusals are refusals for different reasons: those five run inside the server process, while this one is an out-parameter. predict_batch writes the report into a dict the caller already holds a reference to, and an HTTP caller holds none, so a body sending diagnostics would be answered 200 while the report it asked for was filled in and discarded with the request. An explicit null still reads as "not asked for", like every other control.

The rest of the PR is right as written. _takes_predict_diagnostics being exact-name rather than **kwargs-tolerant is the correct call for a report that is a contract with core's own Router, the groups == 1 branch keeping the old wording with the group's own size rather than the file's is a real distinction once a cache hit shrinks the batch, and RemoteRouter accepting and ignoring it matches how it already treats batch_size and sort_by_length.

87/87 suites pass on the merged tree, and the 94-row real-weights decision diff against 0.4.3 is identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: --sort-by-length's note counts the input file, while the reorder compares against each checkpoint group

2 participants