Repository navigation
feat(agent): group batch inputs by encoded length - #294
Merged
NandhaKishorM merged 1 commit intoSep 24, 2026
Merged
Conversation
Mixed input lengths make short requests pay for long padding. Offer bounded opt-in grouping so throughput can improve while callers retain input order and control over batching.
Owner
|
Thank you. Opt-in length sorting with results restored to input order is a clean way to cut padding work, and the default path is untouched: answers are identical to |
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
Mixed input lengths make short requests pay for long padding. Offer bounded opt-in grouping so throughput can improve while callers retain input order and control over batching.
This was referenced Sep 25, 2026
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 29, 2026
…vals and /predict/batch `Agent.predict_batch` has grouped similarly sized states into one forward pass since #294, and a forward pass pads every state in it to the longest one, so the knob is what turns a bounded batch into a cheap one. README teaches it as the library call and `research/` measures it at 2.15x over 10,000 tickets -- but `laya --batch` could set `--batch-size` and not the grouping, `laya-evals run` could not ask for it, and the demo server's `/predict/batch` had no way to say how to pack the batch. The one knob with a measurable effect on a knob every surface already had was reachable only by writing Python. Each surface sends the control only when it is asked for, the way `Router.predict_batch` forwards it: an agent or runner that predates #294 keeps working for every run that does not request the grouping, and `evaluate` checks the runner's own signature before handing it over so a long run cannot die on `TypeError` at the third chunk. Measured on real Yelp reviews through the shipped entry points: - `laya --batch FILE --predict --batch-size 8`, 128 states of 69-2,293 chars, four interleaved rounds: 41.25s median unsorted against 29.23s sorted, paired median 1.42x (1.39-1.44x), 128/128 decisions identical and in input order. - `POST /predict/batch`, 64 states of 91-2,293 chars, `batch_size: 8`, three rounds: the default one-pass shape 5.35s against 3.04s sorted (paired 1.76-1.77x), and 1.66x over `batch_size: 8` unsorted, so the grouping is the second factor; 64/64 labels identical. `laya --batch FILE --sort-by-length` without a `--batch-size` in between 1 and the request count prints why nothing was reordered instead of printing a same-speed result, and `report.config.timing` keeps `sort_by_length` (what the command line said) apart from `sort_by_length_sent` (what reached the runner). Gated weight-free, and every gate green on this tree: tests/test_cli.py (135 checks), tests/test_evals.py (65, of which 7 tests are new), tests/test_evals_api.py (21), tests/test_evals_onnx.py (27) and tests/test_example_server_limits.py (74) -- including two derived anti-drift checks read off `Router.predict_batch`'s own signature, so a rename upstream fails here instead of silently dropping the grouping again, and two doc-parity checks that make the pages teach the flag in the same breath as the batch size. All 21 mutations of this change -- always-send, never-send, dropped signature check, requested-reported-as-sent, shape leaked into the per-state fallback, pages that stop teaching the flag -- fail a named check. `hooks_timeout` and `min_confidence` are still unreachable from `/predict/batch` and the MCP batch tools; named, not fixed here, to keep this to the batch shape.
This was referenced Sep 30, 2026
4 tasks done
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 12, 2026
…er file (#1111) The note's gate read the file: it printed only when --batch-size could not clear 1 < chunk < len(lines). The reorder itself switches per checkpoint and question group inside Router.predict_batch, on each group's own state count (agent.py:1519, identically onnx_agent.py:761), so a file that auto-routes to several checkpoints reordered nothing while the note stayed silent -- the silent same-speed result the note exists to prevent (#1111). Router.predict_batch now takes an optional keyword-only diagnostics dict and, when sort_by_length is set, reports under "sort_by_length" how many question groups reached an agent, how many could reorder, and the smallest / largest group size -- counted where the groups are actually formed, so an agent whose predict_batch predates the knob (#294) reports as unsorted, not as an error. RemoteRouter.predict_batch accepts and ignores the same argument, keeping the signatures identical (test_hooks_api). run_batch asks for the report only from routers whose predict_batch names the parameter, notes after the run from what actually reordered -- keeping the pre-run wording, sized by the group, when one group saw the whole file -- and leaves routers without the parameter (stubs, **kwargs forwarders) exactly as they were: same call arguments, same pre-run note. The grouping semantics in the agents are untouched. The handwritten signature list on the hooks API quick-reference page gains the parameter too. Red (this tree's tests against 6880462 sources): tests/test_cli.py 190 passed, 2 failed -- the two no-op-silence assertions; tests/test_router.py died with TypeError: unexpected keyword argument 'diagnostics'; tests/test_hooks_api.py 547 passed, 1 failed. Green: 192 / 810 / 548 / 78 passed, 0 failed (plus confidence 214, lang-guess 17, blank-lang 24, evals-api 50), ruff and compileall clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in length sorting to
Agent.predict_batchso inputs with different lengths spend less work on padding. On 10,000 synthetic English support tickets using the shipped multilingual checkpoint on CPU, elapsed prediction time fell from 1775.71 s to 825.09 s (2.15x faster), with zero changed choices/action-threshold outcomes and a maximum returned numeric difference of 0.0001.Changes
sort_by_length=False. When enabled with a usefulbatch_size, sort already encoded states by their longest question row within windows of eight batches, then restore results to input order.Testing
Windows 11, Ryzen 9 5950X, 64 GiB RAM, CPU float32, 16 Torch threads; Python 3.12.10, Torch 2.9.1+cpu, Transformers 4.57.1. Model snapshot:
aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8. The benchmark executes the upstream method from1e28ac20c0896b1c37a744cd11f740eb98f8b178for its baseline.python tests/test_batch.py: 43 passed.python tests/test_hooks.py: 86 passed;python tests/test_hooks_api.py: 101 passed;python tests/test_empty_questions.py: 4 passed.git diff --checkpassed locally.Commands and measurement details are in
research/README.mdunder Length batching, with the compact receipt inresearch/results/length_batching_cpu_20260924.json.Notes for reviewer
Agent. It needs1 < batch_size < len(states)to change grouping. Windows of eight batches bound tokenized lookahead but increase temporary CPU storage.