Skip to content

feat(agent): group batch inputs by encoded length - #294

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
MycelAI:alpha912/perf/length-batching
Sep 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
MycelAI:alpha912/perf/length-batching

Conversation

@alpha912

Copy link
Copy Markdown
Contributor

Summary

Add opt-in length sorting to Agent.predict_batch so inputs with different lengths spend less work on padding. On 10,000 synthetic English support tickets using the shipped multilingual checkpoint on CPU, elapsed prediction time fell from 1775.71 s to 825.09 s (2.15x faster), with zero changed choices/action-threshold outcomes and a maximum returned numeric difference of 0.0001.

Changes

  • Add sort_by_length=False. When enabled with a useful batch_size, sort already encoded states by their longest question row within windows of eight batches, then restore results to input order.
  • Preserve per-call hooks, token-budget overrides, per-state usage, and the cap on states per forward pass. Small/all-in-one batches retain their existing grouping.
  • Add regression coverage for ordering across windows, partial batches, question-row mapping, bounded lookahead, hook rewrites/cache/errors, empty inputs, and equal-length ties.
  • Include a reproducible real-checkpoint benchmark, measurements, and guidance about floating-point differences when batch shapes change.

Testing

Windows 11, Ryzen 9 5950X, 64 GiB RAM, CPU float32, 16 Torch threads; Python 3.12.10, Torch 2.9.1+cpu, Transformers 4.57.1. Model snapshot: aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8. The benchmark executes the upstream method from 1e28ac20c0896b1c37a744cd11f740eb98f8b178 for its baseline.

Multilingual checkpoint, batch size 8 Before Grouped Speedup
1,000 inputs 179.09 s 83.77 s 2.14x
10,000 inputs 1775.71 s 825.09 s 2.15x
  • Both scales: zero decision flips and usage mismatches; maximum returned numeric difference 0.0001. The full run reduced padded tensor slots from 3,044,328 to 1,505,560.
  • Repeated, alternating-order small tests: English mixed lengths 1.47x, multilingual with all three question types 1.60x; similar-length English control 1.04x. These returned exactly equal results.
  • Separate 64-input memory profiles: sampled peak process RSS 1934.27 MiB before / 1936.84 MiB grouped (about +0.13%).
  • python tests/test_batch.py: 43 passed.
  • python tests/test_hooks.py: 86 passed; python tests/test_hooks_api.py: 101 passed; python tests/test_empty_questions.py: 4 passed.
  • CI Ruff rules, benchmark Ruff, byte compilation, and git diff --check passed locally.

Commands and measurement details are in research/README.md under Length batching, with the compact receipt in research/results/length_batching_cpu_20260924.json.

Notes for reviewer

  • The workload is synthetic English support text, not a labeled accuracy benchmark. The 1,000 and 10,000 input runs each contain one full before/after timing pair; smaller tests repeat and alternate order. An unrelated adaptation experiment was running on the GPU while these benchmarks used CPU.
  • The option is off by default and exposed on Agent. It needs 1 < batch_size < len(states) to change grouping. Windows of eight batches bound tokenized lookahead but increase temporary CPU storage.
  • Changed batch shapes can change floating-point values, including decisions very close to thresholds. Check the intended workload before enabling it. GPU, MPS, compiled and TileLang performance were not measured.
  • Memory results are sampled at a 10 ms polling interval in separate processes after warm-up, using 64 inputs. They exclude model-load transients and can miss brief allocation peaks.

Mixed input lengths make short requests pay for long padding. Offer bounded opt-in grouping so throughput can improve while callers retain input order and control over batching.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you. Opt-in length sorting with results restored to input order is a clean way to cut padding work, and the default path is untouched: answers are identical to main on six states with it off. Merging.

@NandhaKishorM
NandhaKishorM merged commit dd9a4df into NandhaKishorM:main Sep 24, 2026
19 checks passed
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
Mixed input lengths make short requests pay for long padding. Offer bounded opt-in grouping so throughput can improve while callers retain input order and control over batching.
NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
…vals and /predict/batch

`Agent.predict_batch` has grouped similarly sized states into one forward pass since
#294, and a forward pass pads every state in it to the longest one, so the knob is
what turns a bounded batch into a cheap one. README teaches it as the library call
and `research/` measures it at 2.15x over 10,000 tickets -- but `laya --batch` could
set `--batch-size` and not the grouping, `laya-evals run` could not ask for it, and
the demo server's `/predict/batch` had no way to say how to pack the batch. The one
knob with a measurable effect on a knob every surface already had was reachable only
by writing Python.

Each surface sends the control only when it is asked for, the way `Router.predict_batch`
forwards it: an agent or runner that predates #294 keeps working for every run that
does not request the grouping, and `evaluate` checks the runner's own signature before
handing it over so a long run cannot die on `TypeError` at the third chunk.

Measured on real Yelp reviews through the shipped entry points:
- `laya --batch FILE --predict --batch-size 8`, 128 states of 69-2,293 chars, four
  interleaved rounds: 41.25s median unsorted against 29.23s sorted, paired median
  1.42x (1.39-1.44x), 128/128 decisions identical and in input order.
- `POST /predict/batch`, 64 states of 91-2,293 chars, `batch_size: 8`, three rounds:
  the default one-pass shape 5.35s against 3.04s sorted (paired 1.76-1.77x), and
  1.66x over `batch_size: 8` unsorted, so the grouping is the second factor;
  64/64 labels identical.

`laya --batch FILE --sort-by-length` without a `--batch-size` in between 1 and the
request count prints why nothing was reordered instead of printing a same-speed result,
and `report.config.timing` keeps `sort_by_length` (what the command line said) apart
from `sort_by_length_sent` (what reached the runner).

Gated weight-free, and every gate green on this tree: tests/test_cli.py (135 checks),
tests/test_evals.py (65, of which 7 tests are new), tests/test_evals_api.py (21),
tests/test_evals_onnx.py (27) and tests/test_example_server_limits.py (74) -- including
two derived anti-drift checks read off `Router.predict_batch`'s own signature, so a
rename upstream fails here instead of silently dropping the grouping again, and two
doc-parity checks that make the pages teach the flag in the same breath as the batch
size. All 21 mutations of this change -- always-send, never-send, dropped signature
check, requested-reported-as-sent, shape leaked into the per-state fallback, pages that
stop teaching the flag -- fail a named check.

`hooks_timeout` and `min_confidence` are still unreachable from `/predict/batch` and
the MCP batch tools; named, not fixed here, to keep this to the batch shape.
NandhaKishorM pushed a commit that referenced this pull request Oct 12, 2026
…er file (#1111)

The note's gate read the file: it printed only when --batch-size could not
clear 1 < chunk < len(lines). The reorder itself switches per checkpoint and
question group inside Router.predict_batch, on each group's own state count
(agent.py:1519, identically onnx_agent.py:761), so a file that auto-routes to
several checkpoints reordered nothing while the note stayed silent -- the
silent same-speed result the note exists to prevent (#1111).

Router.predict_batch now takes an optional keyword-only diagnostics dict and,
when sort_by_length is set, reports under "sort_by_length" how many question
groups reached an agent, how many could reorder, and the smallest / largest
group size -- counted where the groups are actually formed, so an agent whose
predict_batch predates the knob (#294) reports as unsorted, not as an error.
RemoteRouter.predict_batch accepts and ignores the same argument, keeping the
signatures identical (test_hooks_api). run_batch asks for the report only from
routers whose predict_batch names the parameter, notes after the run from what
actually reordered -- keeping the pre-run wording, sized by the group, when one
group saw the whole file -- and leaves routers without the parameter (stubs,
**kwargs forwarders) exactly as they were: same call arguments, same pre-run
note. The grouping semantics in the agents are untouched. The handwritten
signature list on the hooks API quick-reference page gains the parameter too.

Red (this tree's tests against 6880462 sources): tests/test_cli.py 190 passed,
2 failed -- the two no-op-silence assertions; tests/test_router.py died with
TypeError: unexpected keyword argument 'diagnostics'; tests/test_hooks_api.py
547 passed, 1 failed. Green: 192 / 810 / 548 / 78 passed, 0 failed (plus
confidence 214, lang-guess 17, blank-lang 24, evals-api 50), ruff and
compileall clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants