Skip to content

feat(agent): predict_batch — score many states in shared forward passes (~10x GPU throughput) - #47

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
AlKor13:feat/predict-batch
Sep 23, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
AlKor13:feat/predict-batch

Conversation

@AlKor13

@AlKor13 AlKor13 commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

What & why

predict/system_one evaluate one state per forward pass, which leaves most of the GPU's batch dimension idle. For any list-shaped workload — a backlog of tickets, rows of a table, a log slice — that's where the throughput goes.

predict_batch(states, questions) collates many states' question rows into shared forward passes, so a call that would take N sequential passes takes one (or ceil(N / batch_size)):

states  = [{"body": t} for t in ticket_texts]
results = agent.predict_batch(states, questions)              # one forward pass for the list
results = agent.predict_batch(states, questions, batch_size=64)  # bound peak memory
# results[i] == agent.predict(states[i], questions), aligned by index

Measured (RTX 5060 Ti, mmBERT-base multilingual, 30 states × 3 questions)

path per-decision latency throughput
one-by-one (predict in a loop) ~10 ms 1×
predict_batch (single pass) ~1 ms ~9–10×

CPU is already compute-bound, so batching there is for API convenience, not speed (I measured ~0.8× — no gain). The README and docstring say so plainly rather than implying a universal win.

This is distinct from the existing async batch(max_concurrency=…) (#2) and the REST /decide/batch fan-out (#3): both still issue one forward pass per state under the hood (asyncio.gather over single-state calls). This PR adds the actual tensor batching in the core engine that those surfaces could sit on top of.

How it stays correct

To guarantee the batched and single-state paths can't drift, system_one is now predict_batch([state], questions)[0] — a single code path. The shared logic is factored into three helpers:

  • _encode_state — tokenize one state against every question (keeps the head_max_len option-ceiling check),
  • _forward — run the collated batch, keeping the GPU→CPU OOM fallback unchanged,
  • _decode_answers — turn one state's logit rows (at a given offset) into typed answers.

Equivalence is verified, not assumed:

  • CPU (fp32): byte-identical to the old single path.
  • GPU (fp16 autocast): decisions and token counts identical, probabilities match to <5e-3 — the wobble is fp16 reordering reductions across padding widths, inherent to batched fp16 matmul, not a logic difference.

Tests

  • tests/test_batch.py (new, weight-free, wired into CI): contract tests (predict_batch exists, system_one delegates to it, predict alias intact, batch_size kwarg) + orchestration tests driven against a stubbed forward — empty input → [], bare str/dict → TypeError, per-state row mapping (state i decoded from row i·|Q|), and batch_size chunking (5 states, size 2 → passes of 4+4+2).
  • tests/test_local_e2e.py (real weights): a new section asserts predict_batch matches system_one decision-for-decision across the full / chunked / one-by-one paths, with the fp16-vs-fp32 tolerance handled per device.

Local run (Python 3.12, torch 2.14+cu130, RTX 5060 Ti)

test_router.py     106 passed
test_criteria.py    34 passed
test_batch.py       15 passed
test_local_e2e.py   28 passed   (incl. new batch section: 5/5)
ruff (E9,F63,F7,F82,F401,F811)  clean
compileall          clean

No public API removed; predict/system_one behavior is preserved. predict_batch and batch_size are additive.


🤖 Generated with Claude Code

@sirgio03

Copy link
Copy Markdown
Contributor

Holy S!!!!!, this is so cooool damn! Dropping GPU latency to ~1ms for list workloads is an insane throughput jump!!!!

I saw the shoutout to PR #3 thank you! This is exactly what the REST API needs. As soon as this gets merged into main, I’ll immediately update the FastAPI /v1/decide/batch endpoint in my PR. I'll rip out the asyncio.gather fan-out and wire it directly into your new predict_batch() method so the HTTP layer gets that full 10x tensor speedup.

Incredibly cool update, great work!
@NandhaKishorM i guess we need to update those speed bunchmarks 😂

@AlKor13

AlKor13 commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @sirgio03! And yeah — this and your REST server (#3) fit together cleanly.

Right now /decide/batch fans out with asyncio.gather over single-state calls, so each request is still one forward pass. If it calls agent.predict_batch(states, questions) instead, the whole BatchDecideRequest collapses into one (or a few) forward passes — that's where the ~10x on GPU comes from. Results stay aligned by index and identical in shape to predict, so the response mapping doesn't change.

Two natural follow-ups once this lands, both building on it:

  • AsyncAgent.predict_batch that runs the batched forward in a worker thread, so the async layer sits on real tensor batching instead of concurrency alone.
  • a small server-side micro-batching queue that coalesces incoming single-item requests within a short window (~5-15ms) into one predict_batch — that's what turns the throughput win from bulk-only into an online-serving win.

Happy to help wire predict_batch into the REST path if useful.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks for putting together predict_batch. Packing multiple states into a single forward pass provides significant throughput gains on GPU. We are reviewing the collation and padding logic.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you. Packing many states into one forward pass is the throughput win people keep asking for. There are now three overlapping proposals: this one, #166 (router-aware grouping plus a consistency harness) and #138 (group by checkpoint). I'd like to land one API rather than three. Could you and @bigdong89 agree on a shape? This PR plus #166's router-aware grouping and harness looks like the right combination. It also needs a rebase onto today's main.

`predict`/`system_one` evaluate one state per forward pass, which leaves most
of the GPU batch dimension idle. `predict_batch(states, questions)` collates
many states' question rows into shared forward passes, so a call that would take
N sequential passes takes one (or `ceil(N / batch_size)`).

Measured on an RTX 5060 Ti (mmBERT-base, 30 states x 3 questions): per-decision
latency drops from ~10 ms one-by-one to ~1 ms batched (~9-10x). CPU is already
compute-bound, so batching there is for API convenience, not speed.

To guarantee the batched and single-state paths cannot drift, `system_one` is now
`predict_batch([state], questions)[0]` -- one code path. The shared forward and
answer-decoding logic are factored into `_encode_state`, `_forward` (keeping the
GPU->CPU OOM fallback), and `_decode_answers`. Results are byte-identical to the
old single path on CPU (fp32); on GPU they match to <5e-3 (fp16 autocast reorders
reductions across padding widths) with identical decisions and token counts.

- tests/test_batch.py: weight-free contract + orchestration tests (empty input,
  bad input, per-state row mapping, batch_size chunking); wired into CI.
- tests/test_local_e2e.py: real-weights section verifying predict_batch matches
  system_one decision-for-decision across full/chunked/one-by-one paths.
- README: Batch Mode section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AlKor13

AlKor13 commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Sounds right — one API, this PR as the Agent-level base plus @bigdong89's router-aware grouping and consistency harness on top. Rebased onto today's main (0.3.7) just now; conflicts were agent.py (took main's __init__, _check_question, noul-key normalization, empty-questions guard and the context-manager, re-applied the predict_batch refactor on top) and the README section. Green locally: full CI test set + test_batch, ruff, and predict vs predict_batch bit-identical on CPU / 0 decision mismatches on GPU.

@bigdong89 — the only hard conflict between us is the predict_batch signature. Mine is predict_batch(states, questions, batch_size=None) (same questions across states + memory chunking); yours is system_one_batch(items=[(state, questions), ...]) (heterogeneous questions). I think one method covers both without either of us losing a use case:

def predict_batch(self, states, questions=None, batch_size=None):
    # homogeneous: predict_batch([s1, s2, ...], questions)     -> same questions for every state
    # heterogeneous: predict_batch([(s1, q1), (s2, q2), ...])  -> questions=None, states carry their own

Then your Router.predict_batch (route → group by checkpoint → call each agent's predict_batch on its group → reassemble in original index order, which your PR already does correctly) and the consistency/benchmark harness land on top unchanged.

If that shape works for you, I can add the questions=None heterogeneous branch to this PR so the Agent layer covers both, and you rebase the Router grouping + harness onto it — or if you'd rather own the combined PR, I'm happy to have this one close in favor of yours once the Agent API matches. Either way we converge on a single predict_batch. @NandhaKishorM which of the two landing spots do you prefer?

# Conflicts:
#	laya/agent.py
#	tests/test_local_e2e.py
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for rebasing and for agreeing on one API. I checked it against main on the English checkpoint: system_one and predict_batch (one pass and batch_size=2) return identical answers and token usage for six states of very different lengths, so padding does not leak into answers. Today's merges changed the same loop (#224 truncates conversation lists from the left, #163 documents labels), so I resolved the conflict on your branch by carrying both into _encode_state and the system_one docstring, and renumbered your end-to-end section after #225's. I re-ran the same comparison after the resolution and it is still identical. Merging. @bigdong89, #166's router-aware grouping and consistency harness would be very welcome on top of this. @Ahmed-3411, the same goes for #138's grouping by checkpoint.

@NandhaKishorM
NandhaKishorM merged commit ffaae82 into NandhaKishorM:main Sep 23, 2026
bigdong89 added a commit to bigdong89/laya that referenced this pull request Sep 23, 2026
Builds on Agent.predict_batch (NandhaKishorM#47): Router.predict_batch routes every state
independently, groups states by (checkpoint, question set), and scores each
group in one shared forward pass via Agent.predict_batch. Mixed-language
traffic now batches too: 32 mixed EN/Hindi states collapse from 32 forwards
to 2 (7.0x end-to-end on an RTX 4070).

Includes a consistency + benchmark harness (benchmarks/bench_predict_batch.py):
900 answers compared batch-vs-single across all three checkpoints, 0 decision
flips, max |dp| = 0.022 (fp16 batch-shape noise, worst case on a score-type
question). 6 offline unit tests on tiny weights.
NandhaKishorM pushed a commit that referenced this pull request Sep 24, 2026
…ollow-up to #47, with consistency harness + benchmarks) (#166)

* fix(router): score reordered-but-equal question schemas separately

Grouping by dict equality can merge two requests whose questions are equal
but keyed in a different insertion order; the group then scores the second
caller with the first caller's option order. Options are positional in
render_options, so that request's batched answers would not match its own
single-request answer -- the same failure mode as serialising questions to
JSON with sort_keys=True and reading them back. Group by an order-sensitive
schema signature instead (json.dumps with sort_keys=False and default=str,
matching render_criterion's tolerance), with a deterministic fake-agent
regression test.

* bench: consistency harness + benchmarks for the merged predict_batch

Point the harness at the merged Router.predict_batch (#138):

* tests/test_predict_batch.py -- tiny-checkpoint tests (CPU): agent-level
  batch==single, router-level mixed-language routing/order preservation, and
  the reordered-schema regression at the router layer
* benchmarks/bench_predict_batch.py -- --router now speaks the requests-dict
  API and adds router_consistency(): predict_batch vs one predict per request
  on 50/50 mixed EN/Hindi, the merge bar on the workload that regroups
  --consistency also writes its JSON now
* plot_results.py renders bench-rtx4070.png from the three result files

RTX 4070, laya 0.3.11, torch 2.6.0+cu124: agent layer 1.5-1.8x (<=64 states,
4 questions), router layer up to 3.4x (32 requests, 93 -> 313 states/s);
consistency 1800 agent answers + 600 router answers, 0 flips, max|dp| 2.2%.
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#47)

`predict`/`system_one` evaluate one state per forward pass, which leaves most
of the GPU batch dimension idle. `predict_batch(states, questions)` collates
many states' question rows into shared forward passes, so a call that would take
N sequential passes takes one (or `ceil(N / batch_size)`).

Measured on an RTX 5060 Ti (mmBERT-base, 30 states x 3 questions): per-decision
latency drops from ~10 ms one-by-one to ~1 ms batched (~9-10x). CPU is already
compute-bound, so batching there is for API convenience, not speed.

To guarantee the batched and single-state paths cannot drift, `system_one` is now
`predict_batch([state], questions)[0]` -- one code path. The shared forward and
answer-decoding logic are factored into `_encode_state`, `_forward` (keeping the
GPU->CPU OOM fallback), and `_decode_answers`. Results are byte-identical to the
old single path on CPU (fp32); on GPU they match to <5e-3 (fp16 autocast reorders
reductions across padding widths) with identical decisions and token counts.

- tests/test_batch.py: weight-free contract + orchestration tests (empty input,
  bad input, per-state row mapping, batch_size chunking); wired into CI.
- tests/test_local_e2e.py: real-weights section verifying predict_batch matches
  system_one decision-for-decision across full/chunked/one-by-one paths.
- README: Batch Mode section.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…ollow-up to NandhaKishorM#47, with consistency harness + benchmarks) (NandhaKishorM#166)

* fix(router): score reordered-but-equal question schemas separately

Grouping by dict equality can merge two requests whose questions are equal
but keyed in a different insertion order; the group then scores the second
caller with the first caller's option order. Options are positional in
render_options, so that request's batched answers would not match its own
single-request answer -- the same failure mode as serialising questions to
JSON with sort_keys=True and reading them back. Group by an order-sensitive
schema signature instead (json.dumps with sort_keys=False and default=str,
matching render_criterion's tolerance), with a deterministic fake-agent
regression test.

* bench: consistency harness + benchmarks for the merged predict_batch

Point the harness at the merged Router.predict_batch (NandhaKishorM#138):

* tests/test_predict_batch.py -- tiny-checkpoint tests (CPU): agent-level
  batch==single, router-level mixed-language routing/order preservation, and
  the reordered-schema regression at the router layer
* benchmarks/bench_predict_batch.py -- --router now speaks the requests-dict
  API and adds router_consistency(): predict_batch vs one predict per request
  on 50/50 mixed EN/Hindi, the merge bar on the workload that regroups
  --consistency also writes its JSON now
* plot_results.py renders bench-rtx4070.png from the three result files

RTX 4070, laya 0.3.11, torch 2.6.0+cu124: agent layer 1.5-1.8x (<=64 states,
4 questions), router layer up to 3.4x (32 requests, 93 -> 313 states/s);
consistency 1800 agent answers + 600 router answers, 0 flips, max|dp| 2.2%.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants