Repository navigation
feat(agent): predict_batch — score many states in shared forward passes (~10x GPU throughput) - #47
Conversation
|
Holy S!!!!!, this is so cooool damn! Dropping GPU latency to ~1ms for list workloads is an insane throughput jump!!!! I saw the shoutout to PR #3 thank you! This is exactly what the REST API needs. As soon as this gets merged into main, I’ll immediately update the FastAPI /v1/decide/batch endpoint in my PR. I'll rip out the asyncio.gather fan-out and wire it directly into your new predict_batch() method so the HTTP layer gets that full 10x tensor speedup. Incredibly cool update, great work! |
|
Thanks @sirgio03! And yeah — this and your REST server (#3) fit together cleanly. Right now Two natural follow-ups once this lands, both building on it:
Happy to help wire |
|
Thanks for putting together predict_batch. Packing multiple states into a single forward pass provides significant throughput gains on GPU. We are reviewing the collation and padding logic. |
7015f6c to
f2d9ab6
Compare
|
Thank you. Packing many states into one forward pass is the throughput win people keep asking for. There are now three overlapping proposals: this one, #166 (router-aware grouping plus a consistency harness) and #138 (group by checkpoint). I'd like to land one API rather than three. Could you and @bigdong89 agree on a shape? This PR plus #166's router-aware grouping and harness looks like the right combination. It also needs a rebase onto today's |
`predict`/`system_one` evaluate one state per forward pass, which leaves most of the GPU batch dimension idle. `predict_batch(states, questions)` collates many states' question rows into shared forward passes, so a call that would take N sequential passes takes one (or `ceil(N / batch_size)`). Measured on an RTX 5060 Ti (mmBERT-base, 30 states x 3 questions): per-decision latency drops from ~10 ms one-by-one to ~1 ms batched (~9-10x). CPU is already compute-bound, so batching there is for API convenience, not speed. To guarantee the batched and single-state paths cannot drift, `system_one` is now `predict_batch([state], questions)[0]` -- one code path. The shared forward and answer-decoding logic are factored into `_encode_state`, `_forward` (keeping the GPU->CPU OOM fallback), and `_decode_answers`. Results are byte-identical to the old single path on CPU (fp32); on GPU they match to <5e-3 (fp16 autocast reorders reductions across padding widths) with identical decisions and token counts. - tests/test_batch.py: weight-free contract + orchestration tests (empty input, bad input, per-state row mapping, batch_size chunking); wired into CI. - tests/test_local_e2e.py: real-weights section verifying predict_batch matches system_one decision-for-decision across full/chunked/one-by-one paths. - README: Batch Mode section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
f2d9ab6 to
bc15464
Compare
|
Sounds right — one API, this PR as the Agent-level base plus @bigdong89's router-aware grouping and consistency harness on top. Rebased onto today's @bigdong89 — the only hard conflict between us is the def predict_batch(self, states, questions=None, batch_size=None):
# homogeneous: predict_batch([s1, s2, ...], questions) -> same questions for every state
# heterogeneous: predict_batch([(s1, q1), (s2, q2), ...]) -> questions=None, states carry their own
Then your If that shape works for you, I can add the |
# Conflicts: # laya/agent.py # tests/test_local_e2e.py
|
Thank you for rebasing and for agreeing on one API. I checked it against |
Builds on Agent.predict_batch (NandhaKishorM#47): Router.predict_batch routes every state independently, groups states by (checkpoint, question set), and scores each group in one shared forward pass via Agent.predict_batch. Mixed-language traffic now batches too: 32 mixed EN/Hindi states collapse from 32 forwards to 2 (7.0x end-to-end on an RTX 4070). Includes a consistency + benchmark harness (benchmarks/bench_predict_batch.py): 900 answers compared batch-vs-single across all three checkpoints, 0 decision flips, max |dp| = 0.022 (fp16 batch-shape noise, worst case on a score-type question). 6 offline unit tests on tiny weights.
…ollow-up to #47, with consistency harness + benchmarks) (#166) * fix(router): score reordered-but-equal question schemas separately Grouping by dict equality can merge two requests whose questions are equal but keyed in a different insertion order; the group then scores the second caller with the first caller's option order. Options are positional in render_options, so that request's batched answers would not match its own single-request answer -- the same failure mode as serialising questions to JSON with sort_keys=True and reading them back. Group by an order-sensitive schema signature instead (json.dumps with sort_keys=False and default=str, matching render_criterion's tolerance), with a deterministic fake-agent regression test. * bench: consistency harness + benchmarks for the merged predict_batch Point the harness at the merged Router.predict_batch (#138): * tests/test_predict_batch.py -- tiny-checkpoint tests (CPU): agent-level batch==single, router-level mixed-language routing/order preservation, and the reordered-schema regression at the router layer * benchmarks/bench_predict_batch.py -- --router now speaks the requests-dict API and adds router_consistency(): predict_batch vs one predict per request on 50/50 mixed EN/Hindi, the merge bar on the workload that regroups --consistency also writes its JSON now * plot_results.py renders bench-rtx4070.png from the three result files RTX 4070, laya 0.3.11, torch 2.6.0+cu124: agent layer 1.5-1.8x (<=64 states, 4 questions), router layer up to 3.4x (32 requests, 93 -> 313 states/s); consistency 1800 agent answers + 600 router answers, 0 flips, max|dp| 2.2%.
NandhaKishorM#47) `predict`/`system_one` evaluate one state per forward pass, which leaves most of the GPU batch dimension idle. `predict_batch(states, questions)` collates many states' question rows into shared forward passes, so a call that would take N sequential passes takes one (or `ceil(N / batch_size)`). Measured on an RTX 5060 Ti (mmBERT-base, 30 states x 3 questions): per-decision latency drops from ~10 ms one-by-one to ~1 ms batched (~9-10x). CPU is already compute-bound, so batching there is for API convenience, not speed. To guarantee the batched and single-state paths cannot drift, `system_one` is now `predict_batch([state], questions)[0]` -- one code path. The shared forward and answer-decoding logic are factored into `_encode_state`, `_forward` (keeping the GPU->CPU OOM fallback), and `_decode_answers`. Results are byte-identical to the old single path on CPU (fp32); on GPU they match to <5e-3 (fp16 autocast reorders reductions across padding widths) with identical decisions and token counts. - tests/test_batch.py: weight-free contract + orchestration tests (empty input, bad input, per-state row mapping, batch_size chunking); wired into CI. - tests/test_local_e2e.py: real-weights section verifying predict_batch matches system_one decision-for-decision across full/chunked/one-by-one paths. - README: Batch Mode section. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Nandakishor <48623612+NandhaKishorM@users.noreply.github.com>
…ollow-up to NandhaKishorM#47, with consistency harness + benchmarks) (NandhaKishorM#166) * fix(router): score reordered-but-equal question schemas separately Grouping by dict equality can merge two requests whose questions are equal but keyed in a different insertion order; the group then scores the second caller with the first caller's option order. Options are positional in render_options, so that request's batched answers would not match its own single-request answer -- the same failure mode as serialising questions to JSON with sort_keys=True and reading them back. Group by an order-sensitive schema signature instead (json.dumps with sort_keys=False and default=str, matching render_criterion's tolerance), with a deterministic fake-agent regression test. * bench: consistency harness + benchmarks for the merged predict_batch Point the harness at the merged Router.predict_batch (NandhaKishorM#138): * tests/test_predict_batch.py -- tiny-checkpoint tests (CPU): agent-level batch==single, router-level mixed-language routing/order preservation, and the reordered-schema regression at the router layer * benchmarks/bench_predict_batch.py -- --router now speaks the requests-dict API and adds router_consistency(): predict_batch vs one predict per request on 50/50 mixed EN/Hindi, the merge bar on the workload that regroups --consistency also writes its JSON now * plot_results.py renders bench-rtx4070.png from the three result files RTX 4070, laya 0.3.11, torch 2.6.0+cu124: agent layer 1.5-1.8x (<=64 states, 4 questions), router layer up to 3.4x (32 requests, 93 -> 313 states/s); consistency 1800 agent answers + 600 router answers, 0 flips, max|dp| 2.2%.
What & why
predict/system_oneevaluate one state per forward pass, which leaves most of the GPU's batch dimension idle. For any list-shaped workload — a backlog of tickets, rows of a table, a log slice — that's where the throughput goes.predict_batch(states, questions)collates many states' question rows into shared forward passes, so a call that would take N sequential passes takes one (orceil(N / batch_size)):Measured (RTX 5060 Ti, mmBERT-base multilingual, 30 states × 3 questions)
predictin a loop)predict_batch(single pass)CPU is already compute-bound, so batching there is for API convenience, not speed (I measured ~0.8× — no gain). The README and docstring say so plainly rather than implying a universal win.
This is distinct from the existing async
batch(max_concurrency=…)(#2) and the REST/decide/batchfan-out (#3): both still issue one forward pass per state under the hood (asyncio.gatherover single-state calls). This PR adds the actual tensor batching in the core engine that those surfaces could sit on top of.How it stays correct
To guarantee the batched and single-state paths can't drift,
system_oneis nowpredict_batch([state], questions)[0]— a single code path. The shared logic is factored into three helpers:_encode_state— tokenize one state against every question (keeps thehead_max_lenoption-ceiling check),_forward— run the collated batch, keeping the GPU→CPU OOM fallback unchanged,_decode_answers— turn one state's logit rows (at a given offset) into typed answers.Equivalence is verified, not assumed:
<5e-3— the wobble is fp16 reordering reductions across padding widths, inherent to batched fp16 matmul, not a logic difference.Tests
tests/test_batch.py(new, weight-free, wired into CI): contract tests (predict_batch exists,system_onedelegates to it,predictalias intact,batch_sizekwarg) + orchestration tests driven against a stubbed forward — empty input →[], barestr/dict→TypeError, per-state row mapping (state i decoded from row i·|Q|), andbatch_sizechunking (5 states, size 2 → passes of 4+4+2).tests/test_local_e2e.py(real weights): a new section assertspredict_batchmatchessystem_onedecision-for-decision across the full / chunked / one-by-one paths, with the fp16-vs-fp32 tolerance handled per device.Local run (Python 3.12, torch 2.14+cu130, RTX 5060 Ti)
No public API removed;
predict/system_onebehavior is preserved.predict_batchandbatch_sizeare additive.🤖 Generated with Claude Code