Repository navigation
feat(ts): port Agent.predict_batch and Router.route_batch/predict_batch - #329
Conversation
|
Short answer: no changes needed to this PR for either fix. Audit against two defect fixes that opened after this branch, both on the Python batched path this ports:
|
|
Heads-up on merge order with #398 (my lang_temperatures port): verified with |
|
Merge-blocker note for @NandhaKishorM: the required |
|
Thank you for this substantial port, the batch API and the per-request hook behaviour look right and are what we want over the other batch proposal. One regression needs fixing: _encodeItems calls sequenceWithState without Array.isArray(state), so conversation lists are truncated from the right again and the newest turn gets dropped (this was fixed in #319). Please pass that argument back in and add a systemOne and predictBatch test showing the last turn survives truncation. #398 and #386 have now landed, so please also rebase so _decodeRow takes the lang argument and emits answer_confidence. |
261f3d5 to
588f492
Compare
|
Rebased onto main and addressed both points: passed |
|
Thank you for the rebase and for both fixes, the truncation test for systemOne and predictBatch does exactly what we wanted, and I confirmed it fails if Array.isArray(state) is removed. One last gap: Router.predictBatch uses each request's lang only for routing and never passes it to agent.predictBatch, so lang_temperatures never apply on the batch path. With a German override, predict gives 0.7311 but predictBatch gives 0.8808. Could you forward each request's effective language the same way Router.predict does (explicit lang, otherwise the detected non-English language), add it to the grouping key so different languages do not share a forward pass, and add a router test for it? After that this is ready. |
|
Short answer: forwarded, grouped by, and tested. Router.predictBatch now computes each request's effective language exactly as predict does: an explicit lang wins, otherwise the detected non-English one (analyse() names English "en", which forwards as null; Python's analyse never names it). That language joins the question-group key, so same-schema requests that differ only by language no longer share a forward pass, and it is passed to agent.predictBatch and the systemOne fallback for agents without a batch method. As in router.py's predict_batch, the key is only computed for agents that carry lang_temperatures; without overrides, lang is unused and splitting would cost a forward pass for nothing. Your German case is the regression test: predict and predictBatch both return 0.7311 now, and if the forwarding line is removed the test fails with the reported 0.8808. Tests: added under "Router.predictBatch lang forwarding" in tests/lang-temperature.test.ts - explicit lang wins over detection, detected language forwarded, nothing forwarded for English, two languages don't share a forward pass (same schema and checkpoint), one shared batch when the agent has no overrides, and predict/predictBatch parity. laya-ts/README.md and the predictBatch JSDoc note the split. 279 TS tests pass; tsc --noEmit and npm run test:package clean. |
…#381 parity) Router.predictBatch routed with each request's language but never passed it to agent.predictBatch, so lang_temperatures never applied on the batched path: the same German request scored 0.7311 through predict and 0.8808 through predictBatch. - Compute each request's effective language exactly as predict does: an explicit lang wins, otherwise the detected non-English language (analyse() names English "en", which forwards as null -- Python's analyse never names it). - Add it to the question-group key, so same-schema requests that differ only by language no longer share a forward pass. As in router.py's predict_batch, the key is only computed for an agent that carries lang_temperatures: otherwise `lang` is unused there and splitting would cost a forward pass for nothing. - Forward it to agent.predictBatch, and to the systemOne fallback for Agent-like objects without a batch method. - Tests: explicit/detected/none forwarding, the group split, no split when the agent carries no overrides, and predict/predictBatch parity on the reported German case (0.8808 vs 0.7311 if the forwarding line is dropped).
4b9772c to
f1e2ac0
Compare
|
Rebased onto current main and force-pushed. Resolved the conflicts in |
|
The only failing required check here is |
|
Passed checks and merged. Thank you @aashish254! It ships in 0.3.22. |
Summary
Ports the batch prediction APIs to
laya-ts— the biggest remaining Python/TS parity gap (previously tracked under #166 follow-ups):Agent.predictBatch(states, questions, opts)— the throughput path. States sharing a question schema are collated into one shared encoder+head forward pass (or one perbatchSizechunk to bound peak memory), instead of one forward pass per state. Results align withstatesby index and are identical in shape tosystemOne's output, including per-stateusage.input_tokens.Router.routeBatch(requests)— validate + route many requests without loading anything; malformed batches fail fast (TypeErrorfor wrong types,Errorfor missing keys, mirroring the Python messages).Router.predictBatch(requests, batchSize?)(+predictManyalias) — route first, group by checkpoint, then score each group in as few agent batches as possible. Same-checkpoint requests still split into separate batches when their question schemas differ order-sensitively (options are positional — the feat(router): predict_batch — route-then-group multi-state scoring (follow-up to #47, with consistency harness + benchmarks) #166 regression) or when per-request start hooks set differentmaxLen/headMaxLenoverrides. Results keep input order and always carry theroutingkey (added, never overwritten, on cache hits).PredictContextwithctx.decision;ctx.skip()short-circuits inference for that request only;onPredictEndruns once per started request even when the batch fails, withelapsedMs/usagestamped on every context before any end hook runs;onErrorfires per started-without-result request and the original batch failure propagates.PredictOptionsgainsmaxLen/headMaxLen/batchSizeoverrides, andctx.maxLen/ctx.headMaxLenset by start hooks are now honored by the agent core (previously declared onPredictContextbut unused).Agent-like objects that only implement
systemOnestill work withRouter.predictBatch— their states simply share no forward pass.Not ported: Python's
sort_by_lengthlength-grouping (noted in the README); happy to follow up if wanted.Internals
_systemOneCorewas split into_encodeItems(state tokenized once per state, unchanged) +_decodeRow(per-row decode, unchanged) +_systemOneMany(chunks states, collates each chunk into one batch, validates provider output row counts/values like the single-state path, then decodes rows back per state). The single-state path runs through the same code with a chunk of one, so numerics are unchanged.Tests
New
laya-ts/tests/router-batch.test.ts(19 tests), mirroringtests/test_router_batch.pyand the orchestration part oftests/test_batch.py:routeBatch, and share one agent batch per checkpoint (with the typed-decisions schema split)maxLoadedpressurebatchSizepassthrough; schema-order-sensitive grouping;langGuess/lang_guessforwardingonError/onPredictEndon failure, override-based group splittingsystemOnefallbackbatchSizechunking 2+2+1, empty states/questions skip encoding, parity withsystemOne, per-state usagenpx tsc --noEmitclean;npx vitest run→ 184/184 (13 files).