Skip to content

feat(ts): port Agent.predict_batch and Router.route_batch/predict_batch - #329

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:feat-ts-predict-batch
Sep 29, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
aashish254:feat-ts-predict-batch

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

Summary

Ports the batch prediction APIs to laya-ts — the biggest remaining Python/TS parity gap (previously tracked under #166 follow-ups):

  • Agent.predictBatch(states, questions, opts) — the throughput path. States sharing a question schema are collated into one shared encoder+head forward pass (or one per batchSize chunk to bound peak memory), instead of one forward pass per state. Results align with states by index and are identical in shape to systemOne's output, including per-state usage.input_tokens.
  • Router.routeBatch(requests) — validate + route many requests without loading anything; malformed batches fail fast (TypeError for wrong types, Error for missing keys, mirroring the Python messages).
  • Router.predictBatch(requests, batchSize?) (+ predictMany alias) — route first, group by checkpoint, then score each group in as few agent batches as possible. Same-checkpoint requests still split into separate batches when their question schemas differ order-sensitively (options are positional — the feat(router): predict_batch — route-then-group multi-state scoring (follow-up to #47, with consistency harness + benchmarks) #166 regression) or when per-request start hooks set different maxLen/headMaxLen overrides. Results keep input order and always carry the routing key (added, never overwritten, on cache hits).
  • Hooks: router-level predict hooks run per request — each request gets its own PredictContext with ctx.decision; ctx.skip() short-circuits inference for that request only; onPredictEnd runs once per started request even when the batch fails, with elapsedMs/usage stamped on every context before any end hook runs; onError fires per started-without-result request and the original batch failure propagates.
  • PredictOptions gains maxLen / headMaxLen / batchSize overrides, and ctx.maxLen / ctx.headMaxLen set by start hooks are now honored by the agent core (previously declared on PredictContext but unused).

Agent-like objects that only implement systemOne still work with Router.predictBatch — their states simply share no forward pass.

Not ported: Python's sort_by_length length-grouping (noted in the README); happy to follow up if wanted.

Internals

_systemOneCore was split into _encodeItems (state tokenized once per state, unchanged) + _decodeRow (per-row decode, unchanged) + _systemOneMany (chunks states, collates each chunk into one batch, validates provider output row counts/values like the single-state path, then decodes rows back per state). The single-state path runs through the same code with a chunk of one, so numerics are unchanged.

Tests

New laya-ts/tests/router-batch.test.ts (19 tests), mirroring tests/test_router_batch.py and the orchestration part of tests/test_batch.py:

  • mixed-language/task batches keep input order, route identically to routeBatch, and share one agent batch per checkpoint (with the typed-decisions schema split)
  • LRU eviction under maxLoaded pressure
  • malformed batches rejected before anything loads
  • inference failure propagates and the router cache stays consistent
  • batchSize passthrough; schema-order-sensitive grouping; langGuess/lang_guess forwarding
  • per-request hooks: skip short-circuit, per-request onError/onPredictEnd on failure, override-based group splitting
  • wrong agent result count raises the internal-error guard; systemOne fallback
  • agent level: one forward pass by default (row offsets verified via row-distinct fake logits), batchSize chunking 2+2+1, empty states/questions skip encoding, parity with systemOne, per-state usage

npx tsc --noEmit clean; npx vitest run → 184/184 (13 files).

@aashish254

aashish254 commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: no changes needed to this PR for either fix.

Audit against two defect fixes that opened after this branch, both on the Python batched path this ports:

@aashish254

Copy link
Copy Markdown
Contributor Author

Heads-up on merge order with #398 (my lang_temperatures port): verified with git merge-tree that the two textually conflict in laya-ts/src/agent.ts only — this PR refactors _systemOneCore into _systemOneMany/_encodeItems/_decodeAnswers while #398 edits _systemOneCore in place. Purely mechanical, not semantic: whichever lands second, I rebase and move the lang param + scale-selection lines into the refactored decode function. (#386 merges clean against both.)

@aashish254

Copy link
Copy Markdown
Contributor Author

Merge-blocker note for @NandhaKishorM: the required tests (py3.9) check is stale — .github/workflows/ci.yml now runs tests (py3.10..3.13), so a check named tests (py3.9) will never report and this PR (like every open PR, also flagged on #363) sits at "Expected — Waiting for status to be reported" forever. Either an admin merge, or dropping tests (py3.9) from branch protection (ideally requiring the 3.10-3.13 names instead), unblocks it. All 19 real checks are green.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for this substantial port, the batch API and the per-request hook behaviour look right and are what we want over the other batch proposal. One regression needs fixing: _encodeItems calls sequenceWithState without Array.isArray(state), so conversation lists are truncated from the right again and the newest turn gets dropped (this was fixed in #319). Please pass that argument back in and add a systemOne and predictBatch test showing the last turn survives truncation. #398 and #386 have now landed, so please also rebase so _decodeRow takes the lang argument and emits answer_confidence.

@aashish254

Copy link
Copy Markdown
Contributor Author

Rebased onto main and addressed both points: passed Array.isArray(state) to sequenceWithState in _encodeItems with truncation tests for both systemOne and predictBatch ensuring the newest turn survives, and wired lang and answer_confidence into _decodeRow. All 273 TS tests are passing.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for the rebase and for both fixes, the truncation test for systemOne and predictBatch does exactly what we wanted, and I confirmed it fails if Array.isArray(state) is removed. One last gap: Router.predictBatch uses each request's lang only for routing and never passes it to agent.predictBatch, so lang_temperatures never apply on the batch path. With a German override, predict gives 0.7311 but predictBatch gives 0.8808. Could you forward each request's effective language the same way Router.predict does (explicit lang, otherwise the detected non-English language), add it to the grouping key so different languages do not share a forward pass, and add a router test for it? After that this is ready.

@aashish254

aashish254 commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor Author

Short answer: forwarded, grouped by, and tested.

Router.predictBatch now computes each request's effective language exactly as predict does: an explicit lang wins, otherwise the detected non-English one (analyse() names English "en", which forwards as null; Python's analyse never names it). That language joins the question-group key, so same-schema requests that differ only by language no longer share a forward pass, and it is passed to agent.predictBatch and the systemOne fallback for agents without a batch method. As in router.py's predict_batch, the key is only computed for agents that carry lang_temperatures; without overrides, lang is unused and splitting would cost a forward pass for nothing.

Your German case is the regression test: predict and predictBatch both return 0.7311 now, and if the forwarding line is removed the test fails with the reported 0.8808.

Tests: added under "Router.predictBatch lang forwarding" in tests/lang-temperature.test.ts - explicit lang wins over detection, detected language forwarded, nothing forwarded for English, two languages don't share a forward pass (same schema and checkpoint), one shared batch when the agent has no overrides, and predict/predictBatch parity. laya-ts/README.md and the predictBatch JSDoc note the split.

279 TS tests pass; tsc --noEmit and npm run test:package clean.

…#381 parity)

Router.predictBatch routed with each request's language but never passed it to
agent.predictBatch, so lang_temperatures never applied on the batched path: the
same German request scored 0.7311 through predict and 0.8808 through predictBatch.

- Compute each request's effective language exactly as predict does: an explicit
  lang wins, otherwise the detected non-English language (analyse() names English
  "en", which forwards as null -- Python's analyse never names it).
- Add it to the question-group key, so same-schema requests that differ only by
  language no longer share a forward pass. As in router.py's predict_batch, the
  key is only computed for an agent that carries lang_temperatures: otherwise
  `lang` is unused there and splitting would cost a forward pass for nothing.
- Forward it to agent.predictBatch, and to the systemOne fallback for Agent-like
  objects without a batch method.
- Tests: explicit/detected/none forwarding, the group split, no split when the
  agent carries no overrides, and predict/predictBatch parity on the reported
  German case (0.8808 vs 0.7311 if the forwarding line is dropped).
@aashish254
aashish254 force-pushed the feat-ts-predict-batch branch from 4b9772c to f1e2ac0 Compare September 28, 2026 02:41
@aashish254

Copy link
Copy Markdown
Contributor Author

Rebased onto current main and force-pushed. Resolved the conflicts in laya-ts/src/agent.ts (kept the _encodeItems/_systemOneMany/_decodeRow refactor, ported upstream's renderCriterion legend change into _decodeRow) and laya-ts/README.md. Local vitest run: 483 tests pass. The only failing check is dependency CVEs, which fails identically on every commit on main (json_repair GHSA + pre-existing vulns), so it's not introduced by this branch.

@aashish254

Copy link
Copy Markdown
Contributor Author

The only failing required check here is dependency CVEs, which is a repo-wide / upstream condition this PR cannot fix (transitive json-repair/chromadb/nltk advisories, 6 of 7 with no patched release; pip-audit --strict has no ignore list). Full breakdown in #657: #657 (comment)

@NandhaKishorM
NandhaKishorM merged commit 6a7355e into NandhaKishorM:main Sep 29, 2026
20 of 21 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @aashish254! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants