Skip to content

feat(ts): add Agent and Router batch prediction - #475

Closed
vp337247 wants to merge 3 commits into
NandhaKishorM:mainfrom
vp337247:feat/laya-ts-batch-prediction
Closed

vp337247 wants to merge 3 commits into
NandhaKishorM:mainfrom
vp337247:feat/laya-ts-batch-prediction

Conversation

@vp337247

Copy link
Copy Markdown

What

Adds batch prediction support to the official TypeScript runtime.

Agent

  • Adds Agent.predictBatch()
  • Encodes multiple states together and performs shared encoder/head forward passes
  • Supports optional batchSize / batch_size chunking
  • Preserves input/result ordering
  • Preserves existing hook behavior
  • Keeps existing predict() and systemOne() behavior compatible
  • Preserves the latest answer_confidence and language-temperature behavior

Router

  • Adds Router.routeBatch()
  • Adds Router.predictBatch()
  • Routes requests before model loading
  • Groups requests by routed checkpoint
  • Groups compatible requests by question schema and token-budget configuration
  • Preserves original request ordering
  • Supports partial ctx.skip()
  • Preserves hook lifecycle and error handling

Documentation

  • Added TypeScript batch prediction documentation and examples to laya-ts/README.md

Why

The Python runtime already supports batch prediction, while the official
TypeScript runtime did not expose equivalent Agent/Router batch APIs.

This brings the TypeScript runtime closer to the existing Python
capabilities while keeping batching within the existing local runtime.

How it was verified

The following checks have already passed:

cd laya-ts
npm test -- --run      # 281 TypeScript tests passed (16 test files)
npm run build          # TypeScript compilation and dist packaging passed
npm run test:package   # End-to-end packed npm module test passed
git diff --check       # Clean (zero whitespace/lint errors)

Performance

Agent.predictBatch() uses shared 2D encoder/head forward passes for
multiple states, with optional batchSize chunking, instead of invoking
the model independently for every state.

No benchmark numbers are claimed in this PR.

Scope

This PR intentionally does not include:

  • LangChain.js integration
  • LangGraph integration
  • predictMany
  • systemOneBatch
  • Python runtime changes

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for putting so much work into batch prediction for the TypeScript runtime, the ordering, chunking and hook handling are carefully done and the tests are thorough. We already have #329 open for the same feature, and we have chosen its design because it mirrors Python's predict_batch API and per-request hooks, so we are going to close this one to avoid two competing implementations. One thing we found that may help if you would like to contribute to #329: the batched router path should forward each request's lang to the agent, and group by it, so lang_temperatures apply as they do in Router.predict. Thanks again, and we would be glad to see your review or follow-ups on #329.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants