Repository navigation
Conversation
Owner
|
Thank you for putting so much work into batch prediction for the TypeScript runtime, the ordering, chunking and hook handling are carefully done and the tests are thorough. We already have #329 open for the same feature, and we have chosen its design because it mirrors Python's predict_batch API and per-request hooks, so we are going to close this one to avoid two competing implementations. One thing we found that may help if you would like to contribute to #329: the batched router path should forward each request's lang to the agent, and group by it, so lang_temperatures apply as they do in Router.predict. Thanks again, and we would be glad to see your review or follow-ups on #329. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds batch prediction support to the official TypeScript runtime.
Agent
Agent.predictBatch()batchSize/batch_sizechunkingpredict()andsystemOne()behavior compatibleanswer_confidenceand language-temperature behaviorRouter
Router.routeBatch()Router.predictBatch()ctx.skip()Documentation
laya-ts/README.mdWhy
The Python runtime already supports batch prediction, while the official
TypeScript runtime did not expose equivalent Agent/Router batch APIs.
This brings the TypeScript runtime closer to the existing Python
capabilities while keeping batching within the existing local runtime.
How it was verified
The following checks have already passed:
Performance
Agent.predictBatch()uses shared 2D encoder/head forward passes formultiple states, with optional
batchSizechunking, instead of invokingthe model independently for every state.
No benchmark numbers are claimed in this PR.
Scope
This PR intentionally does not include:
predictManysystemOneBatchChecklist
main