Repository navigation
fix(ts): preserve token budgets in Router batches - #1441
Conversation
jev_confidence no longer returns a value outside [0, 1] when finite weights sum to infinity (#1442), language detection flattens large states lazily (#1448), laya-ts counts and cuts code points like Python (#1445, #1446), and Router.predictBatch honours per-request token budgets (#1441). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged into 0.4.22. What made this an easy call is that the machinery was already there and already documented: Seeding the context before the start hooks is also the right order rather than a convenience: it is what lets a hook read the per-request budget, and it means the existing grouping keys on the final hook-adjusted value instead of on the request. The Verified here: 1427 laya-ts tests across 48 files, the type-check, the build, and the packed external consumer all pass on the merged branch, together with #1445 and #1446 which touch the same area. |
What
Preserve
maxLen/headMaxLenthroughRouter.predictBatch, includingRouter.decideBatchand loaders that only implementsystemOne.BatchRequestand seed each predict context before start hooks. Existing grouping uses the final hook-adjusted budgets.systemOnefallback without forwardingbatchSize.The two commits separate request/context handling from fallback forwarding and structured-entry validation.
Why
decideBatchalready copies budget options into each Router request, but the Router never read them. With a deterministic 160-token state,maxLen: 64andheadMaxLen: 16,decideencoded 64 tokens whiledecideBatchencoded 184 at the Agent's defaultmax_len=512, or only 32 atmax_len=32. After this change both batch cases encode 64. Requests with different budgets were also incorrectly grouped together.Separately, the
systemOnefallback dropped even hook-set budgets. This completes the batch path alongside the single-call fix in #1338 and structured option forwarding in #1340; Python already seeds per-request budgets before its start hooks.Omitted/null budgets still use the Agent configuration. Hooks may replace or clear an override, zero remains an explicit value, and result order is unchanged. Callers whose explicit budgets were previously ignored can now observe the intended different truncation/results. No registry, long-prediction, model or calibration changes.
How it was verified
BatchRequesttype examples failed compilation. After the first fix: all 32 passed and types compiled.EPERM; the successful full run isolatedos.homedir()to a disposable audit directory with a local Node preload, preserving the user's cache. Package installation also required network access and a writable npm cache. Neither workaround changes repository files or skips assertions.Encoding evidence uses the real Agent encoding path, its deterministic default tokenizer and a fixed-output recording provider. No model-quality or performance claim is made. Node 20, real-weight inference and browser backends were not run locally; Node 20 remains covered by the repository's CI matrix. Review checked hook precedence, null/zero semantics, grouping/order, default-hook suppression, and omission of batch-only options from the fallback.
Checklist
main(0565524a27bcb9821bcd7c42aaf49d925f6f8560)