Skip to content

fix(ts): preserve token budgets in Router batches - #1441

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
xinpingzh:fix/ts-router-batch-budgets
Nov 3, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
xinpingzh:fix/ts-router-batch-budgets

Conversation

@xinpingzh

Copy link
Copy Markdown
Contributor

What

Preserve maxLen / headMaxLen through Router.predictBatch, including Router.decideBatch and loaders that only implement systemOne.

  • Add the optional budget fields to BatchRequest and seed each predict context before start hooks. Existing grouping uses the final hook-adjusted budgets.
  • Forward non-null budgets through the systemOne fallback without forwarding batchSize.
  • Add request/grouping, hook override/reset, fallback, compile-time and real Agent encoding regressions; document per-request budgets.

The two commits separate request/context handling from fallback forwarding and structured-entry validation.

Why

decideBatch already copies budget options into each Router request, but the Router never read them. With a deterministic 160-token state, maxLen: 64 and headMaxLen: 16, decide encoded 64 tokens while decideBatch encoded 184 at the Agent's default max_len=512, or only 32 at max_len=32. After this change both batch cases encode 64. Requests with different budgets were also incorrectly grouped together.

Separately, the systemOne fallback dropped even hook-set budgets. This completes the batch path alongside the single-call fix in #1338 and structured option forwarding in #1340; Python already seeds per-request budgets before its start hooks.

Omitted/null budgets still use the Agent configuration. Hooks may replace or clear an override, zero remains an explicit value, and result order is unchanged. Callers whose explicit budgets were previously ignored can now observe the intended different truncation/results. No registry, long-prediction, model or calibration changes.

How it was verified

cd laya-ts
npm test -- --run
npm run test:package
python scripts/predict_long_fixtures.py --check
python scripts/tokenizer_decode_fixtures.py --check
cd ..
python tests/test_hooks_api.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
git diff --check
  • Before the first fix: 3 new request-budget tests failed, 29 existing router-batch tests passed, and the new BatchRequest type examples failed compilation. After the first fix: all 32 passed and types compiled.
  • Before the second fix: both new fallback cases failed. After it: the affected router-batch and structured suites passed all 106 tests.
  • Full result on Node 22.15.1: 48 files / 1,425 tests passed, including the type-check step. The packed external consumer (including build), 679 Python API checks, both fixture drift checks, lint and compilation passed.
  • Existing revision-pinning tests write/remove the home cache. Their initial sandbox run hit EPERM; the successful full run isolated os.homedir() to a disposable audit directory with a local Node preload, preserving the user's cache. Package installation also required network access and a writable npm cache. Neither workaround changes repository files or skips assertions.

Encoding evidence uses the real Agent encoding path, its deterministic default tokenizer and a fixed-output recording provider. No model-quality or performance claim is made. Node 20, real-weight inference and browser backends were not run locally; Node 20 remains covered by the repository's CI matrix. Review checked hook precedence, null/zero semantics, grouping/order, default-hook suppression, and omission of batch-only options from the fallback.

Checklist

  • Focused on one bug fix
  • Based on latest main (0565524a27bcb9821bcd7c42aaf49d925f6f8560)
  • Regression tests fail before the fixes and pass afterward; relevant checks pass locally
  • Request types, documentation and compile-time coverage updated

@NandhaKishorM
NandhaKishorM merged commit 6ec112b into NandhaKishorM:main Nov 3, 2026
65 checks passed
NandhaKishorM added a commit that referenced this pull request Nov 3, 2026
jev_confidence no longer returns a value outside [0, 1] when finite
weights sum to infinity (#1442), language detection flattens large
states lazily (#1448), laya-ts counts and cuts code points like
Python (#1445, #1446), and Router.predictBatch honours per-request
token budgets (#1441).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged into 0.4.22. What made this an easy call is that the machinery was already there and already documented: router.ts on main grouped by g.maxLen === ctx.maxLen and forwarded group.maxLen to the batch options, and the predictBatch docstring already told callers their requests would be grouped when they "set different maxLen/headMaxLen overrides". decideBatch already copied the budgets in. The one missing link was seeding ctx.maxLen from requests[i], so a documented feature was dead on arrival for every batch caller.

Seeding the context before the start hooks is also the right order rather than a convenience: it is what lets a hook read the per-request budget, and it means the existing grouping keys on the final hook-adjusted value instead of on the request. The systemOne fallback half matters too, since a loader that implements only systemOne was silently dropping even a hook-set budget, and your choice not to forward batchSize there is correct because that option has no meaning for a one-state call.

Verified here: 1427 laya-ts tests across 48 files, the type-check, the build, and the packed external consumer all pass on the merged branch, together with #1445 and #1446 which touch the same area.

@xinpingzh xinpingzh mentioned this pull request Nov 6, 2026
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants