Skip to content

fix(serve): a caller batch_size may shrink a pass, never widen one - #1332

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
thread3d:fix/serve-batch-size-cap
Oct 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
thread3d:fix/serve-batch-size-cap

Conversation

@thread3d

Copy link
Copy Markdown
Contributor

What

/v1/systemone/batch computes the token-budget plan for every request and caps the caller's batch_size at it. A smaller value is still honoured; a request that already fits is still passed no batch_size at all.

The old contract ("the caller's own batch_size always wins") is changed deliberately, and docs/http-api.md says so.

Why

The plan that keeps one forward pass inside LAYA_MAX_BATCH_TOKENS was only consulted when the body did not name a batch_size, and the validator had no upper bound. batch_size=10**18 on a 4 KB body of 64 states at max_len 8192 collated 524 288 tokens in one pass -- four times the budget -- so the documented amplification guard was opt-out by the very request it protects against.

How it was verified

python -m pytest tests/test_serve.py

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (branch cut from upstream/main at 0.4.14)
  • Tests pass locally
  • Docs or examples updated when the public API changed

/v1/systemone/batch plans a batch_size from LAYA_MAX_BATCH_TOKENS so one forward pass
stays inside a token budget, but only when the body does not name one: any value the
caller sent was forwarded verbatim, with no upper bound. So the guard was opt-out by the
request it protects -- a 4 KB body of 64 states at max_len 8192 is 524 288 tokens, four
times the budget, and batch_size=10**18 sent it all in one pass.

The plan is now computed for every request and the caller's value is capped at it. A
smaller value is still honoured (asking for a narrower pass asks for less work), and a
request that already fits is still passed no batch_size at all, so the byte-identical
path is unchanged. The old test asserted the previous contract ('the caller asked for a
shape'); it now asserts both directions, and docs/http-api.md says which one wins.
@thread3d thread3d closed this Oct 24, 2026
@thread3d thread3d reopened this Oct 24, 2026
@NandhaKishorM
NandhaKishorM merged commit cdb0cb2 into NandhaKishorM:main Oct 24, 2026
97 of 111 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thread3d
thread3d deleted the fix/serve-batch-size-cap branch October 28, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants