Repository navigation
docs(http-api): say what the option budget truncates, and what it refuses - #1149
Conversation
…uses The paragraph claimed a question inside the HTTP option caps can still be refused as a 422 when the option texts together exceed that budget. Long option text inside the caps is not refused -- the head caps it at tokens_per_option and answers, silently. The refusal is option markers running past max_len, and because head_max_len is spent out of max_len rather than on top of it, a caller inside the caps reaches it only by raising the budget itself. Re-measured on c4a805b through the public call, with every figure pinned to the option shape that produced it and the boundary + 1 refusal recorded for each of the four ceilings. The 126-sentence-label ceiling also collapses the options onto one span (distinct: 1), which is the only thing that puts options in usage at all.
…ot 338 #1149 states four option ceilings and the claim that each 'answers at the ceiling and refuses one option above it'. Three reproduce exactly on the `laya` checkpoint (CPU, offline): 168 and 126 at max_len=512, and 254 for the sentence label at max_len=1024. The fourth answers one higher: with `T <i>` labels at max_len=1024 and head_max_len=192, 339 options answer and 340 refuses with 'only 339 of its 340 option markers fit'. Bisected through the public call, and the ceiling does not move with the state: the same four numbers come back for a one-word state and for the eleven-word one the paragraph above measures (the option markers are what run past max_len). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Merged, with a one-number correction from me on top ( Three of the four ceilings reproduce exactly on the
The refusal text is yours verbatim: One thing worth adding to the page, since I checked it while hunting for why 338 might have been right for you: the ceiling does not move with the state. The same four numbers come back for a one-word state and for the eleven-word one the paragraph above measures. The option markers are what run past Everything else on the page checks out, including the parts that are easiest to get backwards: that the budget truncates option text rather than refusing for it, that |
…ions too #1149 printed four ceilings without naming the instruction they were measured with, and the paragraph reads as if any caller sending that budget gets those numbers. They do not: build_head truncates the encoded `choice question: <instructions>` prefix to max(8, opt_budget) tokens, so the first option marker sits at position 7, 8 or 10 depending on the wording and every marker after it shifts. Measured through the public call on 682ac67, the four cells are 169/127/339/255 at two tokens of instruction, 168/126/339/254 at three, 168/126/338/254 at eight -- and identical at seventeen, because the prefix is saturated. The state does not move them. Pins the published 168/126/339/254 to the three-token wording rather than replacing it, and adds the instruction to what a caller has to quote when comparing "maximum N options" across servers.
What
docs/http-api.mdexplained the option caps by saying the model "fits option tokens into ahead_max_len=192window, so a question inside the HTTP caps can still be refused as a422when theoption texts together exceed that budget". That names the wrong mechanism. Long option text inside the
caps is not refused -- it is truncated to
tokens_per_optionand answered, with nothing in theresponse saying so. The refusal is option markers running past
max_len, and inside the HTTP capsthat needs the caller to raise
head_max_lenthemselves. The paragraph now says which is which, givesthe measured ceilings pinned to the option text that produced them, and says what a caller has to
quote to compare this endpoint's "maximum N options" with another server's. Docs only --
docs/http-api.md+34/-4, no code line changes.Why
Measured on the
layacheckpoint (CPU, offline) with one choice question of 100 options, each oneopt<i>plus the same 15-word description, and with the label sets"T 0"…/"Tickets routed to queue 0"…over the 11-token state"We were billed twice. Please refund the duplicate charge.".Head cost from
build_head, every answer and refusal from the publicAgent.predict:Four things follow, none of them in the page before:
head_max_len=192the head caps each of those 15-worddescriptions at 4 tokens, keeps all 100 markers in 411 head tokens, and the call answers -- while
usagecarries nooptionskey, and its two truncation-shaped keys (truncated,state_tokens_dropped) are about the state. A caller that sent long option text and checked theresponse for a signal gets none.
head_max_lenis spent out ofmax_len, not on top of it.Raising
head_max_lento 1024 gives each option 10 tokens, the head grows to 1014 tokens, andmax_len=512refuses -- whichmax_len=1024then answers.laya/agent.py:1163-1168keeps thecomment that says exactly this: naming only
head_max_len"pointed at the wrong knob in bothdirections".
that is not refused carries 168 options when each label is
T <i>and 126 when each isTickets routed to queue <i>atmax_len=512/head_max_len=192; 338 and 254 atmax_len=1024.The old paragraph implied option text was the refusal's trigger; it is an input to the ceiling
either way, just not the one that trips inside the caps.
optionsinusageis a collision signal, and the sentence ceiling collides. The 126-optionand 254-option answered calls are the only two in the transcript that report
optionsat all, andboth report
distinct: 1-- the 4-token cap gave every option the same span, so the question couldnot tell its choices apart.
usagegains the key only whenoptions_distinct < options(
laya/common.py:499-500), so the one case that surfaces the cap is a case where the cap hasalready made the question unanswerable in practice.
Which cap a caller actually meets first: both default ceilings sit above
MAX_CHOICE_OPTIONS = 100(
laya/serve.py:126), so a one-pass caller hits the count cap -- 413, not a budget refusal;tests/test_example_server_limits.py:171-172assertschoice(MAX_CHOICE_OPTIONS + 1) == 413. Abudget refusal needs the caller to raise
max_lenorhead_max_len, each accepted up toLAYA_MAX_TOKEN_BUDGET(8192,laya/serve.py:147).MAX_TOURNAMENT_CHOICE_OPTIONS = 255(
laya/serve.py:130) raises the count but narrows the choices into groups, so it does not reach theone-pass budget.
One arithmetic point worth pre-empting, because the paragraph prints both numbers: the head alone is
411 tokens (
build_head), the answered call'susage.input_tokensis 423, and the state is 11 tokens(
build_sequenceon the same question reportsstate_tokens=11,state_tokens_dropped=0,truncated=False). So 423 is not head + state -- the encoded sequence carries one token more thanthose two. The paragraph therefore says "the state's 11 tokens among them" rather than presenting 423
as a sum.
The probe that produced the transcript imports this branch's tree through a
sys.path.insertandasserts
laya.__file__starts with it, so the numbers can only come from the code this PR is basedon.
How it was verified
The new prose is checked in the rendered page, not just the source: parsing
site/http-api/index.html, stripping tags and collapsing whitespace, each of these literals appearsexactly once and each stale figure exactly zero times --
Every figure the paragraph prints appears verbatim in the transcript above, including the refusal
text, which is quoted from the exception rather than paraphrased.
Deconfliction and scanning
gh pr list --state open --limit 200 --json number,createdAt,title,files(73 open PRs as of2026-10-12T01:33Z): the only open PR touching
docs/http-api.mdis perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687, and its hunks here are@@ -128,10 +128,12 @@,@@ -166,8 +168,7 @@and@@ -210,14 +211,19 @@, while this change is@@ -361,10 +361,40 @@. No shared line, so the two diffs cannot collide.bffc4974onc4a805b5, whichgit fetch upstream+git rev-parse upstream/mainstill shows as current
main(release 0.4.3) as of 01:33Z;git rev-list --count c4a805b5..upstream/mainis 0.
Checklist
mainFixes #