Skip to content

docs(http-api): say what the option budget truncates, and what it refuses - #1149

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:docs-option-budget-remedy
Oct 12, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:docs-option-budget-remedy

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

docs/http-api.md explained the option caps by saying the model "fits option tokens into a
head_max_len=192 window, so a question inside the HTTP caps can still be refused as a 422 when the
option texts together exceed that budget". That names the wrong mechanism. Long option text inside the
caps is not refused -- it is truncated to tokens_per_option and answered, with nothing in the
response saying so. The refusal is option markers running past max_len, and inside the HTTP caps
that needs the caller to raise head_max_len themselves. The paragraph now says which is which, gives
the measured ceilings pinned to the option text that produced them, and says what a caller has to
quote to compare this endpoint's "maximum N options" with another server's. Docs only --
docs/http-api.md +34/-4, no code line changes.

Why

Measured on the laya checkpoint (CPU, offline) with one choice question of 100 options, each one
opt<i> plus the same 15-word description, and with the label sets "T 0"… / "Tickets routed to queue 0"… over the 11-token state "We were billed twice. Please refund the duplicate charge.".
Head cost from build_head, every answer and refusal from the public Agent.predict:

[state] 'We were billed twice. Please refund the duplicate charge.' -> state_tokens=11 dropped=0

[head] 100 options, each 'opt<i>: <15-word description>'
  head_max_len=192   head_tokens=411   markers=100  {'options': 100, 'options_distinct': 100, 'tokens_per_option': 4}
  head_max_len=512   head_tokens=414   markers=100  {'options': 100, 'options_distinct': 100, 'tokens_per_option': 4}
  head_max_len=1024  head_tokens=1014  markers=100  {'options': 100, 'options_distinct': 100, 'tokens_per_option': 10}

[predict] the 100-option, 15-word-description question
  max_len=512   head_max_len=192   n=100  ANSWERED choice='opt0'  input_tokens=423   options=absent
  max_len=512   head_max_len=1024  n=100  ValueError: question 'q': only 50 of its 100 option markers fit in max_len=512 with head_max_len=1024 spent on the question; lower head_max_len, raise max_len, or use fewer options
  max_len=1024  head_max_len=192   n=100  ANSWERED choice='opt0'  input_tokens=423   options=absent
  max_len=1024  head_max_len=1024  n=100  ANSWERED choice='opt10' input_tokens=1024  options=absent

[predict] label-only ceilings, boundary and boundary+1
  'T 0' at max_len=512/head_max_len=192:
  max_len=512   head_max_len=192  n=168  ANSWERED choice='T 1'                  input_tokens=512   options=absent
  max_len=512   head_max_len=192  n=169  ValueError: ... only 168 of its 169 option markers fit in max_len=512 with head_max_len=192 ...
  'Tickets routed to queue 0' at max_len=512/head_max_len=192:
  max_len=512   head_max_len=192  n=126  ANSWERED choice='Tickets routed to queue 0' input_tokens=512 options={'q': {'total': 126, 'distinct': 1, 'tokens_per_option': 4}}
  max_len=512   head_max_len=192  n=127  ValueError: ... only 126 of its 127 option markers fit in max_len=512 ...
  'T 0' at max_len=1024/head_max_len=192:
  max_len=1024  head_max_len=192  n=338  ANSWERED choice='T 7'   input_tokens=1024  options=absent
  max_len=1024  head_max_len=192  n=339  ValueError: ... only 338 of its 339 option markers fit in max_len=1024 ...
  'Tickets routed to queue 0' at max_len=1024/head_max_len=192:
  max_len=1024  head_max_len=192  n=254  ANSWERED input_tokens=1024 options={'q': {'total': 254, 'distinct': 1, 'tokens_per_option': 4}}
  max_len=1024  head_max_len=192  n=255  ValueError: ... only 254 of its 255 option markers fit in max_len=1024 ...

Four things follow, none of them in the page before:

  1. Truncation is silent. At the default head_max_len=192 the head caps each of those 15-word
    descriptions at 4 tokens, keeps all 100 markers in 411 head tokens, and the call answers -- while
    usage carries no options key, and its two truncation-shaped keys (truncated,
    state_tokens_dropped) are about the state. A caller that sent long option text and checked the
    response for a signal gets none.
  2. The refusal is about markers, and head_max_len is spent out of max_len, not on top of it.
    Raising head_max_len to 1024 gives each option 10 tokens, the head grows to 1014 tokens, and
    max_len=512 refuses -- which max_len=1024 then answers. laya/agent.py:1163-1168 keeps the
    comment that says exactly this: naming only head_max_len "pointed at the wrong knob in both
    directions".
  3. How many options fit at all depends on their text. With label-only options the largest question
    that is not refused carries 168 options when each label is T <i> and 126 when each is
    Tickets routed to queue <i> at max_len=512/head_max_len=192; 338 and 254 at max_len=1024.
    The old paragraph implied option text was the refusal's trigger; it is an input to the ceiling
    either way, just not the one that trips inside the caps.
  4. options in usage is a collision signal, and the sentence ceiling collides. The 126-option
    and 254-option answered calls are the only two in the transcript that report options at all, and
    both report distinct: 1 -- the 4-token cap gave every option the same span, so the question could
    not tell its choices apart. usage gains the key only when options_distinct < options
    (laya/common.py:499-500), so the one case that surfaces the cap is a case where the cap has
    already made the question unanswerable in practice.

Which cap a caller actually meets first: both default ceilings sit above MAX_CHOICE_OPTIONS = 100
(laya/serve.py:126), so a one-pass caller hits the count cap -- 413, not a budget refusal;
tests/test_example_server_limits.py:171-172 asserts choice(MAX_CHOICE_OPTIONS + 1) == 413. A
budget refusal needs the caller to raise max_len or head_max_len, each accepted up to
LAYA_MAX_TOKEN_BUDGET (8192, laya/serve.py:147). MAX_TOURNAMENT_CHOICE_OPTIONS = 255
(laya/serve.py:130) raises the count but narrows the choices into groups, so it does not reach the
one-pass budget.

One arithmetic point worth pre-empting, because the paragraph prints both numbers: the head alone is
411 tokens (build_head), the answered call's usage.input_tokens is 423, and the state is 11 tokens
(build_sequence on the same question reports state_tokens=11, state_tokens_dropped=0,
truncated=False). So 423 is not head + state -- the encoded sequence carries one token more than
those two. The paragraph therefore says "the state's 11 tokens among them" rather than presenting 423
as a sum.

The probe that produced the transcript imports this branch's tree through a sys.path.insert and
asserts laya.__file__ starts with it, so the numbers can only come from the code this PR is based
on.

How it was verified

python -B tests/test_example_server_limits.py   # 155 passed, 0 failed
python -B tests/test_env_docs.py                # 63 passed, 0 failed
python -B scripts/test_suites.py                # rc=0, "all 87 suites passed"; 497 passed, 1 skipped in 15.65s
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/            # rc=0
zensical build --strict --clean                 # rc=0, "No issues found", finished in 1.42s

The new prose is checked in the rendered page, not just the source: parsing
site/http-api/index.html, stripping tags and collapsing whitespace, each of these literals appears
exactly once and each stale figure exactly zero times --

1  only 50 of its 100 option markers fit in max_len=512 with head_max_len=1024 spent on the question; ...
1  markers in 411 tokens
1  input_tokens=423
1  the head grows to 1014
1  338 and 254
1  options: {"q": {"total": 126, "distinct": 1, "tokens_per_option": 4}}
1  the budget it sent and the option text, not just N
1  pointed at the wrong knob in both directions
0  409 tokens     0  only 51     0  339

Every figure the paragraph prints appears verbatim in the transcript above, including the refusal
text, which is quoted from the exception rather than paraphrased.

Deconfliction and scanning

  • gh pr list --state open --limit 200 --json number,createdAt,title,files (73 open PRs as of
    2026-10-12T01:33Z): the only open PR touching docs/http-api.md is perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687, and its hunks here are
    @@ -128,10 +128,12 @@, @@ -166,8 +168,7 @@ and @@ -210,14 +211,19 @@, while this change is
    @@ -361,10 +361,40 @@. No shared line, so the two diffs cannot collide.
  • One commit bffc4974 on c4a805b5, which git fetch upstream + git rev-parse upstream/main
    still shows as current main (release 0.4.3) as of 01:33Z; git rev-list --count c4a805b5..upstream/main
    is 0.
  • The diff and this body were scanned for credentials and absolute local paths: rc=1, no matches.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed (this is the docs change; no API change)

Fixes #

…uses

The paragraph claimed a question inside the HTTP option caps can still be
refused as a 422 when the option texts together exceed that budget. Long
option text inside the caps is not refused -- the head caps it at
tokens_per_option and answers, silently. The refusal is option markers
running past max_len, and because head_max_len is spent out of max_len rather
than on top of it, a caller inside the caps reaches it only by raising the
budget itself.

Re-measured on c4a805b through the public call, with every figure pinned to
the option shape that produced it and the boundary + 1 refusal recorded for
each of the four ceilings. The 126-sentence-label ceiling also collapses the
options onto one span (distinct: 1), which is the only thing that puts
options in usage at all.
@NandhaKishorM
NandhaKishorM merged commit a47f4ce into NandhaKishorM:main Oct 12, 2026
40 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 12, 2026
…ot 338

#1149 states four option ceilings and the claim that each 'answers at the ceiling and
refuses one option above it'. Three reproduce exactly on the `laya` checkpoint (CPU,
offline): 168 and 126 at max_len=512, and 254 for the sentence label at max_len=1024.
The fourth answers one higher: with `T <i>` labels at max_len=1024 and head_max_len=192,
339 options answer and 340 refuses with 'only 339 of its 340 option markers fit'.

Bisected through the public call, and the ceiling does not move with the state: the same
four numbers come back for a one-word state and for the eleven-word one the paragraph
above measures (the option markers are what run past max_len).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged, with a one-number correction from me on top (d4f5740).

Three of the four ceilings reproduce exactly on the laya checkpoint here, CPU and offline, bisected through the public call:

label max_len you say measured
T <i> 512 168 168 answers, 169 refuses
Tickets routed to queue <i> 512 126 126 answers, 127 refuses
Tickets routed to queue <i> 1024 254 254 answers, 255 refuses
T <i> 1024 338 339 answers, 340 refuses

The refusal text is yours verbatim: only 169 of its 169 option markers fit in max_len=512 with head_max_len=192 spent on the question. The fourth row is the only one off, and it matters because the paragraph claims each boundary "answers at the ceiling and refuses one option above it", which is false at 338 since 339 also answers. Changed to 339.

One thing worth adding to the page, since I checked it while hunting for why 338 might have been right for you: the ceiling does not move with the state. The same four numbers come back for a one-word state and for the eleven-word one the paragraph above measures. The option markers are what run past max_len, and the state is what gets dropped to make room, so a reader cannot recover a different ceiling by sending a shorter state.

Everything else on the page checks out, including the parts that are easiest to get backwards: that the budget truncates option text rather than refusing for it, that head_max_len is spent out of max_len rather than on top of it (which is why raising it is how a caller reaches the refusal), and that both default ceilings sit above MAX_CHOICE_OPTIONS=100 so a one-pass HTTP caller meets the count cap first and sees a 413 rather than a budget refusal. The options usage entry appearing only on a token-span collision is a genuinely useful thing to have written down, since that is the case where a question cannot tell its own choices apart.

NandhaKishorM pushed a commit that referenced this pull request Oct 12, 2026
…ions too

#1149 printed four ceilings without naming the instruction they were measured with, and the
paragraph reads as if any caller sending that budget gets those numbers. They do not: build_head
truncates the encoded `choice question: <instructions>` prefix to max(8, opt_budget) tokens, so the
first option marker sits at position 7, 8 or 10 depending on the wording and every marker after it
shifts. Measured through the public call on 682ac67, the four cells are 169/127/339/255 at two
tokens of instruction, 168/126/339/254 at three, 168/126/338/254 at eight -- and identical at
seventeen, because the prefix is saturated. The state does not move them.

Pins the published 168/126/339/254 to the three-token wording rather than replacing it, and adds the
instruction to what a caller has to quote when comparing "maximum N options" across servers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants