Repository navigation
docs(http-api): the option ceiling moves with the question's instructions too - #1151
Conversation
…ions too NandhaKishorM#1149 printed four ceilings without naming the instruction they were measured with, and the paragraph reads as if any caller sending that budget gets those numbers. They do not: build_head truncates the encoded `choice question: <instructions>` prefix to max(8, opt_budget) tokens, so the first option marker sits at position 7, 8 or 10 depending on the wording and every marker after it shifts. Measured through the public call on 682ac67, the four cells are 169/127/339/255 at two tokens of instruction, 168/126/339/254 at three, 168/126/338/254 at eight -- and identical at seventeen, because the prefix is saturated. The state does not move them. Pins the published 168/126/339/254 to the three-token wording rather than replacing it, and adds the instruction to what a caller has to quote when comparing "maximum N options" across servers.
|
Merged in 0.4.5. Every number reproduces, which is worth saying plainly because this PR and #1149 appeared to contradict each other and neither was wrong. Bisected on the Columns are So 338 and 339 were both measured correctly, with different instructions. I corrected #1149's 338 to 339 after bisecting it myself, and that correction stands for the three-token wording the published cells use. Keeping those cells and labelling them is the right call rather than replacing them. The instruction is now the third thing a caller has to quote. That makes the paragraph less quotable and more true. |
What
Reworks the option-ceiling paragraph in
docs/http-api.md(the one #1149 added) so the fourceilings it publishes are pinned to the question
instructionsthey were measured with, and namesthe mechanism that makes them move.
Before → after, same file, same section:
and by the question's own
instructions", with the baseline cells now labelled as measured withthe three-token instruction
Which queue?.an eight-token one gives 168, 126, 338 and 254, and a seventeen-token one gives the
eight-token numbers exactly.
laya/common.py::build_headtruncates the encodedchoice question: <instructions>prefix tomax(8, opt_budget)tokens (laya/common.py:259), sothe first
[MASK]lands at position 7, 8 or 10 for those three wordings and every marker afterit shifts with it.
max_len=1024withthe eight-token instruction whether the state is one word or the eleven-token sentence the section
already uses.
option text and the instruction, not just N".
The published
168 / 126 / 339 / 254are kept, not replaced: they are what a three-tokeninstruction gives, which is the wording the earlier measurements used.
Why
Those four numbers were printed in #1149 without the instruction they were taken with, so the
paragraph reads as if any caller sending
max_len=512/1024athead_max_len=192gets them. Itdoes not. The head is a fixed-length prefix plus one marker per option, and
build_headtruncatesthat prefix to
max(8, opt_budget)tokens, so a shorter instruction leaves more room for markersand the ceiling rises; once the instruction reaches the 8-token cap the prefix is saturated and the
ceiling stops moving.
That is also what reconciles the follow-up correction in
d4f5740c("the 1024-budget ceiling forthe short label is 339, not 338"): 339 and 338 are both correct, for a three-token and an
eight-or-longer-token instruction respectively. Naming the instruction is what makes the section
reproducible instead of arguable — a caller comparing this endpoint against another server's stated
"maximum N options" needs the budget, the option text and the instruction, and N alone is not one of
the three.
How it was verified
Every cell in the paragraph was met through the public call on this exact tree (
682ac671,release 0.4.4), CPU and offline, answering at the ceiling and refusing one option above it. The
driver asserts
laya.__file__resolves inside the worktree before loading the checkpoint, so thetranscript below is from this branch's code and not an installed copy.
Each refusal is the budget error the section already quotes, e.g.
question 'q': only 168 of its 169 option markers fit in max_len=512 with head_max_len=192 spent on the question; lower head_max_len, raise max_len, or use fewer options.One wording error was caught while re-checking the paragraph against the probe: it said
"an eleven-word state" for a sentence that is nine words and eleven tokens. The paragraph now says
"the eleven-token sentence this section has been using", which matches the
state_tokens=11figurein the paragraph above it.
Gates run on this tree:
The rendered page was then parsed with tags stripped and whitespace collapsed, and each claim the
paragraph makes was required to appear exactly once:
169, 127, 339 and 255, an eight-token one gives 168, 126, 338 and 254,max(8, opt_budget),lands at position 7, 8 or 10,the same seventeen-token wording returns the eight-token numbers exactly,338 options answer and 339 refuse whether the state is one word or the eleven-token sentence this section has been using,the budget it sent, the option text and the instruction, not just N. Both superseded strings —eleven-wordand
the budget it sent and the option text, not just N— count zero, and the single remaining338 and 254is the qualified eight-token cell.[verdict] failures = 0.Line lengths: the file's own p90 is 101 characters and its longest line is 341, so the five new
101–103 character lines match the surrounding convention; nothing else was reflowed.
Deconfliction
43 open PRs as of 2026-10-12T04:46Z. Only one of them touches
docs/http-api.md— #687, whosehunks in that file are at lines 128, 166 and 210, all above this change at 379–406. Disjoint.
This is not linked to an open issue; it is a correction to text that merged in #1149 and to the
ceiling disagreement in
d4f5740c.Checklist
main(682ac671, release 0.4.4)