Repository navigation
Conversation
52a4be2 to
b2958c4
Compare
ade029e to
eac5d9d
Compare
|
Holding this for 0.3.23, and the reason is the question you yourself raised rather than anything I found wrong with the code. You flagged that
Your option 1 is the one I want: keep Nothing else blocks this. #689 landed in this release on its own and did not conflict with your branch. Rebase onto |
c5bc1a4 to
10c3703
Compare
Done — option 1 exactly as you described it, rebased onto
The result I most wanted to show you — asking gives back Why Available on Three things worth flagging:
CI: 24 checks pass, 1 skipped ( Taking it out of draft. |
Bruce-Yii
left a comment
There was a problem hiding this comment.
Rechecked this head specifically against the two conditions Nandha left before taking it, rather than re-auditing the whole PR. It is rebased onto current main, and option 1 is wired as requested: the default usage block omits only state_tokens / state_tokens_dropped; state_token_counts=True switches back to full state tokenization so those counts are exact; truncated / truncated_questions remain always available.
I also traced the flag through Agent, ONNXAgent, Router, both HTTP paths and the CLI. It is forwarded only when enabled, with the Router compatibility retry preserving older attached Agent-like objects. CI is green on the current head.
I did not independently rerun the author's real-checkpoint 9-shape equivalence matrix, so I am treating that part as author evidence. The original rebase + opt-in-counts blocker itself looks closed.
|
@NandhaKishorM PR pending your review. |
|
Not merging from this branch. It is 767 commits behind The idea is right and I want it: tokenizing only the prefix of What a rebase has to reckon with, and why I am not doing it for you:
The thing that would make this merge easily is a measurement, which the current description does not have: tokenization time saved as a fraction of total predict time, on a long state, at a couple of state lengths. The change is justified by waste, so the size of the waste is the argument. If it turns out to be small on realistic states, the opt-in contract change is not worth it; if it is large, that number is what settles the Leaving this open. A narrower first PR, just the prefix tokenization with |
…ake the state-token counts opt-in build_sequence keeps state_ids[:room] (room < max_len) for every state but a conversation list, so the tail was tokenized only to be discarded. encode_state_head tokenizes a prefix cut where the cut cannot change the ids before it, and falls back to the full tokenization otherwise. usage["state_tokens"] / ["state_tokens_dropped"] count the whole state, so they are now absent unless state_token_counts=True, which restores the full tokenization for that call. truncated / truncated_questions stay always-on and exact. predict_long tokenizes in full (every window is scored) through the same encode_state, and forwards the flag on both of its paths. Surfaces: Agent, ONNXAgent, Router, laya.serve, CLI, MCP remote, the TypeScript SDK; the golden generators ask for the counts so regenerated fixtures keep them.
… absent keys are absent and strict usage is fixed
…nnot take it; MCP tools and hooks carry it Router.predict, predict_batch and predict_long now check the agent's signature and raise StateTokenCountsUnsupported (a TypeError and ValueError, so laya.serve answers 422) instead of one path raising a raw TypeError and the batch retry silently dropping the counts. RemoteRouter.predict raised KeyError on every call, because state_token_counts is in BODY_CONTROLS; it now sends the flag through the same wire-controls loop. laya_predict / laya_predict_long / laya_predict_batch take state_token_counts. PredictContext.state_token_counts tells a result cache to key on it. Tests: the four refusal paths, serve forwarding on the tournament and predict fallback, CLI --batch, RemoteRouter bodies, MCP tools, hook context, and encode_state_head against a real tokenizers object.
…surfaces instead of being retried away
… so positional construction is unchanged; re-point a shifted citation
dc85053 to
8cb3dc0
Compare
…nly common.py, generators ask for the counts - common.py held a literal U+2581; test_criteria reads it with the platform codec and failed on Windows (cp1252). Written as an escape now. - The two counts are placed after output_tokens again (state_counts_in_place), so the usage dict keeps the key order every SDK golden compares; laya-go/fixtures and laya-java/fixtures regenerate byte-identical to main. - The tournament recorder, the Go predict_plan generator and the .NET predict_long dumper ask for the counts, as the other generators already did. - test_state_token_counts pins the key order (fails when the helper is a no-op).
…tate counts Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
374f907 to
10125bd
Compare
perf(agent): tokenize only as much state as a prefix-truncated sequence can hold, and make the two state-token counts opt-in
laya/common.py+367 ·laya/agent.py+76/-26 ·laya/onnx_agent.py+45/-20 ·laya/router.py+23/-4 ·laya/serve.py+38/-2 ·laya/cli.py+20/-2 ·tests/test_truncation_direction.py+1207 ·tests/test_state_token_counts.py+295 (new) ·docs, examples and
scripts/test_suites.pyBranch
perf/state-prefix-tokenization, 6 commits onupstream/main@fa9a2a7. 20 files,+2301/-116.
In short
The problem.
Agent._encode_statetokenizes the entire serialized state.build_sequencethenkeeps
state_ids[:room], androomis always belowmax_len. For every state shape except aconversation list, the tokens past that slice were produced only to be thrown away.
The fix. Measure the state's density on a small probe (at most 512 characters, a sixty-fourth of
the state), then slice the serialized text to the prefix that density says will fill the budget, at a
cut point that cannot change how the tokenizer reads what comes before it, and tokenize only that. If
it still comes up short, retry from the density that attempt measured — at most three attempts — then
tokenize the whole state as before: correct, just not faster. The prefix is taken only when it is
worth taking, and only for a tokenizer whose added tokens, pre-tokenizer and normalizer are each
checked to permit the cut.
The consequence, and the half of this PR you asked for. Reading only a prefix means
usage["state_tokens"]andusage["state_tokens_dropped"]can no longer be counts of the wholestate — they would be floors. So they are now opt-in: absent unless the caller passes
state_token_counts=True, and asking restores the full tokenization for that call so what comes backis exact.
truncatedandtruncated_questionsstay always-on and free, and that is sound ratherthan convenient — see The two counts are opt-in below. This follows
usage["options"]from #569,by the same mechanism in the same function.
Where it matters, and where it does not. Over HTTP
laya.servecaps a state at 50 000characters, so the win there is ~9 ms of tokenizing (10.22 ms → 0.95 ms, 10.8×), which is
8.7% of a one-question request on the english checkpoint and 3.0% of a three-question one. A
trivial request is unchanged. The large multiples need a direct library call with a state bigger
than the HTTP cap: a 25 000-field dict serializes to 500 000 characters and goes from 214 ms of
tokenizing to 0.84 ms (254×), which is 64.9% of that request. States with no space in reach —
CJK without spaces, base64 — keep the old cost, within noise; see No path gets slower below for the
one shape where the probe is a measurable cost.
What does not change. Any answer. 10 485 differential comparisons against both real checkpoints'
tokenizers, 0 differences; 9 real-model request shapes on each of english and multilingual, 0
answer differences against
upstream/mainwith the counts asked for and without; andONNXAgentanswering through a real fused ONNX graph in
onnxruntime.What does change for a caller. Those two
usagekeys are absent unless asked for. That is abreak, it is deliberate, and it is the one you chose over a count that is exact on one path and a
floor on another. Two of this repo's own examples read them and are updated here — see What this
breaks, and what it does not.
In one line: the sequence only ever held a prefix of the state, so now we only tokenize a prefix —
and the two counts that describe the whole state are asked for rather than guessed at.
Every line added and removed, and why
Removed — 12 lines, all of it replaced rather than dropped.
encode_text(self.tok, serialize_state(state).replace(...), add_special_tokens=False)["input_ids"]callagent.py,onnx_agent.py(once each)text = ...plus a two-path choice. The identical call is retained verbatim for thetruncate_leftbranch, so conversation lists tokenize exactly as before.state_ids")No function, parameter, test or public name is deleted. This change adds no new tokenizer entry
point — both call sites it touches route through
encode_text, as they did — andbuild_sequence'scontract is untouched. (
encode_textis not the only entry point onmain:Agent.predict_longcalls
self.tok(...)directly atagent.py:1177. That line is untouched here, andONNXAgent'sequivalent already goes through
encode_text.)Added — what each piece is for. Nothing here is defensive decoration; each item exists because
something measurable went wrong without it.
encode_state_head_state_cutSTATE_HEAD_CHARS_PER_TOKEN = 8_STATE_HEAD_PROBE_CHARS/_STATE_HEAD_PROBE_MIN_STATE_HEAD_MIN_RATIO = 1.25_STATE_HEAD_MAX_ATTEMPTS = 3_added_tokens_permit_cut_pre_tokenizer_permits_cut+_splits_on_whitespace_normalizer_permits_cut_CUT_CHARS = (" ", "▁")if not cut or cut <= last_cut_state_cutclamps to the last space, so a bigger budget can pick the same cut: measured as three identical prefix tokenizations before the fallbackget is None→FalseFalsefor the same situationTests added — 20 checks, and the mutation each one exists to catch is in The checks, and that
they fail without the change below. Five of these were written only after a mutation showed the fix
could be deleted with the suite still green.
The waste
laya/agent.py:771(before this change):and
laya/common.py:192:idsat that point is[CLS]+ question text +[SEP]+ the option block, soroomis at mostmax_len - 3. Fortruncate_left=Falsethe slice is a prefix, and everything after indexroomis discarded. On the english checkpoint atmax_len=512a state at serve's 50 000-charactercap produces 10 001 tokens and keeps 512.
The constraint that shapes the fix: conversation lists
_encode_statesetstruncate_left = isinstance(state, list), andbuild_sequencethen takesstate_ids[len - room:]. That is deliberate — a chronological conversation serializes newest-last,so a prefix would silently drop the newest turn — and the comment above the tokenization says so.
A prefix shortcut is therefore only valid for
stranddictstates.So the branch is explicit, and the list path is byte-for-byte what it was:
Two checks hold that line.
test_agent_tokenizes_a_conversation_list_in_fullasserts the wholeserialized conversation reaches the tokenizer and that the assembled sequence still ends on the
newest turn's last token;
test_agent_sequences_are_unchanged_by_the_head_budgetcompares thefinished rows for list states against rows built from the full ids. Applying the head budget to
lists as well fails both — measured, by making that exact mutation:
predict_longis also left alone. It windows the whole document and reportstoken_endagainstthe full length, so it needs every token; the second full-state tokenization at
agent.py:1170isnot waste.
The multiplier is a starting point, and the probe is what makes it safe to be wrong
A token is at least one character, but a BPE token can be many, so the size of the first prefix
decides whether it reaches the budget. Measured over both shipped tokenizers' vocabularies — the
decoded character length of every token — rather than assumed:
Densest real text measured,
chars/tokenover whole documents, re-measured on the shippedtokenizers with varied prose:
An earlier version of this file quoted 6.74 and 16.00 for the first two rows. Neither reproduced;
the lower real densities only make a small first estimate safer.
No multiplier is a proof, so the constant is not load-bearing. For multilingual 32 would be one —
no token covers more than 31 characters — but for english it is not, because its whitespace-run
tokens reach 512: a state of 16 384 spaces is 32 tokens. Correctness therefore comes from checking
the result, never from the constant:
That makes an under-estimate safe, which means a conservative multiplier buys nothing and costs
throughput. It cost a great deal: at 32, a
max_len=1024preset did not attempt a cut until thestate exceeded
1024 * 32 = 32 768characters, andlaya.servecaps state at 50 000 — so no statebetween 8 000 and 32 000 characters ever took the fast path on the multilingual preset, most of its
reachable range.
But simply lowering the constant to 8 created a different regression, and a worse one: a state
denser than 8 chars/token would tokenize a large prefix, find it short, and tokenize the whole state
as well. Measured on a 256-character dash run per token, that was 1.64× the characters of the
state — a 0.57× slowdown against simply tokenizing it.
So the density is measured on a small probe before any full-size prefix is committed to:
The probe bounds the waste to its own size — a sixty-fourth of the state at most — and it also has
to be small, because it is pure overhead on the states that go on to succeed: shrinking it from a
thirty-second to a sixty-fourth moved English prose at 6 000 characters from 1.50× to 1.64×, and at
50 000 from 11.72× to 12.15×.
Two ordering decisions came out of measurement rather than taste:
_STATE_HEAD_MIN_RATIO = 1.25. Below that the prefix is nearly the whole state, so the headtokenizes almost the same text and the scan plus gates are pure overhead — measured at 0.94× on
english and 0.88× on multilingual just above the budget. At 1.25× the head is already ahead.
They cost 53–91 µs, which is free against a millisecond tokenization but 20% against a cheap one —
and a dense state tokenizes few ids from many characters, so its own tokenization is cheap. Asking
the gates first was a measured 0.82× on exactly those shapes.
Which makes the constant's status worth stating plainly, because it is a design property and not an
unverified claim: 8 chars/token is a measurement, not a bound, and nothing anywhere in the change
treats it as one. Correctness comes from
len(head) >= needand the retry; the constant only decideshow often the first guess is right. A third-party checkpoint denser than the retry can reach in three
attempts, or one with an internal-space added token, or one with no whitespace-splitting
pre-tokenizer, takes the full tokenization — still correct, just not faster.
The cut point was the hard part, and reasoning alone got it wrong
The first version of
_state_cutaccepted a space as a cut point unless whitespace sat on bothsides of it, reasoning from the ByteLevel regex:
\s+(?!\S)matches a whitespace run minus its lastcharacter, and that last character opens the following word, so both ends of a run start a
pre-token. The differential run against the real tokenizers rejected it — 14 mismatches out of 740,
smallest case
'\n b\xa0b'cut at index 2:The cause is not pre-tokenization at all. Both checkpoints carry added tokens that are runs of one
whitespace character — 2–24 spaces on english, 1–31 newlines or tabs on multilingual — and the
added-token trie matches them greedily before the normalizer or the pre-tokenizer runs. A run's ids
therefore depend on how long the run is, so a cut inside one rewrites the ids before it.
The rule that survives is narrower:
text[i]is a plain space andtext[i - 1]is notwhitespace. Three things make that safe, each checked against the shipped tokenizers:
text[i]whilestarting earlier would need
text[i - 1]to be whitespace. And neither checkpoint has an addedtoken holding a space anywhere but its first character (audited over all 116 and 249 added
tokens), so no non-whitespace added token can span the cut either.
ByteLevel only ever puts a space first in a piece (
" ?\p{L}+"," ?\p{N}+"," ?[^\s\p{L}\p{N}]+","\s+"). BPE merges never cross a piece.starter with combining class 0, so it never composes with what precedes it. The multilingual
normalizer only rewrites single spaces.
Non-space whitespace is deliberately not a cut point: on the multilingual tokenizer
"a\n b"pre-tokenizes as
("a" + newline),(U+2581 + "b"), so cutting at the newline changes the firstpiece.
The search is one
str.rfindin the ordinary case, stepping back over a whitespace run the budgetlanded inside, capped at 64 probes.
The cut rule is checked against the tokenizer, not assumed
_state_cut's safety rests on a property of the tokenizer's added tokens: every space-bearingone is a run of a single whitespace character, so a token cannot begin inside a run and cover the
cut. Both shipped checkpoints have it — 23 of english's 116 added tokens hold a space, and each is
2–24 spaces and nothing else.
Agentloads any directory a training run wrote, though, andagent.pyadvertises exactly that.A checkpoint that added a domain phrase breaks the property:
The trie matches
"New York"greedily across a space whose predecessor is a letter — a cut therule accepts — so the head stops being a prefix of the full ids. Measured on the shipped english
tokenizer with those two tokens added,
need1–59:That is the silent-truncation class this whole change exists to avoid, so
_added_tokens_permit_cuttests the property per call and falls back to the unconditional fulltokenization when it does not hold. It mirrors
_state_cut's own rule: reject any added tokenholding a space whose predecessor inside the token is not whitespace.
The same argument applies to pre-tokenization, which is the second of the three properties
_state_cutdocuments. ByteLevel's regex only ever puts a space first in a piece, and Metaspacewith
split=Truesplits on U+2581 — but those are facts about the shipped tokenizers, not abouttokenizers. A T5- or Llama-shaped one keeps Metaspace in the normalizer as
Prepend+Replaceand leaves
pre_tokenizernull, so BPE merges run over the whole string and the vocabulary holdspieces spanning what used to be a space. Built one that way (real
tokenizersBPE, merges grownleft to right over repeated text so a 40-character cross-space piece forms):
And the added-token gate has to know both spellings of a space. A sentencepiece-derived
checkpoint stores its added tokens against the normalized text, where a Metaspace-style normalizer
has already rewritten
" "to"▁"— so the phrase this gate exists to catch arrives as▁New▁Yorkand contains no U+0020 at all. A U+0020-only scan passed it:Verified on a real
tokenizersfixture (Replace(" ", "▁")normalizer,Metaspace(split=True)pre-tokenizer,
AddedToken("▁New▁York", normalized=True)) at phrase lengths of 2, 6 and 10 words.Worth being precise about the blast radius: in those fixtures the divergence landed past
room,so
build_sequencesliced it away and the model saw identical input — I could not make it corrupt asequence. But the documented contract was broken, and relying on
room's slack to hide that is nota guarantee, so it is fixed and pinned rather than argued away.
_pre_tokenizer_permits_cutrequires a pre-tokenizer that splits on whitespace, and reads theflags that actually decide it rather than the class name alone:
ByteLevel.use_regexandMetaspace.splitcan each be off, and with either off the guarantee is gone. ASequencepassesif any member splits. A tokenizer whose pre-tokenizer cannot be inspected — a slow, pure-Python one
has no
backend_tokenizer— is treated as unverifiable and gets the full tokenization, because"cannot establish it" is not the same as "safe".
Both shipped tokenizers still take the fast path with both gates in place (checked: english
ByteLevel
use_regex=True, multilingual Metaspacesplit=True). A tokenizer that raises onget_added_vocabis treated as unverifiable and gets the full tokenization; one without the methodhas no added tokens to enumerate.
Deliberately not cached: the scan costs 67.7 µs against 3.18 ms for the one tokenization it
guards (best of 5 × min of 200, warmed), and a cache keyed on the tokenizer would go stale the
moment a caller added a token to it.
Measured: tokenization, before and after
English checkpoint,
max_len=512(first budget 4 096 characters). Method: interleaved A/B — 5warm-up pairs discarded, then 25 alternating pairs, best of each. Interleaved because a sequential
all-before-then-all-after run drifted 25% on this laptop and reported a 24 ms case as 28.23 ms vs
35.26 ms on identical work.
Measured on the final code, so the density probe and all three tokenizer gates are inside every
"after" figure.
Measured: share of a real request
Same checkpoint, real weights, MPS.
beforeisencode_state_headreplaced by the fulltokenization at the call site — literally what
maindoes — in the same process on the sameloaded model, so the comparison is not across two runs. Interleaved, 2 warm-up pairs, best-of-9.
One question (
noul), the shape where tokenizing is the largest fraction of a request:Three questions, best-of-7 — the state is tokenized once but the forward pass triples, so the
share falls:
No path gets slower
A prefix optimization is easy to measure only on the case it improves. This is the sweep that looks
for the cases it hurts: shapes × sizes × both checkpoints, interleaved A/B, reporting the worst
ratio anywhere rather than the average. Sizes run from 500 characters to
laya.serve's50 000-character cap; shapes are English and Russian prose, CJK without spaces, flat JSON, dash runs
at 16/64/256 characters, all-spaces, tabs-only, base64, newline runs and emoji.
Three separate regressions were found this way and fixed, none of which the headline cases showed:
_STATE_HEAD_MIN_RATIO = 1.25Re-measured on the current base, three independent trials per shape, interleaved best-of-31, both
checkpoints. An earlier revision of this description said no shape at any size is slower and put
the worst case at 0.98×. That was too strong, and the honest numbers are:
So: flat within noise on every incompressible shape, except english dash-runs, which sit
consistently at 0.96–0.98×. That is the irreducible cost of the probe on a state that then turns
out not to be worth cutting — the density cannot be known without measuring it, and the probe is
bounded to a sixty-fourth of the state precisely to bound this. It is a ~3% cost on a synthetic shape
against 12× on English prose at the same size, and it is reported rather than omitted.
On the measurement itself. These were interleaved A/B — alternating within one loop, best of 31
pairs — because a sequential "all before, then all after" run drifts on this machine and invented
regressions of 0.89× that did not exist. Every flagged case was re-run three independent times before
being treated as real: of eleven flagged in one sweep, ten were noise and one was genuine. The sweep
also records whether the new code path actually executed for each measurement, because several
apparent regressions were at sizes where it cannot run.
Parity: differential, against the real tokenizers
Three tokenizers — both shipped ones, plus english with
add_tokens(["New York", "order id"])—over 438 state shapes × 5 budgets (
need= 1, 8, 64, 512, 1024). No model is loaded; thisis about the ids, not inference.
and must cover the budget or be the whole state
build_sequencerows (ids and marker positions)from the head ids vs from the full ids, for a
choice, ascoreand anoulquestionAnd the same run with
_added_tokens_permit_cutforced toTrue, so the zero above is aresult rather than an absence of checking:
One, and only on the tokenizer carrying an internal-space added token. It was 5 before the retry
landed; a smaller first budget happens to avoid the bad cut for four of the five budgets, which is
luck rather than safety — the gate is what makes it 0. Two of the 436 shapes exist solely to make
this control possible, because they contain the added phrase. Without them the control reported
0 differences too, which would have looked like confirmation and meant only that the corpus
never reached the guard.
Shapes: empty string, one word, short prose, exactly at / one character under / one character over
the budget, prose past the budget, serve's 50 000-character cap, no space at all, base64, space
runs, newline runs, space-then-newline, newline-then-space, tabs only, indented JSON, leading
whitespace, all whitespace, CJK, Russian, Arabic, emoji, combining marks, an NFC trap
(
"a" + U+0301 × 3, repeated), NBSP-separated, ZWSP-separated, a literal[MASK], the twoadded-phrase shapes, dict, dict past the budget, nested dict with a 20 000-element list, unicode
keys, a conversation list, a 6 000-turn list, a 20 002-element list of strings — plus 400
randomized cases over an adversarial alphabet biased toward whitespace runs, at lengths 1, 2, 7,
63, 511, budget−1, budget, budget+1 and 40 000.
The 9 list rows per tokenizer are exempt from the sequence comparison rather than passing it: a
list is left-truncated, so production tokenizes it in full and never builds a head for it. Feeding
one a prefix-truncated head would only re-discover the constraint this change is built around. The
harness had that wrong at first and reported 6 differences per shipped tokenizer, all on the two
oversized list shapes — worth stating, because it is the invariant the whole change rests on.
It needs local checkpoints, so it is not a CI suite; what ships in CI is below.
Real-model end-to-end
Agenton the shipped english checkpoint (ModernBERT-large, 421 M params,max_len=512,head_max_len=192),HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1, three questions(
choice/score/noul), 11 requests covering all the shapes in the latency table above.Compared field by field — every answer, every probability to 6 decimal places, and
usage— notjust the picked labels.
And the same corpus run against a deliberately broken variant, so the zero above is a result
rather than an absence of checking. Three trees, same harness, same checkpoint:
upstream/main,this branch, and this branch with the
len(head) >= needguard deleted.dense_dash256dense_dash256_dictdense_dash64prose_short,prose_past_budget,dict,no_space_at_all,turns,turns_past_budgetThe third row is reported rather than omitted because it is a zero that proves nothing on its own:
the shipped english tokenizer has no internal-space added token, so bypassing
_added_tokens_permit_cutcannot change anything on this checkpoint. Its evidence is thereal-tokenizer measurement in the section above (14 of 59
needvalues non-prefix without thecheck, 0 with it) — and, since an earlier revision of this description, inference too: resizing the
embeddings after
add_tokensmakes the model valid, and the 398-pair search below reproduces 125non-prefix heads. None of them moves an answer, for a structural reason worth reading rather than
assuming; see The three gaps, closed.
The same corpus, the same three questions, against the real multilingual checkpoint — mmBERT-base,
321 908 995 parameters,
max_len 1024 / head_max_len 256:Both shipped checkpoints are therefore observed answering, not just tokenizing.
The six controls staying identical is the point: the divergence is the deleted guard, not run
noise. A state reaches that guard only by being both dense (the character budget buys fewer
tokens than
need) and cuttable (_state_cutreturns non-zero); dash runs separated by singlespaces are both, because every space is preceded by
-, which is not whitespace. 256 is alsoexactly the longest all-dash token surface in the english vocabulary, so the density is a property
of the real tokenizer and not an artefact of the fake one.
usage: what the two token counts now meanusage["input_tokens"]is unaffected, and that was checked rather than assumed. It is written inexactly two places and both take it from the attention mask:
That is the padded sequence length, summed over the state's question rows and capped by
max_len— not the state's token total. Measured on the real checkpoint: a 50 000-character statewith three questions reports
input_tokens: 1536(3 × 512) and a 79-character one reports149.grep -n 'len(state_ids)' laya/finds no other consumer of the total beyondbuild_sequence's ownslice and
predict_long's window scan.What does change is the two counts, and the rest of this section is about them.
The two counts are opt-in
The problem, in your own measurement. You integrated the branch and diffed against
mainon thereal checkpoints over a 4 000-character state at
max_len=256:mainstate_tokensstate_tokens_droppedstate_tokensstate_tokens_droppedThe counts move because the prefix is all that was tokenized, so
len(state_ids)is the prefixlength. They were not wrong — they were floors. And as you put it, a count that is exact on one path
and a floor on another is the outcome most worth avoiding.
What is implemented.
state_token_counts, defaultFalse:truncatedandtruncated_questionsare present and exact.Reported by the same mechanism
usage["options"]uses (#569): a conditional assignment after theusageliteral, in the same function.Why
truncatedcan stay free, and why that is not a compromise.encode_state_headonly everreturns early with at least
need=max_lenids; every other exit returns the full tokenization.And
room = max(0, max_len - len(ids) - 1)where the question head is at least[CLS] … [SEP] [SEP],so
room <= max_len - 4 < max_len. On the head path, therefore,len(state_ids) >= max_len > roomalways holds: the clamp really did drop tokens, for every question, so
truncatedisTrueandgenuinely true. Your own diff found
truncated/truncated_questionsidentical, and this is why.Asking costs what it should, and nothing more. Asking takes the same
encode_textcallupstream/maintakes, so an asking call is upstream's speed, not worse. The opt-in's own overhead —one boolean test and one dict construction — measures 24 ns per call, against a tokenization in
milliseconds.
Verified against upstream, both checkpoints, 9 state shapes (prose at three sizes, dict, CJK,
dense dash runs, no-space, and two conversation lists):
The third line is the one that matters most: when a caller asks, the response is not merely similar
to
main's — it ismain's, key for key and token for token.Surface.
state_token_countsis accepted byAgent.system_one/predict_batch/predict_long,the same three on
ONNXAgent, all threeRouterentry points, both HTTP routes (added toBODY_CONTROLSandBATCH_BODY_CALL_CONTROLS, boolean-only,422otherwise), the CLI as--state-token-counts, andexamples/server.py.It is passed only when it is on, at every hop. That is not tidiness: three separate contract tests
in this repo caught a caller broken by passing it unconditionally — a fake
_encode_stateintests/test_batch.pyandtests/test_hooks.py, and the forwarded-kwarg settests/test_predict_long.pypins. So the default call shape is byte-identical to what it was, and asubclass, an attached agent, or a replacement
predict_batchthat predates the argument keeps workingon every call that does not ask.
What this breaks, and what it does not
One thing breaks: a caller reading
usage["state_tokens"]orusage["state_tokens_dropped"]getsnothing back unless it asks. Two of this repo's own examples did exactly that and would have
raised
KeyError:examples/15_truncation_basics.py— prints both counts from a plainpredict()examples/37_long_document.py— same, in two placesBoth now pass
state_token_counts=Trueand say why, which suits them: they are the pages that teachthe truncation report.
docs/http-api.md,docs/structured.mdandexamples/03_reading_the_result.pydescribe the keys as opt-in and name the field that asks.Nothing else moves.
truncated,truncated_questions,input_tokens,options, every answer, everyprobability, and the
routingblock are untouched — measured above, not argued.The opt-in's own checks, and the mutation table
tests/test_state_token_counts.py(new, 295 lines, 32 checks, weight-free, registered inscripts/test_suites.py) pins the behaviour rather than the wording:laya.agent, noton
laya.common, becauseagent.pydoesfrom .common import encode_state_head, encode_textandpatching the defining module would leave the test measuring nothing
usagekey set in both modes, and that asking'sstate_tokensequals a full tokenizationsystem_one,predict_long, bothONNXAgentpaths and all threeRouterentry pointsforward it, and that none of them forwards it when it is off
A stub tokenizer cannot reach the prefix path at all — all three of
_state_cut's soundness gatesinspect tokenizer internals a stub does not have, so a stub always takes the full path. That is why
this suite pins which call runs and leaves numerical exactness to the real-checkpoint runs above,
and it says so in its docstring rather than implying coverage it does not have.
Mutation table: 15 of 15 killed.
test_truncation,test_state_token_countstest_state_token_countstest_state_token_counts_encode_state(torch)test_state_token_countspredict_longstops forwarding it (torch)test_state_token_countstest_onnx_truncation_paritytest_onnx_truncation_parity_encode_state(ONNX)test_onnx_truncation_paritypredict_longstops forwarding it (ONNX)test_state_token_countsRouter.predictdrops the overridetest_state_token_countsRouter.predict_batchdrops the kwargtest_state_token_counts_ScanLongdrops it forpredict_longtest_state_token_countstest_serveBODY_CONTROLStest_servetest_cliWorth recording because it is the reason this took a second pass: 7 of the first 11 mutants
survived. The feature worked and every suite was green, and the opt-in could have been deleted in
either direction without a single test noticing. The suite above is what closed that, and the table
was re-run after the rebase.
The checks, and that they fail without the change
Added to
tests/test_truncation_direction.py, which theci.ymlpytest lane already runs(
Pytest suites (serve, batch, system-one, audit regressions, truncation, compile)), so there is noCI wiring to add.
The fake tokenizer models what matters: ids are a function of a whole piece, and every maximal
whitespace run gets an id of its own, which is the added-token behaviour that broke the first cut
rule. So a wrong cut point fails in CI without a checkpoint.
Revert check. Source copied aside,
laya/agent.py,laya/common.pyandlaya/onnx_agent.pyreplaced with
upstream/main's content in the working tree, the tests kept, suite re-run, then thesource copied back and confirmed byte-identical by checksum:
37 checks go red, and they are the ones that assert the new behaviour: the prefix path, the three
cut gates, the retry and its budget, and the two paths each agent takes. The opt-in suite does not
merely fail — it cannot run, because the argument it is about does not exist on the reverted source.
Two checks pass both ways on purpose — a conversation list stays tokenized in full, and
input_tokensis the sequence length rather than the state total. They pin behaviour this change mustnot move.
Mutation checks, each made and reverted, to show the assertions bite rather than merely run.
Every row below was re-executed on 2026-09-29 — applied to the source, suite run, source restored —
rather than carried over from when it was first written. That audit is what found the two unpinned
short-circuits above. Of the 24 rows that predate it, 22 behaved exactly as recorded and two named
the wrong test, both corrected here; one of those two,
the retry budget never grows, was greenagainst the old suite as well and is closed by the same new check. A mutation table is only worth the
last time someone ran it.
test_state_head_ids_match_a_full_tokenizationtest_state_head_ids_match_a_full_tokenization..._sequences_are_unchanged...,..._list_in_fullSTATE_HEAD_CHARS_PER_TOKEN = 1..._leaves_a_state_barely_over_the_budget_alone— the retry recovers the ids, but a state barely over the budget gets tokenized whole_STATE_HEAD_MAX_ATTEMPTS = 1(no retry)..._retries_from_the_density_it_measured..._guard_rejects_a_short_head_when_density_rises_past_the_probelen(head) >= needguard deleted..._guard_rejects_a_short_head_when_density_rises_past_the_probelen(probe) >= needshort-circuit deleted..._stops_at_the_probe_when_the_probe_already_covers_the_budget_added_tokens_permit_cutbypassed..._refuses_to_cut_when_an_added_token_holds_an_internal_space..._refuses_to_cut_when_an_added_token_holds_an_internal_space_pre_tokenizer_permits_cutbypassed..._refuses_to_cut_when_nothing_splits_on_whitespace_normalizer_permits_cutbypassed..._refuses_to_cut_when_the_normalizer_drops_the_cut..._refuses_to_cut_when_the_normalizer_drops_the_cut..._refuses_to_cut_when_the_normalizer_drops_the_cut_STATE_HEAD_MIN_RATIO = 1.0(no break-even margin)..._leaves_a_state_barely_over_the_budget_alone..._does_not_pay_for_a_prefix_it_will_not_use..._stops_before_the_end_of_a_large_state,..._does_not_pay_for_a_prefix......_onnx_agent_takes_the_same_two_paths_as_the_torch_agent..._onnx_agent_takes_the_same_two_paths_as_the_torch_agent..._onnx_agent_takes_the_same_two_paths_as_the_torch_agent_CUT_CHARS(either branch)..._added_token_gate_fails_closed_and_covers_both_spellings,..._refuses_to_cut_when_the_normalizer_drops_the_cut..._added_token_gate_fails_closed_and_covers_both_spellings..._state_head_retry_does_not_repeat_an_identical_prefixMetaspace.splitignored..._splits_on_whitespace_reads_the_flags_that_actually_decideByteLevel.use_regexignored..._splits_on_whitespace_reads_the_flags_that_actually_decide..._refuses_to_cut_when_nothing_splits_on_whitespacetest_state_head_ids_match_a_full_tokenizationitself makes 160 comparisons (2 fake tokenizers× 4 budgets × 20 state shapes) and
test_agent_sequences_are_unchanged_by_the_head_budgetmakes60 (3 questions × 20 shapes), both asserted counts so a shape list that silently shrinks fails.
Every line of the fix has a mutation that turns a test red — 14 of them. That is not incidental:
five times on this change a fix shipped whose removal left the whole suite green (the
len(head) >= needguard, the retry,Metaspace.split, the normalizer prefix check, and the gateordering). Each is now pinned, the gate ordering by asserting the gates are not consulted for a
state that was never going to be cut.
One of those five stayed unpinned for longer than this description admitted, and it took a
re-audit to notice. An earlier revision credited the
len(head) >= needguard to thedense-but-cuttable shape. That is wrong, and provably so: a dense-and-cuttable state is caught by
the worth-it projection and returns a full tokenization before the retry loop is ever entered —
the 82 240-character dash state exits there, which is why it produces the full 321 ids. So deleting
the guard left all 22 checks green, exactly the failure mode this suite exists to prevent.
The guard is reachable only when the text past the probe is denser than the text the probe read,
by more than the 1.25× margin, because then
wantis extrapolated from a density that no longerholds.
..._guard_rejects_a_short_head_when_density_rises_past_the_probeis that shape: 512 sparsecharacters (4 chars/token) then 82 000 dense ones (20.5), so the probe measures 4 and the retry has
to climb to ~17.
Deleting the guard returns 229 ids where 512 were asked for. The same audit found
if len(probe) >= need: return probeunpinned as well — a throughput line rather than a correctnessone, since falling through pays a second tokenization for a correct but needlessly large head, which
no ids-only assertion can see. It is now asserted by counting tokenizations. Both new checks are
verified to go red on deleting their own line and to stay green when
_STATE_HEAD_MAX_ATTEMPTSchanges, so neither is pinned to that constant.
Two of the 20 shapes exist for guards rather than for coverage. Dense-but-cuttable is the only
shape that reaches the worth-it projection with a cut available, so it is what keeps that branch
honest. A state carrying the mask token makes the
.replace(mask_token, " ")that_encode_stateapplies something the parity test actually exercises, rather than a no-op on every shape, so the
test tokenizes the text production tokenizes instead of only coincidentally matching it.
Gates
The canonical suite list now lives in
scripts/test_suites.py, which bothci.ymland the releasegate invoke, so
tests/test_state_token_counts.pyis registered there rather than in a workflow.Everything in that list passes except one suite, which fails on
mainas well:tests/test_shortlist_cosine.py— 3 failures, 1 error, identically onupstream/main@fa9a2a7and on this branch, and this PR does not touch
laya/shortlist.pyor that test. Flagging it ratherthan leaving you to find it in this PR's output.
CI on the pushed head: 24 checks pass, 1 skipped (
docs (deploy), which does not run on a PR) —including
testson py3.10/3.11/3.12/3.13 and Windows,http,onnx export,evals,lint,package,CodeQL, bothsmokejobs and the TypeScript SDK lanes.The three gaps, closed
An earlier revision of this description listed three things the evidence did not reach: multilingual
never observed answering, no ONNX run through a real fused graph, and the added-token guard checked
through the tokenizer rather than through inference. All three are now measured.
1. Multilingual, through real inference. The e2e harness is parameterised by checkpoint, and the
same 9-shape corpus ran against the real multilingual checkpoint — mmBERT-base, 321 908 995
parameters,
max_len 1024 / head_max_len 256from its ownrl_agent_config.json:The second line matters as much as the first: a zero-difference run on a corpus that never reached
the new code would prove nothing. On the dense-and-cuttable shapes the head really is a fraction of
the state, and it really is a prefix of it.
2. ONNX, end to end at real weights. Two earlier revisions of this description got this wrong, in
opposite directions, and the correction is the useful part. The first said no fused graph was
available —
ONNXAgentcannot consume the two-graphencoder.onnx+head.onnxpair, and I recorded"no local artefact" without checking. The second built a 209 KiB fused graph from a tiny encoder +
head and called the gap closed, noting that real weights "are not the point". They are the point, and
they were available the whole time: the rig holds three real fused exports of the 322M multilingual
checkpoint, each a 1.2 GB external-data pair. So the run below is the real one.
ONNXAgenton the real fused graph (1.20 GB,CPUExecutionProvider, batch axisbatch_size) withthe real multilingual tokenizer,
max_len 1024 / head_max_len 256, the same 9 shapes the torchharness uses, branch against
upstream/main:dense_dash256technicaldense_dash256_dicttechnicaldense_dash64technicalprose_past_budgetbillingno_space_at_alltechnicaldictbillingprose_shortbillingturnsbillingturns_past_budgetbillingFour shapes reach the optimization and take roughly a fifth of the ids they used to; the answers do
not move. The list shapes keep the full tokenization, which is the constraint this whole change is
built around. Harness:
e2e-onnx-state-head.py, parameterised by tree and by graph.One thing worth passing on from the tiny-graph run that preceded this: instrumenting it needs the spy
on both
common.encode_textandonnx_agent.encode_text— the prefix path routes through thefirst and
truncate_leftthrough the second, so patching one reports an empty trace for three of fourrows — and it has to filter to calls on the state text, or the question tokenizations land in the
count. Both mistakes were made before they were caught.
3. The added-token guard, through inference. The earlier reasoning was that a checkpoint adding
tokens also resizes its embeddings, so an added id on this model would be out of range and any
answer comparison would be measuring a broken model. That is avoidable —
agent.model.encoder.resize_token_embeddings(len(agent.tok))afteradd_tokens(["New York", "order id"])makes the model valid — so the comparison was run:So the guard protects an invariant that is real but, on the shapes reachable here, not
answer-visible — and there is a structural reason, which is the useful part. A head is only returned
when
len(head) >= need = max_len; the divergence sits at the cut, i.e. in the head's tail; andbuild_sequencekeeps onlyroom < max_lenids. The corruption is always past the slice. The guardstays because
roomis a function of question size and nothing pins it belowmax_len— a callerwith tiny questions moves
roomup towardmax_len, and the margin that is absorbing this today isnot guaranteed by anything.
Integration with the other PRs from this fork
#689 landed in 0.3.22, as you noted, and this branch is rebased past it with no conflict. The
interaction it posed is worth recording because it was the one real risk between the two: #689 caps
each
predict_longwindow at the room the questions leave, andencode_state_headthen takes aprefix of that window, so the concern was the second re-truncating what the first had just sized. It
does not — a window is at most
roomtokens androom < max_len, so the head path cannot cut oneshort. Window counts and widths were identical with both changes present, and document coverage at
100 options stayed at 0% unread.
Still open from this fork: #688 (ONNX batch-dynamic export) and #690 (router pinning
controls). Neither shares a file with this PR.
This rebase crossed 130 upstream commits and four files conflicted. Each was resolved in
upstream's favour on the facts, which is worth stating explicitly since three of the four were your
own corrections landing against text this branch had edited:
.github/workflows/ci.ymlscripts/test_suites.py, so the new suite is registered therelaya/agent.pypredict_longwindow is cut, and theaskedargument from the in-place-rewrite fixtests/test_serve.pyexamples/03_reading_the_result.pyOne upstream test needed a change, and it is the only place this PR touches logic it did not add.
tests/test_truncation.pyidentified "the truncation report" by subtracting the two token totals fromthe
usageliteral. That stops picking out the report once any key in that block is conditional — andoptionswas already a counter-example waiting, since #569 publishes it the same way. The four keysare now named explicitly and then checked to be published, which is the half of the old check that had
teeth.