Skip to content

perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in - #687

Open
keemsisi wants to merge 7 commits into
NandhaKishorM:mainfrom
keemsisi:perf/state-prefix-tokenization
Open

keemsisi wants to merge 7 commits into
NandhaKishorM:mainfrom
keemsisi:perf/state-prefix-tokenization

Conversation

@keemsisi

@keemsisi keemsisi commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

perf(agent): tokenize only as much state as a prefix-truncated sequence can hold, and make the two state-token counts opt-in

laya/common.py +367 · laya/agent.py +76/-26 · laya/onnx_agent.py +45/-20 ·
laya/router.py +23/-4 · laya/serve.py +38/-2 · laya/cli.py +20/-2 ·
tests/test_truncation_direction.py +1207 · tests/test_state_token_counts.py +295 (new) ·
docs, examples and scripts/test_suites.py
Branch perf/state-prefix-tokenization, 6 commits on upstream/main @ fa9a2a7. 20 files,
+2301/-116.

In short

The problem. Agent._encode_state tokenizes the entire serialized state. build_sequence then
keeps state_ids[:room], and room is always below max_len. For every state shape except a
conversation list, the tokens past that slice were produced only to be thrown away.

The fix. Measure the state's density on a small probe (at most 512 characters, a sixty-fourth of
the state), then slice the serialized text to the prefix that density says will fill the budget, at a
cut point that cannot change how the tokenizer reads what comes before it, and tokenize only that. If
it still comes up short, retry from the density that attempt measured — at most three attempts — then
tokenize the whole state as before: correct, just not faster. The prefix is taken only when it is
worth taking, and only for a tokenizer whose added tokens, pre-tokenizer and normalizer are each
checked to permit the cut.

The consequence, and the half of this PR you asked for. Reading only a prefix means
usage["state_tokens"] and usage["state_tokens_dropped"] can no longer be counts of the whole
state — they would be floors. So they are now opt-in: absent unless the caller passes
state_token_counts=True, and asking restores the full tokenization for that call so what comes back
is exact. truncated and truncated_questions stay always-on and free, and that is sound rather
than convenient — see The two counts are opt-in below. This follows usage["options"] from #569,
by the same mechanism in the same function.

Where it matters, and where it does not. Over HTTP laya.serve caps a state at 50 000
characters, so the win there is ~9 ms of tokenizing (10.22 ms → 0.95 ms, 10.8×), which is
8.7% of a one-question request on the english checkpoint and 3.0% of a three-question one. A
trivial request is unchanged. The large multiples need a direct library call with a state bigger
than the HTTP cap: a 25 000-field dict serializes to 500 000 characters and goes from 214 ms of
tokenizing to 0.84 ms (254×), which is 64.9% of that request. States with no space in reach —
CJK without spaces, base64 — keep the old cost, within noise; see No path gets slower below for the
one shape where the probe is a measurable cost.

What does not change. Any answer. 10 485 differential comparisons against both real checkpoints'
tokenizers, 0 differences; 9 real-model request shapes on each of english and multilingual, 0
answer differences
against upstream/main with the counts asked for and without; and ONNXAgent
answering through a real fused ONNX graph in onnxruntime.

What does change for a caller. Those two usage keys are absent unless asked for. That is a
break, it is deliberate, and it is the one you chose over a count that is exact on one path and a
floor on another. Two of this repo's own examples read them and are updated here — see What this
breaks, and what it does not
.

In one line: the sequence only ever held a prefix of the state, so now we only tokenize a prefix —
and the two counts that describe the whole state are asked for rather than guessed at.

Every line added and removed, and why

Removed — 12 lines, all of it replaced rather than dropped.

Removed Where Why, and what preserves the behaviour
the 5-line encode_text(self.tok, serialize_state(state).replace(...), add_special_tokens=False)["input_ids"] call agent.py, onnx_agent.py (once each) Split into text = ... plus a two-path choice. The identical call is retained verbatim for the truncate_left branch, so conversation lists tokenize exactly as before.
2 comment lines ("Tokenize in full and let build_sequence slice per question", "the PyTorch Agent already does this via state_ids") same two sites They described the single-path behaviour that no longer exists; replaced by comments stating why a list is exempt.

No function, parameter, test or public name is deleted. This change adds no new tokenizer entry
point — both call sites it touches route through encode_text, as they did — and build_sequence's
contract is untouched. (encode_text is not the only entry point on main: Agent.predict_long
calls self.tok(...) directly at agent.py:1177. That line is untouched here, and ONNXAgent's
equivalent already goes through encode_text.)

Added — what each piece is for. Nothing here is defensive decoration; each item exists because
something measurable went wrong without it.

Added Why it exists
encode_state_head the change itself: tokenize only the prefix the sequence can hold
_state_cut choosing a slice point that cannot alter the ids before it
STATE_HEAD_CHARS_PER_TOKEN = 8 first density estimate, near real text rather than above all of it
_STATE_HEAD_PROBE_CHARS / _STATE_HEAD_PROBE_MIN bound the cost of guessing wrong: without the probe, a dense state cost 1.64× the characters of the state (0.57× slower)
_STATE_HEAD_MIN_RATIO = 1.25 below it the prefix is nearly the whole state: measured 0.94×/0.88×
_STATE_HEAD_MAX_ATTEMPTS = 3 bound the retry; prose and JSON land on the first attempt
_added_tokens_permit_cut an added token with an internal space made 14 of 59 budgets return non-prefix ids
_pre_tokenizer_permits_cut + _splits_on_whitespace a T5/Llama-shaped tokenizer diverged at token 0
_normalizer_permits_cut a whitespace-stripping normalizer diverged at token 0, past both other gates
_CUT_CHARS = (" ", "▁") a sentencepiece added token spells the space U+2581, so a U+0020-only scan passed it and the head stopped being a prefix
if not cut or cut <= last_cut _state_cut clamps to the last space, so a bigger budget can pick the same cut: measured as three identical prefix tokenizations before the fallback
get is None → False the gate answered "cannot establish this" as "it holds", while the other two answered False for the same situation
gates asked after the worth-it decision asking them first cost 0.82× on states that were never going to be cut

Tests added — 20 checks, and the mutation each one exists to catch is in The checks, and that
they fail without the change
below. Five of these were written only after a mutation showed the fix
could be deleted with the suite still green.

The waste

laya/agent.py:771 (before this change):

state_ids = encode_text(self.tok, serialize_state(state).replace(self.tok.mask_token, " "),
                        add_special_tokens=False)["input_ids"]

and laya/common.py:192:

room = max(0, max_len - len(ids) - 1)
st = state_ids[max(0, len(state_ids) - room):] if truncate_left else state_ids[:room]

ids at that point is [CLS] + question text + [SEP] + the option block, so room is at most
max_len - 3. For truncate_left=False the slice is a prefix, and everything after index
room is discarded. On the english checkpoint at max_len=512 a state at serve's 50 000-character
cap produces 10 001 tokens and keeps 512.

The constraint that shapes the fix: conversation lists

_encode_state sets truncate_left = isinstance(state, list), and build_sequence then takes
state_ids[len - room:]. That is deliberate — a chronological conversation serializes newest-last,
so a prefix would silently drop the newest turn — and the comment above the tokenization says so.
A prefix shortcut is therefore only valid for str and dict states.

So the branch is explicit, and the list path is byte-for-byte what it was:

text = serialize_state(state).replace(self.tok.mask_token, " ")
if truncate_left:
    state_ids = encode_text(self.tok, text, add_special_tokens=False)["input_ids"]
else:
    state_ids = encode_state_head(self.tok, text, max_len)

Two checks hold that line. test_agent_tokenizes_a_conversation_list_in_full asserts the whole
serialized conversation reaches the tokenizer and that the assembled sequence still ends on the
newest turn's last token; test_agent_sequences_are_unchanged_by_the_head_budget compares the
finished rows for list states against rows built from the full ids. Applying the head budget to
lists as well fails both — measured, by making that exact mutation:

mutation: state_ids = encode_state_head(...) for lists too
  FAILED test_agent_sequences_are_unchanged_by_the_head_budget
  FAILED test_agent_tokenizes_a_conversation_list_in_full
  2 failed, 10 passed

predict_long is also left alone. It windows the whole document and reports token_end against
the full length, so it needs every token; the second full-state tokenization at agent.py:1170 is
not waste.

The multiplier is a starting point, and the probe is what makes it safe to be wrong

A token is at least one character, but a BPE token can be many, so the size of the first prefix
decides whether it reaches the budget. Measured over both shipped tokenizers' vocabularies — the
decoded character length of every token — rather than assumed:

tokenizer vocab longest token longest bearing a letter or digit
multilingual (Metaspace + BPE, mmBERT) 256 000 31 chars (a run of spaces, newlines or tabs) 16
english (ByteLevel + BPE, ModernBERT) 50 368 512 chars (a run of 512 spaces) 35

Densest real text measured, chars/token over whole documents, re-measured on the shipped
tokenizers with varied prose:

content english multilingual
English prose 5.34 5.34
Russian prose 1.98 3.80
flat JSON of short fields 1.5–2.2 1.5–2.2

An earlier version of this file quoted 6.74 and 16.00 for the first two rows. Neither reproduced;
the lower real densities only make a small first estimate safer.

No multiplier is a proof, so the constant is not load-bearing. For multilingual 32 would be one —
no token covers more than 31 characters — but for english it is not, because its whitespace-run
tokens reach 512: a state of 16 384 spaces is 32 tokens. Correctness therefore comes from checking
the result, never from the constant:

head = encode_text(tok, text[:cut], add_special_tokens=False)["input_ids"]
if len(head) >= need:
    return head

That makes an under-estimate safe, which means a conservative multiplier buys nothing and costs
throughput. It cost a great deal: at 32, a max_len=1024 preset did not attempt a cut until the
state exceeded 1024 * 32 = 32 768 characters, and laya.serve caps state at 50 000 — so no state
between 8 000 and 32 000 characters ever took the fast path on the multilingual preset
, most of its
reachable range.

But simply lowering the constant to 8 created a different regression, and a worse one: a state
denser than 8 chars/token would tokenize a large prefix, find it short, and tokenize the whole state
as well. Measured on a 256-character dash run per token, that was 1.64× the characters of the
state
— a 0.57× slowdown against simply tokenizing it.

So the density is measured on a small probe before any full-size prefix is committed to:

probe_budget = max(64, min(512, budget // 8, len(text) // 64))
cut = _state_cut(text, probe_budget)
probe = encode_text(tok, text[:cut], ...)["input_ids"]
density = cut / len(probe)
want = need * density * 1.25 + 64          # 1.25x: density is not uniform across a document
if len(probe) < need and want * _STATE_HEAD_MIN_RATIO >= len(text):
    return full()                          # a prefix that big is not worth taking

The probe bounds the waste to its own size — a sixty-fourth of the state at most — and it also has
to be small, because it is pure overhead on the states that go on to succeed: shrinking it from a
thirty-second to a sixty-fourth moved English prose at 6 000 characters from 1.50× to 1.64×, and at
50 000 from 11.72× to 12.15×.

Two ordering decisions came out of measurement rather than taste:

  • _STATE_HEAD_MIN_RATIO = 1.25. Below that the prefix is nearly the whole state, so the head
    tokenizes almost the same text and the scan plus gates are pure overhead — measured at 0.94× on
    english and 0.88× on multilingual just above the budget. At 1.25× the head is already ahead.
  • The three soundness gates are asked last, after the probe and after the worth-it decision.
    They cost 53–91 µs, which is free against a millisecond tokenization but 20% against a cheap one —
    and a dense state tokenizes few ids from many characters, so its own tokenization is cheap. Asking
    the gates first was a measured 0.82× on exactly those shapes.

Which makes the constant's status worth stating plainly, because it is a design property and not an
unverified claim: 8 chars/token is a measurement, not a bound, and nothing anywhere in the change
treats it as one. Correctness comes from len(head) >= need and the retry; the constant only decides
how often the first guess is right. A third-party checkpoint denser than the retry can reach in three
attempts, or one with an internal-space added token, or one with no whitespace-splitting
pre-tokenizer, takes the full tokenization — still correct, just not faster.

The cut point was the hard part, and reasoning alone got it wrong

The first version of _state_cut accepted a space as a cut point unless whitespace sat on both
sides of it, reasoning from the ByteLevel regex: \s+(?!\S) matches a whitespace run minus its last
character, and that last character opens the following word, so both ends of a run start a
pre-token. The differential run against the real tokenizers rejected it — 14 mismatches out of 740,
smallest case '\n b\xa0b' cut at index 2:

prefix ids: ['Ċ', 'Ġ']            # cut after "\n "
full   ids: ['Ċ', '  ', 'b', …]   # "  " is one token, not two

The cause is not pre-tokenization at all. Both checkpoints carry added tokens that are runs of one
whitespace character
— 2–24 spaces on english, 1–31 newlines or tabs on multilingual — and the
added-token trie matches them greedily before the normalizer or the pre-tokenizer runs. A run's ids
therefore depend on how long the run is, so a cut inside one rewrites the ids before it.

The rule that survives is narrower: text[i] is a plain space and text[i - 1] is not
whitespace.
Three things make that safe, each checked against the shipped tokenizers:

  • No added token can straddle it. A whitespace-run added token that covered text[i] while
    starting earlier would need text[i - 1] to be whitespace. And neither checkpoint has an added
    token holding a space anywhere but its first character (audited over all 116 and 249 added
    tokens), so no non-whitespace added token can span the cut either.
  • No pre-token can straddle it. Metaspace rewrites each space to U+2581 and splits on it.
    ByteLevel only ever puts a space first in a piece (" ?\p{L}+", " ?\p{N}+",
    " ?[^\s\p{L}\p{N}]+", "\s+"). BPE merges never cross a piece.
  • Normalization cannot reach across it. The english tokenizer normalizes NFC; a space is a
    starter with combining class 0, so it never composes with what precedes it. The multilingual
    normalizer only rewrites single spaces.

Non-space whitespace is deliberately not a cut point: on the multilingual tokenizer "a\n b"
pre-tokenizes as ("a" + newline), (U+2581 + "b"), so cutting at the newline changes the first
piece.

The search is one str.rfind in the ordinary case, stepping back over a whitespace run the budget
landed inside, capped at 64 probes.

The cut rule is checked against the tokenizer, not assumed

_state_cut's safety rests on a property of the tokenizer's added tokens: every space-bearing
one is a run of a single whitespace character, so a token cannot begin inside a run and cover the
cut. Both shipped checkpoints have it — 23 of english's 116 added tokens hold a space, and each is
2–24 spaces and nothing else.

Agent loads any directory a training run wrote, though, and agent.py advertises exactly that.
A checkpoint that added a domain phrase breaks the property:

tok.add_tokens(["New York", "order id"])
text = "shipped to New York and billed to New York " * 200

The trie matches "New York" greedily across a space whose predecessor is a letter — a cut the
rule accepts — so the head stops being a prefix of the full ids. Measured on the shipped english
tokenizer with those two tokens added, need 1–59:

without the check:  14 of 59 need values return ids that are NOT a prefix
                    e.g. need=8, cut=252 lands inside ' New York ', first divergence at token 53
                    len(head)=54 >= need=8, so the `len(head) >= need` guard cannot see it
with the check   :   0 of 59

That is the silent-truncation class this whole change exists to avoid, so
_added_tokens_permit_cut tests the property per call and falls back to the unconditional full
tokenization when it does not hold. It mirrors _state_cut's own rule: reject any added token
holding a space whose predecessor inside the token is not whitespace.

The same argument applies to pre-tokenization, which is the second of the three properties
_state_cut documents.
ByteLevel's regex only ever puts a space first in a piece, and Metaspace
with split=True splits on U+2581 — but those are facts about the shipped tokenizers, not about
tokenizers. A T5- or Llama-shaped one keeps Metaspace in the normalizer as Prepend + Replace
and leaves pre_tokenizer null, so BPE merges run over the whole string and the vocabulary holds
pieces spanning what used to be a space. Built one that way (real tokenizers BPE, merges grown
left to right over repeated text so a 40-character cross-space piece forms):

without the check:  budgets 1, 2, 8, 64 return non-prefix ids
                    divergence at token 0 -- the very first id the model sees
with the check   :  none

And the added-token gate has to know both spellings of a space. A sentencepiece-derived
checkpoint stores its added tokens against the normalized text, where a Metaspace-style normalizer
has already rewritten " " to "▁" — so the phrase this gate exists to catch arrives as
▁New▁York and contains no U+0020 at all. A U+0020-only scan passed it:

before:  gate returns True, and the head stops being a prefix of the full ids
after :  gate returns False for both the U+0020 and the U+2581 spelling

Verified on a real tokenizers fixture (Replace(" ", "▁") normalizer, Metaspace(split=True)
pre-tokenizer, AddedToken("▁New▁York", normalized=True)) at phrase lengths of 2, 6 and 10 words.
Worth being precise about the blast radius: in those fixtures the divergence landed past room,
so build_sequence sliced it away and the model saw identical input — I could not make it corrupt a
sequence. But the documented contract was broken, and relying on room's slack to hide that is not
a guarantee, so it is fixed and pinned rather than argued away.

_pre_tokenizer_permits_cut requires a pre-tokenizer that splits on whitespace, and reads the
flags that actually decide it rather than the class name alone: ByteLevel.use_regex and
Metaspace.split can each be off, and with either off the guarantee is gone. A Sequence passes
if any member splits. A tokenizer whose pre-tokenizer cannot be inspected — a slow, pure-Python one
has no backend_tokenizer — is treated as unverifiable and gets the full tokenization, because
"cannot establish it" is not the same as "safe".

Both shipped tokenizers still take the fast path with both gates in place (checked: english
ByteLevel use_regex=True, multilingual Metaspace split=True). A tokenizer that raises on
get_added_vocab is treated as unverifiable and gets the full tokenization; one without the method
has no added tokens to enumerate.

Deliberately not cached: the scan costs 67.7 µs against 3.18 ms for the one tokenization it
guards (best of 5 × min of 200, warmed), and a cache keyed on the tokenizer would go stale the
moment a caller added a token to it.

Measured: tokenization, before and after

English checkpoint, max_len=512 (first budget 4 096 characters). Method: interleaved A/B — 5
warm-up pairs discarded, then 25 alternating pairs, best of each. Interleaved because a sequential
all-before-then-all-after run drifted 25% on this laptop and reported a 24 ms case as 28.23 ms vs
35.26 ms on identical work.

state serialized tokens produced kept before after ratio
trivial request 34 ch 12 12 0.02 ms 0.02 ms 0.99×
typical ticket 2 025 ch 410 410 0.47 ms 0.45 ms 1.04×
serve's 50 000-character cap 50 008 ch 8 931 512 10.22 ms 0.95 ms 10.80×
25 000 six-character fields (dict) 500 000 ch 249 990 512 214.32 ms 0.84 ms 254.49×
CJK, no spaces (fallback path) 50 011 ch 76 940 512 33.07 ms 32.50 ms 1.02×

Measured on the final code, so the density probe and all three tokenizer gates are inside every
"after" figure.

Measured: share of a real request

Same checkpoint, real weights, MPS. before is encode_state_head replaced by the full
tokenization at the call site — literally what main does — in the same process on the same
loaded model
, so the comparison is not across two runs. Interleaved, 2 warm-up pairs, best-of-9.

One question (noul), the shape where tokenizing is the largest fraction of a request:

request before after saved share of request
trivial string (79 ch) 34.8 ms 34.6 ms +0.2 ms nothing (within noise)
serve's 50 000-character cap 147.5 ms 134.7 ms 12.8 ms 8.7%
25 000 six-character fields 434.7 ms 152.4 ms 282.3 ms 64.9%

Three questions, best-of-7 — the state is tokenized once but the forward pass triples, so the
share falls:

request serialized before after delta
trivial string 67 ch 62.2 ms 61.4 ms −0.9 ms
typical ticket 2 000 ch 487.9 ms 485.7 ms −2.2 ms
dict 246 ch 102.3 ms 102.0 ms −0.2 ms
nested dict 873 ch 319.0 ms 317.4 ms −1.6 ms
conversation list 121 ch 86.8 ms 85.6 ms −1.1 ms
long conversation list (2 001 turns) 146 039 ch 596.1 ms 594.3 ms −1.9 ms
unicode string 1 500 ch 559.6 ms 569.6 ms +10.0 ms (1.8%, below the gate — noise)
serve's 50 000-character cap 50 000 ch 579.9 ms 562.6 ms −17.2 ms (3.0%)
25 000 six-character fields 500 000 ch 848.4 ms 515.0 ms −333.5 ms (39.3%)
CJK no spaces 50 011 ch 607.5 ms 608.6 ms +1.1 ms
prose past the budget 158 000 ch 609.2 ms 548.1 ms −61.1 ms (10.0%)

No path gets slower

A prefix optimization is easy to measure only on the case it improves. This is the sweep that looks
for the cases it hurts: shapes × sizes × both checkpoints, interleaved A/B, reporting the worst
ratio anywhere rather than the average. Sizes run from 500 characters to laya.serve's
50 000-character cap; shapes are English and Russian prose, CJK without spaces, flat JSON, dash runs
at 16/64/256 characters, all-spaces, tabs-only, base64, newline runs and emoji.

Three separate regressions were found this way and fixed, none of which the headline cases showed:

found cause fix
0.57× dense-but-cuttable states a full-size first attempt, thrown away, then the whole state the density probe
0.94×/0.88× states just over the budget the prefix was nearly the whole state _STATE_HEAD_MIN_RATIO = 1.25
0.82× states with no cut point the gates were paid for before it was known they were needed gates asked last

Re-measured on the current base, three independent trials per shape, interleaved best-of-31, both
checkpoints.
An earlier revision of this description said no shape at any size is slower and put
the worst case at 0.98×. That was too strong, and the honest numbers are:

                       english                  multilingual
prose 50k        12.20x 11.93x 12.41x       6.02x 6.11x 5.74x
dash runs 20k     0.98x  0.97x  0.96x       0.98x 0.98x 0.99x
base64 20k        0.98x  0.98x  1.00x       0.99x 0.97x 1.00x
CJK 16k           1.04x  0.98x  1.01x       0.98x 0.99x 1.00x
no-space 20k      1.00x  0.96x  0.99x       1.01x 1.00x 1.00x
emoji 10k         0.95x  1.01x  1.02x       0.94x 1.00x 1.03x

So: flat within noise on every incompressible shape, except english dash-runs, which sit
consistently at 0.96–0.98×.
That is the irreducible cost of the probe on a state that then turns
out not to be worth cutting — the density cannot be known without measuring it, and the probe is
bounded to a sixty-fourth of the state precisely to bound this. It is a ~3% cost on a synthetic shape
against 12× on English prose at the same size, and it is reported rather than omitted.

On the measurement itself. These were interleaved A/B — alternating within one loop, best of 31
pairs — because a sequential "all before, then all after" run drifts on this machine and invented
regressions of 0.89× that did not exist. Every flagged case was re-run three independent times before
being treated as real: of eleven flagged in one sweep, ten were noise and one was genuine. The sweep
also records whether the new code path actually executed for each measurement, because several
apparent regressions were at sizes where it cannot run.

Parity: differential, against the real tokenizers

Three tokenizers — both shipped ones, plus english with add_tokens(["New York", "order id"]) —
over 438 state shapes × 5 budgets (need = 1, 8, 64, 512, 1024). No model is loaded; this
is about the ids, not inference.

  • 6 570 head-id comparisons — the head ids must equal the full ids' prefix of the same length,
    and must cover the budget or be the whole state
  • 3 915 assembled-sequence comparisons — build_sequence rows (ids and marker positions)
    from the head ids vs from the full ids, for a choice, a score and a noul question
  • 10 485 comparisons total, 0 differences
english          head 0/2190   seq 0/1305   permit_cut=True
multilingual     head 0/2190   seq 0/1305   permit_cut=True
english+phrase   head 0/2190   seq 0/1305   permit_cut=False   <- refuses to cut, as it must

And the same run with _added_tokens_permit_cut forced to True, so the zero above is a
result rather than an absence of checking:

english+phrase   head 1/2180 bad            <- and only on that tokenizer

One, and only on the tokenizer carrying an internal-space added token. It was 5 before the retry
landed; a smaller first budget happens to avoid the bad cut for four of the five budgets, which is
luck rather than safety — the gate is what makes it 0. Two of the 436 shapes exist solely to make
this control possible, because they contain the added phrase. Without them the control reported
0 differences too, which would have looked like confirmation and meant only that the corpus
never reached the guard.

Shapes: empty string, one word, short prose, exactly at / one character under / one character over
the budget, prose past the budget, serve's 50 000-character cap, no space at all, base64, space
runs, newline runs, space-then-newline, newline-then-space, tabs only, indented JSON, leading
whitespace, all whitespace, CJK, Russian, Arabic, emoji, combining marks, an NFC trap
("a" + U+0301 × 3, repeated), NBSP-separated, ZWSP-separated, a literal [MASK], the two
added-phrase shapes, dict, dict past the budget, nested dict with a 20 000-element list, unicode
keys, a conversation list, a 6 000-turn list, a 20 002-element list of strings — plus 400
randomized cases
over an adversarial alphabet biased toward whitespace runs, at lengths 1, 2, 7,
63, 511, budget−1, budget, budget+1 and 40 000.

The 9 list rows per tokenizer are exempt from the sequence comparison rather than passing it: a
list is left-truncated, so production tokenizes it in full and never builds a head for it. Feeding
one a prefix-truncated head would only re-discover the constraint this change is built around. The
harness had that wrong at first and reported 6 differences per shipped tokenizer, all on the two
oversized list shapes — worth stating, because it is the invariant the whole change rests on.

It needs local checkpoints, so it is not a CI suite; what ships in CI is below.

Real-model end-to-end

Agent on the shipped english checkpoint (ModernBERT-large, 421 M params, max_len=512,
head_max_len=192), HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1, three questions
(choice / score / noul), 11 requests covering all the shapes in the latency table above.

requests: 11   mismatches: 0
RESULT: IDENTICAL

Compared field by field — every answer, every probability to 6 decimal places, and usage — not
just the picked labels.

And the same corpus run against a deliberately broken variant, so the zero above is a result
rather than an absence of checking.
Three trees, same harness, same checkpoint: upstream/main,
this branch, and this branch with the len(head) >= need guard deleted.

case branch vs upstream guard deleted vs upstream
dense_dash256 IDENTICAL DIFFERS
dense_dash256_dict IDENTICAL DIFFERS
dense_dash64 IDENTICAL DIFFERS
prose_short, prose_past_budget, dict, no_space_at_all, turns, turns_past_budget IDENTICAL IDENTICAL
branch            vs upstream: 0 differences of 9
guard deleted     vs upstream: 3 differences of 9   <- and only the three dense states
added-check bypassed         : 0 differences of 9   <- expected; see below

The third row is reported rather than omitted because it is a zero that proves nothing on its own:
the shipped english tokenizer has no internal-space added token, so bypassing
_added_tokens_permit_cut cannot change anything on this checkpoint. Its evidence is the
real-tokenizer measurement in the section above (14 of 59 need values non-prefix without the
check, 0 with it) — and, since an earlier revision of this description, inference too: resizing the
embeddings after add_tokens makes the model valid, and the 398-pair search below reproduces 125
non-prefix heads. None of them moves an answer, for a structural reason worth reading rather than
assuming; see The three gaps, closed.

The same corpus, the same three questions, against the real multilingual checkpoint — mmBERT-base,
321 908 995 parameters, max_len 1024 / head_max_len 256:

multilingual, branch vs upstream: 0 differences of 9
optimization active:  head 1 275 ids vs full 5 441,  is_prefix=True

Both shipped checkpoints are therefore observed answering, not just tokenizing.

The six controls staying identical is the point: the divergence is the deleted guard, not run
noise. A state reaches that guard only by being both dense (the character budget buys fewer
tokens than need) and cuttable (_state_cut returns non-zero); dash runs separated by single
spaces are both, because every space is preceded by -, which is not whitespace. 256 is also
exactly the longest all-dash token surface in the english vocabulary, so the density is a property
of the real tokenizer and not an artefact of the fake one.

usage: what the two token counts now mean

usage["input_tokens"] is unaffected, and that was checked rather than assumed. It is written in
exactly two places and both take it from the attention mask:

n_tokens = int(att[row:row + nrows].sum())

That is the padded sequence length, summed over the state's question rows and capped by
max_len — not the state's token total. Measured on the real checkpoint: a 50 000-character state
with three questions reports input_tokens: 1536 (3 × 512) and a 79-character one reports 149.
grep -n 'len(state_ids)' laya/ finds no other consumer of the total beyond build_sequence's own
slice and predict_long's window scan.

What does change is the two counts, and the rest of this section is about them.

The two counts are opt-in

The problem, in your own measurement. You integrated the branch and diffed against main on the
real checkpoints over a 4 000-character state at max_len=256:

main this branch (before the opt-in)
english state_tokens 709 320
english state_tokens_dropped 506 117
multilingual state_tokens 869 316
multilingual state_tokens_dropped 664 111

The counts move because the prefix is all that was tokenized, so len(state_ids) is the prefix
length. They were not wrong — they were floors. And as you put it, a count that is exact on one path
and a floor on another is the outcome most worth avoiding.

What is implemented. state_token_counts, default False:

  • Not asked — the head path runs, the two keys are absent. truncated and
    truncated_questions are present and exact.
  • Asked — the full tokenization is restored for that call, and both counts are exact.

Reported by the same mechanism usage["options"] uses (#569): a conditional assignment after the
usage literal, in the same function.

always (the dict literal) : input_tokens, output_tokens, truncated, truncated_questions
conditional               : options (#569), state_tokens (#687), state_tokens_dropped (#687)

Why truncated can stay free, and why that is not a compromise. encode_state_head only ever
returns early with at least need = max_len ids; every other exit returns the full tokenization.
And room = max(0, max_len - len(ids) - 1) where the question head is at least [CLS] … [SEP] [SEP],
so room <= max_len - 4 < max_len. On the head path, therefore, len(state_ids) >= max_len > room
always holds: the clamp really did drop tokens, for every question, so truncated is True and
genuinely true. Your own diff found truncated / truncated_questions identical, and this is why.

Asking costs what it should, and nothing more. Asking takes the same encode_text call
upstream/main takes, so an asking call is upstream's speed, not worse. The opt-in's own overhead —
one boolean test and one dict construction — measures 24 ns per call, against a tokenization in
milliseconds.

Verified against upstream, both checkpoints, 9 state shapes (prose at three sizes, dict, CJK,
dense dash runs, no-space, and two conversation lists):

answer differences, not asking ......................... 0 of 9   (english and multilingual)
answer differences, asking ............................. 0 of 9   (english and multilingual)
asking reproduces upstream's whole `usage` dict ........ 9 of 9   (english and multilingual)
keys absent by default ................................. exactly {state_tokens, state_tokens_dropped}

The third line is the one that matters most: when a caller asks, the response is not merely similar
to main's — it is main's, key for key and token for token.

Surface. state_token_counts is accepted by Agent.system_one / predict_batch / predict_long,
the same three on ONNXAgent, all three Router entry points, both HTTP routes (added to
BODY_CONTROLS and BATCH_BODY_CALL_CONTROLS, boolean-only, 422 otherwise), the CLI as
--state-token-counts, and examples/server.py.

It is passed only when it is on, at every hop. That is not tidiness: three separate contract tests
in this repo caught a caller broken by passing it unconditionally — a fake _encode_state in
tests/test_batch.py and tests/test_hooks.py, and the forwarded-kwarg set
tests/test_predict_long.py pins. So the default call shape is byte-identical to what it was, and a
subclass, an attached agent, or a replacement predict_batch that predates the argument keeps working
on every call that does not ask.

What this breaks, and what it does not

One thing breaks: a caller reading usage["state_tokens"] or usage["state_tokens_dropped"] gets
nothing back unless it asks. Two of this repo's own examples did exactly that and would have
raised KeyError:

  • examples/15_truncation_basics.py — prints both counts from a plain predict()
  • examples/37_long_document.py — same, in two places

Both now pass state_token_counts=True and say why, which suits them: they are the pages that teach
the truncation report. docs/http-api.md, docs/structured.md and
examples/03_reading_the_result.py describe the keys as opt-in and name the field that asks.

Nothing else moves. truncated, truncated_questions, input_tokens, options, every answer, every
probability, and the routing block are untouched — measured above, not argued.

The opt-in's own checks, and the mutation table

tests/test_state_token_counts.py (new, 295 lines, 32 checks, weight-free, registered in
scripts/test_suites.py) pins the behaviour rather than the wording:

  • which of the two tokenizer entry points actually runs, on each path — patched on laya.agent, not
    on laya.common, because agent.py does from .common import encode_state_head, encode_text and
    patching the defining module would leave the test measuring nothing
  • the usage key set in both modes, and that asking's state_tokens equals a full tokenization
  • that a conversation list tokenizes in full either way, so asking changes only what is reported
  • that system_one, predict_long, both ONNXAgent paths and all three Router entry points
    forward it, and that none of them forwards it when it is off

A stub tokenizer cannot reach the prefix path at all — all three of _state_cut's soundness gates
inspect tokenizer internals a stub does not have, so a stub always takes the full path. That is why
this suite pins which call runs and leaves numerical exactness to the real-checkpoint runs above,
and it says so in its docstring rather than implying coverage it does not have.

Mutation table: 15 of 15 killed.

mutation killed by
asking no longer restores the full tokenization (torch) test_truncation, test_state_token_counts
counts always reported — the opt-in removed (torch) test_state_token_counts
counts never reported (torch) test_state_token_counts
asking never reaches _encode_state (torch) test_state_token_counts
predict_long stops forwarding it (torch) test_state_token_counts
asking no longer restores the full tokenization (ONNX) test_onnx_truncation_parity
counts always reported (ONNX) test_onnx_truncation_parity
asking never reaches _encode_state (ONNX) test_onnx_truncation_parity
predict_long stops forwarding it (ONNX) test_state_token_counts
Router.predict drops the override test_state_token_counts
Router.predict_batch drops the kwarg test_state_token_counts
_ScanLong drops it for predict_long test_state_token_counts
the serve validator always says "not asked" test_serve
the control leaves BODY_CONTROLS test_serve
the CLI flag is never forwarded test_cli

Worth recording because it is the reason this took a second pass: 7 of the first 11 mutants
survived.
The feature worked and every suite was green, and the opt-in could have been deleted in
either direction without a single test noticing. The suite above is what closed that, and the table
was re-run after the rebase.

The checks, and that they fail without the change

Added to tests/test_truncation_direction.py, which the ci.yml pytest lane already runs
(Pytest suites (serve, batch, system-one, audit regressions, truncation, compile)), so there is no
CI wiring to add.

The fake tokenizer models what matters: ids are a function of a whole piece, and every maximal
whitespace run gets an id of its own, which is the added-token behaviour that broke the first cut
rule. So a wrong cut point fails in CI without a checkpoint.

Revert check. Source copied aside, laya/agent.py, laya/common.py and laya/onnx_agent.py
replaced with upstream/main's content in the working tree, the tests kept, suite re-run, then the
source copied back and confirmed byte-identical by checksum:

with the change:  44 passed, and tests/test_state_token_counts.py 32 passed
source reverted:   7 passed, 37 failed, and tests/test_state_token_counts.py cannot run at all:
                   TypeError: Agent.predict_batch() got an unexpected keyword argument
                   'state_token_counts'

37 checks go red, and they are the ones that assert the new behaviour: the prefix path, the three
cut gates, the retry and its budget, and the two paths each agent takes. The opt-in suite does not
merely fail — it cannot run, because the argument it is about does not exist on the reverted source.

Two checks pass both ways on purpose — a conversation list stays tokenized in full, and
input_tokens is the sequence length rather than the state total. They pin behaviour this change must
not move.

Mutation checks, each made and reverted, to show the assertions bite rather than merely run.
Every row below was re-executed on 2026-09-29 — applied to the source, suite run, source restored —
rather than carried over from when it was first written. That audit is what found the two unpinned
short-circuits above. Of the 24 rows that predate it, 22 behaved exactly as recorded and two named
the wrong test
, both corrected here; one of those two, the retry budget never grows, was green
against the old suite as well and is closed by the same new check. A mutation table is only worth the
last time someone ran it.

mutation red
the first (wrong) cut rule: a space unless whitespace on both sides test_state_head_ids_match_a_full_tokenization
no boundary logic at all — a raw character slice test_state_head_ids_match_a_full_tokenization
head budget applied to list states too ..._sequences_are_unchanged..., ..._list_in_full
STATE_HEAD_CHARS_PER_TOKEN = 1 ..._leaves_a_state_barely_over_the_budget_alone — the retry recovers the ids, but a state barely over the budget gets tokenized whole
_STATE_HEAD_MAX_ATTEMPTS = 1 (no retry) ..._retries_from_the_density_it_measured
the retry budget never grows ..._guard_rejects_a_short_head_when_density_rises_past_the_probe
the len(head) >= need guard deleted ..._guard_rejects_a_short_head_when_density_rises_past_the_probe
the len(probe) >= need short-circuit deleted ..._stops_at_the_probe_when_the_probe_already_covers_the_budget
_added_tokens_permit_cut bypassed ..._refuses_to_cut_when_an_added_token_holds_an_internal_space
the added-token scan never rejects anything ..._refuses_to_cut_when_an_added_token_holds_an_internal_space
_pre_tokenizer_permits_cut bypassed ..._refuses_to_cut_when_nothing_splits_on_whitespace
_normalizer_permits_cut bypassed ..._refuses_to_cut_when_the_normalizer_drops_the_cut
the normalizer probe never rejects ..._refuses_to_cut_when_the_normalizer_drops_the_cut
the normalizer prefix check removed ..._refuses_to_cut_when_the_normalizer_drops_the_cut
_STATE_HEAD_MIN_RATIO = 1.0 (no break-even margin) ..._leaves_a_state_barely_over_the_budget_alone
the worth-it projection removed ..._does_not_pay_for_a_prefix_it_will_not_use
the probe sized at the full budget ..._stops_before_the_end_of_a_large_state, ..._does_not_pay_for_a_prefix...
the ONNX site reverted to a full tokenization ..._onnx_agent_takes_the_same_two_paths_as_the_torch_agent
the ONNX site applying the head budget to lists too ..._onnx_agent_takes_the_same_two_paths_as_the_torch_agent
the ONNX site dropping the mask replacement ..._onnx_agent_takes_the_same_two_paths_as_the_torch_agent
U+2581 dropped from _CUT_CHARS (either branch) ..._added_token_gate_fails_closed_and_covers_both_spellings, ..._refuses_to_cut_when_the_normalizer_drops_the_cut
the added-token gate failing open on an uninspectable tokenizer ..._added_token_gate_fails_closed_and_covers_both_spellings
the non-advancing retry allowed again ..._state_head_retry_does_not_repeat_an_identical_prefix
Metaspace.split ignored ..._splits_on_whitespace_reads_the_flags_that_actually_decide
ByteLevel.use_regex ignored ..._splits_on_whitespace_reads_the_flags_that_actually_decide
a tokenizer that cannot be inspected treated as safe ..._refuses_to_cut_when_nothing_splits_on_whitespace

test_state_head_ids_match_a_full_tokenization itself makes 160 comparisons (2 fake tokenizers
× 4 budgets × 20 state shapes) and test_agent_sequences_are_unchanged_by_the_head_budget makes
60 (3 questions × 20 shapes), both asserted counts so a shape list that silently shrinks fails.

Every line of the fix has a mutation that turns a test red — 14 of them. That is not incidental:
five times on this change a fix shipped whose removal left the whole suite green (the
len(head) >= need guard, the retry, Metaspace.split, the normalizer prefix check, and the gate
ordering). Each is now pinned, the gate ordering by asserting the gates are not consulted for a
state that was never going to be cut.

One of those five stayed unpinned for longer than this description admitted, and it took a
re-audit to notice.
An earlier revision credited the len(head) >= need guard to the
dense-but-cuttable shape. That is wrong, and provably so: a dense-and-cuttable state is caught by
the worth-it projection and returns a full tokenization before the retry loop is ever entered —
the 82 240-character dash state exits there, which is why it produces the full 321 ids. So deleting
the guard left all 22 checks green, exactly the failure mode this suite exists to prevent.

The guard is reachable only when the text past the probe is denser than the text the probe read,
by more than the 1.25× margin, because then want is extrapolated from a density that no longer
holds. ..._guard_rejects_a_short_head_when_density_rises_past_the_probe is that shape: 512 sparse
characters (4 chars/token) then 82 000 dense ones (20.5), so the probe measures 4 and the retry has
to climb to ~17.

attempt 0 (probe)     511 chars ->  127 ids   short
attempt 1            2 602 chars ->  229 ids   short   <- returned, without the guard
attempt 2            7 317 chars ->  459 ids   short
attempt 3           10 228 chars ->  601 ids   covers 512

Deleting the guard returns 229 ids where 512 were asked for. The same audit found
if len(probe) >= need: return probe unpinned as well — a throughput line rather than a correctness
one, since falling through pays a second tokenization for a correct but needlessly large head, which
no ids-only assertion can see. It is now asserted by counting tokenizations. Both new checks are
verified to go red on deleting their own line and to stay green when _STATE_HEAD_MAX_ATTEMPTS
changes, so neither is pinned to that constant.

Two of the 20 shapes exist for guards rather than for coverage. Dense-but-cuttable is the only
shape that reaches the worth-it projection with a cut available, so it is what keeps that branch
honest. A state carrying the mask token makes the .replace(mask_token, " ") that _encode_state
applies something the parity test actually exercises, rather than a no-op on every shape, so the
test tokenizes the text production tokenizes instead of only coincidentally matching it.

Gates

ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   All checks passed!
python -m compileall -q laya/ tests/                                  clean
python scripts/test_suites.py                                         see below

The canonical suite list now lives in scripts/test_suites.py, which both ci.yml and the release
gate invoke, so tests/test_state_token_counts.py is registered there rather than in a workflow.

Everything in that list passes except one suite, which fails on main as well:
tests/test_shortlist_cosine.py — 3 failures, 1 error, identically on upstream/main @ fa9a2a7
and on this branch
, and this PR does not touch laya/shortlist.py or that test. Flagging it rather
than leaving you to find it in this PR's output.

CI on the pushed head: 24 checks pass, 1 skipped (docs (deploy), which does not run on a PR) —
including tests on py3.10/3.11/3.12/3.13 and Windows, http, onnx export, evals, lint,
package, CodeQL, both smoke jobs and the TypeScript SDK lanes.

The three gaps, closed

An earlier revision of this description listed three things the evidence did not reach: multilingual
never observed answering, no ONNX run through a real fused graph, and the added-token guard checked
through the tokenizer rather than through inference. All three are now measured.

1. Multilingual, through real inference. The e2e harness is parameterised by checkpoint, and the
same 9-shape corpus ran against the real multilingual checkpoint — mmBERT-base, 321 908 995
parameters
, max_len 1024 / head_max_len 256 from its own rl_agent_config.json:

multilingual, 9 request shapes, branch vs upstream/main:   0 answer differences
optimization confirmed active:  head 1 275 ids vs full 5 441,  is_prefix=True

The second line matters as much as the first: a zero-difference run on a corpus that never reached
the new code would prove nothing. On the dense-and-cuttable shapes the head really is a fraction of
the state, and it really is a prefix of it.

2. ONNX, end to end at real weights. Two earlier revisions of this description got this wrong, in
opposite directions, and the correction is the useful part. The first said no fused graph was
available — ONNXAgent cannot consume the two-graph encoder.onnx + head.onnx pair, and I recorded
"no local artefact" without checking. The second built a 209 KiB fused graph from a tiny encoder +
head and called the gap closed, noting that real weights "are not the point". They are the point, and
they were available the whole time: the rig holds three real fused exports of the 322M multilingual
checkpoint
, each a 1.2 GB external-data pair. So the run below is the real one.

ONNXAgent on the real fused graph (1.20 GB, CPUExecutionProvider, batch axis batch_size) with
the real multilingual tokenizer, max_len 1024 / head_max_len 256, the same 9 shapes the torch
harness uses, branch against upstream/main:

shape verdict answer state ids
dense_dash256 IDENTICAL technical head 1 275 of 5 441, prefix ✓
dense_dash256_dict IDENTICAL technical head 1 125 of 5 451, prefix ✓
dense_dash64 IDENTICAL technical head 1 280 of 5 001, prefix ✓
prose_past_budget IDENTICAL billing head 1 291 of 6 401, prefix ✓
no_space_at_all IDENTICAL technical 2 503 of 2 503 — no cut in reach, unchanged
dict IDENTICAL billing 34 of 34 — under the ratio
prose_short IDENTICAL billing 17 of 17 — under the ratio
turns IDENTICAL billing list: tokenized in full, left-truncated
turns_past_budget IDENTICAL billing list: tokenized in full, left-truncated
0 answer differences of 9

Four shapes reach the optimization and take roughly a fifth of the ids they used to; the answers do
not move. The list shapes keep the full tokenization, which is the constraint this whole change is
built around. Harness: e2e-onnx-state-head.py, parameterised by tree and by graph.

One thing worth passing on from the tiny-graph run that preceded this: instrumenting it needs the spy
on both common.encode_text and onnx_agent.encode_text — the prefix path routes through the
first and truncate_left through the second, so patching one reports an empty trace for three of four
rows — and it has to filter to calls on the state text, or the question tokenizations land in the
count. Both mistakes were made before they were caught.

3. The added-token guard, through inference. The earlier reasoning was that a checkpoint adding
tokens also resizes its embeddings, so an added id on this model would be out of range and any
answer comparison would be measuring a broken model. That is avoidable —
agent.model.encoder.resize_token_embeddings(len(agent.tok)) after add_tokens(["New York", "order id"]) makes the model valid — so the comparison was run:

randomised search, 5 added phrases x 4 question shapes, 398 pairs, guard bypassed:
  125 heads that are NOT a prefix of the full ids      (the bug reproduces)
    0 that diverge inside `room`                       (no answer moves)
  earliest divergence index 578, against room 487

So the guard protects an invariant that is real but, on the shapes reachable here, not
answer-visible — and there is a structural reason, which is the useful part. A head is only returned
when len(head) >= need = max_len; the divergence sits at the cut, i.e. in the head's tail; and
build_sequence keeps only room < max_len ids. The corruption is always past the slice. The guard
stays because room is a function of question size and nothing pins it below max_len — a caller
with tiny questions moves room up toward max_len, and the margin that is absorbing this today is
not guaranteed by anything.

Integration with the other PRs from this fork

#689 landed in 0.3.22, as you noted, and this branch is rebased past it with no conflict. The
interaction it posed is worth recording because it was the one real risk between the two: #689 caps
each predict_long window at the room the questions leave, and encode_state_head then takes a
prefix of that window, so the concern was the second re-truncating what the first had just sized. It
does not — a window is at most room tokens and room < max_len, so the head path cannot cut one
short. Window counts and widths were identical with both changes present, and document coverage at
100 options stayed at 0% unread.

Still open from this fork: #688 (ONNX batch-dynamic export) and #690 (router pinning
controls). Neither shares a file with this PR.

This rebase crossed 130 upstream commits and four files conflicted. Each was resolved in
upstream's favour on the facts, which is worth stating explicitly since three of the four were your
own corrections landing against text this branch had edited:

file resolution
.github/workflows/ci.yml upstream's, untouched — the suite list moved to scripts/test_suites.py, so the new suite is registered there
laya/agent.py kept upstream's corrected note on when a predict_long window is cut, and the asked argument from the in-place-rewrite fix
tests/test_serve.py kept the new answer-table gate alongside this change
examples/03_reading_the_result.py this branch's rewrite of the paragraph, which supersedes the wording tweak

One upstream test needed a change, and it is the only place this PR touches logic it did not add.
tests/test_truncation.py identified "the truncation report" by subtracting the two token totals from
the usage literal. That stops picking out the report once any key in that block is conditional — and
options was already a counter-example waiting, since #569 publishes it the same way. The four keys
are now named explicitly and then checked to be published, which is the half of the old check that had
teeth.

@keemsisi
keemsisi marked this pull request as draft September 28, 2026 13:56
@keemsisi keemsisi changed the title perf(agent): tokenize only as much state as a prefix-truncated sequen… perf(agent): tokenize only as much state as a prefix-truncated sequence can hold Sep 28, 2026
@keemsisi
keemsisi force-pushed the perf/state-prefix-tokenization branch 3 times, most recently from 52a4be2 to b2958c4 Compare September 28, 2026 19:57
@keemsisi
keemsisi marked this pull request as ready for review September 29, 2026 13:22
@keemsisi
keemsisi marked this pull request as draft September 29, 2026 14:03
@keemsisi
keemsisi force-pushed the perf/state-prefix-tokenization branch from ade029e to eac5d9d Compare October 1, 2026 01:06
@NandhaKishorM

Copy link
Copy Markdown
Owner

Holding this for 0.3.23, and the reason is the question you yourself raised rather than anything I found wrong with the code.

You flagged that state_tokens and state_tokens_dropped become lower bounds on the head path, offered three ways to compose with #668, and asked which one to take. #668 is on main, so that question is now live rather than hypothetical, and I measured what it costs before answering. Integrating this branch and diffing decisions against main on the real checkpoints, over a 4,000-character state at max_len=256:

main this branch
english state_tokens 709 320
english state_tokens_dropped 506 117
multilingual state_tokens 869 316
multilingual state_tokens_dropped 664 111

input_tokens, truncated and truncated_questions are identical, and so is every answer: zero decision flips and a max probability delta of 0.000000 across the whole diff. So the analysis in your description is exactly right, including the part about the truncation signal staying exact. The two counts are the only thing that moves, and they move a lot.

Your option 1 is the one I want: keep truncated / truncated_questions always on and free, and make state_tokens / state_tokens_dropped opt-in, with asking restoring the full tokenization for that call. The usage["options"] precedent from #569 is the right model. A count that is exact on one path and a floor on another is the outcome I most want to avoid, and your own ranking said the same.

Nothing else blocks this. #689 landed in this release on its own and did not conflict with your branch. Rebase onto main with the counts opt-in and I will take it.

@keemsisi

keemsisi commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Holding this for 0.3.23, and the reason is the question you yourself raised rather than anything I found wrong with the code.

You flagged that state_tokens and state_tokens_dropped become lower bounds on the head path, offered three ways to compose with #668, and asked which one to take. #668 is on main, so that question is now live rather than hypothetical, and I measured what it costs before answering. Integrating this branch and diffing decisions against main on the real checkpoints, over a 4,000-character state at max_len=256:

main this branch
english state_tokens 709 320
english state_tokens_dropped 506 117
multilingual state_tokens 869 316
multilingual state_tokens_dropped 664 111
input_tokens, truncated and truncated_questions are identical, and so is every answer: zero decision flips and a max probability delta of 0.000000 across the whole diff. So the analysis in your description is exactly right, including the part about the truncation signal staying exact. The two counts are the only thing that moves, and they move a lot.

Your option 1 is the one I want: keep truncated / truncated_questions always on and free, and make state_tokens / state_tokens_dropped opt-in, with asking restoring the full tokenization for that call. The usage["options"] precedent from #569 is the right model. A count that is exact on one path and a floor on another is the outcome I most want to avoid, and your own ranking said the same.

Nothing else blocks this. #689 landed in this release on its own and did not conflict with your branch. Rebase onto main with the counts opt-in and I will take it.

Done — option 1 exactly as you described it, rebased onto main @ fa9a2a7.

  • truncated / truncated_questions: always on, free, unchanged.
  • state_tokens / state_tokens_dropped: absent unless the caller passes state_token_counts=True.
  • Asking restores the full tokenization for that call, so the counts are exact.
  • Published the same way usage["options"] is (feat(common): report options that collapse into the same token span (#538) #569): a conditional assignment after the usage
    literal, in the same function.

The result I most wanted to show you — asking gives back main's whole usage block exactly, on
both checkpoints, over 9 state shapes:

answers, not asking ............................. 0 differences of 9
answers, asking ................................. 0 differences of 9
asking reproduces main's whole `usage` dict ..... 9 of 9
absent by default ............................... exactly the two counts

Why truncated stays exact, since you measured it identical: encode_state_head only returns early
with at least max_len ids, and a question's room is always below max_len. So on the head path
tokens really were dropped — the flag is true, and true for the right reason.

Available on Agent and ONNXAgent (system_one / predict_batch / predict_long), all three
Router entry points, both HTTP routes (boolean only, 422 otherwise), --state-token-counts on the
CLI, and examples/server.py. Passed only when it is on, so the default call shape is unchanged.

Three things worth flagging:

  1. Two shipped examples read those counts from a plain predict() and would now raise
    KeyError — examples/15_truncation_basics.py and examples/37_long_document.py. Both ask now,
    and explain why. Docs updated to match.

  2. One upstream test needed a small logic change, the only thing here I did not add.
    tests/test_truncation.py found the truncation report by subtracting the two token totals from the
    usage literal, which no longer identifies it once a key in that block is conditional. The four
    keys are named explicitly now. Happy to split it into its own commit if you prefer.

  3. tests/test_shortlist_cosine.py fails on main — 3 failures, 1 error, identically with and
    without this branch, which touches neither that test nor laya/shortlist.py. Just so it does not
    look like fallout from here.

CI: 24 checks pass, 1 skipped (docs (deploy), which does not run on PRs), including tests on
3.10–3.13 and Windows. New suite tests/test_state_token_counts.py (32 checks, no weights) is
registered in scripts/test_suites.py; its mutation table is 15 of 15 killed.

Taking it out of draft.

@keemsisi keemsisi changed the title perf(agent): tokenize only as much state as a prefix-truncated sequence can hold perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in Oct 2, 2026

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rechecked this head specifically against the two conditions Nandha left before taking it, rather than re-auditing the whole PR. It is rebased onto current main, and option 1 is wired as requested: the default usage block omits only state_tokens / state_tokens_dropped; state_token_counts=True switches back to full state tokenization so those counts are exact; truncated / truncated_questions remain always available.

I also traced the flag through Agent, ONNXAgent, Router, both HTTP paths and the CLI. It is forwarded only when enabled, with the Router compatibility retry preserving older attached Agent-like objects. CI is green on the current head.

I did not independently rerun the author's real-checkpoint 9-shape equivalence matrix, so I am treating that part as author evidence. The original rebase + opt-in-counts blocker itself looks closed.

@keemsisi

keemsisi commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

@NandhaKishorM PR pending your review.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Not merging from this branch. It is 767 commits behind main and touches fourteen files across the core, the CLI, the MCP remote, the ONNX agent, the router and the server, so the conflict surface is the whole request path rather than one module.

The idea is right and I want it: tokenizing only the prefix of state that a sequence can actually hold, instead of tokenizing the whole thing and discarding the tail, is pure waste removed, and it gets worse the longer the document. Making state_tokens opt-in is the correct consequence, because the full count stops being free once you stop computing it.

What a rebase has to reckon with, and why I am not doing it for you:

  • predict_long now exists and scans windows. The truncation path and the window path both ask "how much of this state fits", and after this change they would answer from different code. feat(mcp): add laya_predict_long for states past the context window #738 landed laya_predict_long over MCP in 0.4.11, so that path has callers now.
  • state_tokens becoming opt-in is a visible contract change. usage is read by laya-serve responses, by the evals harness and by the Jev-compatible wire shape under LAYA_JEV_STRICT, which did not exist when this branch was cut. A field that silently becomes absent is worse than one that is explicitly None, and the strict shape has its own rules about which keys appear.
  • The truncation direction work moved. tests/test_truncation_direction.py is in the suite list now and pins which end is kept.

The thing that would make this merge easily is a measurement, which the current description does not have: tokenization time saved as a fraction of total predict time, on a long state, at a couple of state lengths. The change is justified by waste, so the size of the waste is the argument. If it turns out to be small on realistic states, the opt-in contract change is not worth it; if it is large, that number is what settles the usage question.

Leaving this open. A narrower first PR, just the prefix tokenization with state_tokens unchanged and still computed, would be straightforward to review and would bank most of the win.

…ake the state-token counts opt-in

build_sequence keeps state_ids[:room] (room < max_len) for every state but a conversation
list, so the tail was tokenized only to be discarded. encode_state_head tokenizes a prefix cut
where the cut cannot change the ids before it, and falls back to the full tokenization
otherwise.

usage["state_tokens"] / ["state_tokens_dropped"] count the whole state, so they are now absent
unless state_token_counts=True, which restores the full tokenization for that call.
truncated / truncated_questions stay always-on and exact. predict_long tokenizes in full (every
window is scored) through the same encode_state, and forwards the flag on both of its paths.
Surfaces: Agent, ONNXAgent, Router, laya.serve, CLI, MCP remote, the TypeScript SDK; the
golden generators ask for the counts so regenerated fixtures keep them.
… absent keys are absent and strict usage is fixed
…nnot take it; MCP tools and hooks carry it

Router.predict, predict_batch and predict_long now check the agent's signature and raise
StateTokenCountsUnsupported (a TypeError and ValueError, so laya.serve answers 422) instead of
one path raising a raw TypeError and the batch retry silently dropping the counts.
RemoteRouter.predict raised KeyError on every call, because state_token_counts is in
BODY_CONTROLS; it now sends the flag through the same wire-controls loop.
laya_predict / laya_predict_long / laya_predict_batch take state_token_counts.
PredictContext.state_token_counts tells a result cache to key on it.
Tests: the four refusal paths, serve forwarding on the tournament and predict fallback,
CLI --batch, RemoteRouter bodies, MCP tools, hook context, and encode_state_head against a
real tokenizers object.
… so positional construction is unchanged; re-point a shifted citation
@keemsisi
keemsisi force-pushed the perf/state-prefix-tokenization branch from dc85053 to 8cb3dc0 Compare October 20, 2026 02:08
…nly common.py, generators ask for the counts

- common.py held a literal U+2581; test_criteria reads it with the platform codec and
  failed on Windows (cp1252). Written as an escape now.
- The two counts are placed after output_tokens again (state_counts_in_place), so the
  usage dict keeps the key order every SDK golden compares; laya-go/fixtures and
  laya-java/fixtures regenerate byte-identical to main.
- The tournament recorder, the Go predict_plan generator and the .NET predict_long dumper
  ask for the counts, as the other generators already did.
- test_state_token_counts pins the key order (fails when the helper is a no-op).
keemsisi added a commit to keemsisi/laya that referenced this pull request Oct 21, 2026
…tate counts

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants