Skip to content

fix(serve): measure the state limit on the text the tokenizer receives - #691

Merged
NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
keemsisi:fix/serve-state-size-gate
Sep 29, 2026
Merged

NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
keemsisi:fix/serve-state-size-gate

Conversation

@keemsisi

@keemsisi keemsisi commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

In short: laya/serve.py's MAX_STATE_CHARS = 50000 gate measures len(str(state)), but the
text that reaches the tokenizer is serialize_state(state) — json.dumps(state, ensure_ascii=False) for a dict or list. Those are different lengths in both directions, so over
HTTP the declared 50 000-character limit admitted a state that tokenized as 99 988 characters,
and refused a legitimate state that tokenized as 8 344. The fix measures the serialized form,
which costs 0.133 ms at the largest state the gate admits — three orders of magnitude under
the forward pass that state then gets. The same lines also answered 413 "state too large (50001 > 50000 chars)" for a state nothing had measured; that becomes a 400.

This is an observable behaviour change, not only a hardening. A state that serializes above
50 000 characters but whose repr was under it used to be answered 200 and is now 413; a state
whose repr was over but which serializes under is now answered 200 where it used to be 413.
Nothing else about the route moves, and the docs table now states the contract explicitly.

The bug

laya/serve.py:249:

state_len = len(state) if isinstance(state, str) else len(str(state))

laya/common.py:

def serialize_state(state):
    if isinstance(state, str):
        return state
    return json.dumps(state, ensure_ascii=False)

build_sequence tokenizes serialize_state(state). For a string state len(state) is already
exact, so this is only about dicts and lists — and for those, repr and JSON disagree.

What a request can actually carry

Every key in a state that arrives over HTTP is a str, because json.loads makes it one, so the
divergence has to come from the values. Measured on a {"a": <char> * 1000} state:

dict value of 1000× len(str) len(json) ratio
plain ASCII x 1009 1009 1.000
double quote " 1009 2009 1.991
NUL \x00 4009 6009 1.499
zero-width space U+200B 6009 1009 0.168
U+E0001, a non-BMP format char 10009 1009 0.101
single quote ' 1009 1009 1.000
newline 2009 2009 1.000
CJK U+4E2D 1009 1009 1.000
emoji U+1F600 1009 1009 1.000

repr picks ' quotes for a value that has no apostrophe in it, so a " inside stays one
character; JSON escapes it to \" and makes it two. Going the other way, repr escapes a
zero-width space to the six characters \u200b and a non-BMP character to the ten characters
\U000e0001, while ensure_ascii=False writes the one character each of them is. So the two
measurements diverge by up to 2x upward and 10x downward on states a client can send.

Both directions are reachable with a body far under the existing 2 MiB MAX_BODY_BYTES cap:

{"body": '"'      * 49988}   len(str) = 50000  -> passed the gate
                             len(json) = 99988  -> 49 988 characters past the declared limit
{"body": "\u200b" * 8332}    len(str) = 50004  -> refused, "state too large (50004 > 50000 chars)"
                             len(json) = 8344   -> a sixth of the limit
{"a": "\U000e0001" * 8000}   len(str) = 80009  -> refused, "state too large (80009 > 50000 chars)"
                             len(json) = 8009   -> a TENTH of the limit, refused as 16x over it

Those two bodies are 50 032 and 96 037 bytes as a default JSON encoder writes them
(ensure_ascii=True, so the zero-width space goes out as the six bytes \u200b and U+E0001 as a
surrogate pair) — both far inside the 2 MiB cap. The non-BMP case is the worst false refusal of the
four; I checked that json.loads really does hand back U+E0001 from that pair, that the fixed gate
measures it as 8 009 exactly as serialize_state does, and that it now answers 200.

Non-string keys, for callers that are not the HTTP route

repr leaves a non-string key bare where JSON must quote it, so a dict keyed on integers
diverges by two characters per key. This cannot arrive over HTTP — json.loads returns string
keys — but _check_request_limits is also what examples/server.py mirrors, and an in-process
caller can build such a state. Measured on {i: 0 for i in range(n)}:

int keys len(str) len(json) ratio
1 6 8 1.333
10 60 80 1.333
1 000 7 890 9 890 1.253
10 000 88 890 108 890 1.225

At 5 601 keys that state's str() is 49 299 characters, which passes the 50 000 gate, and its
serialization is 60 501 — 11 202 characters more than the gate measured, and 10 501 past the
declared limit.

The except Exception

except Exception:
    state_len = MAX_STATE_CHARS + 1

str() on an integer of more than 4 300 digits raises ValueError (CPython's own int-to-str
guard), so this answered 413 "state too large (50001 > 50000 chars)" — a count nothing had
measured, for a state of a few kilobytes. It is not HTTP-reachable: json.loads refuses that
integer itself, and _systemone_inner already answers that 400 "request body must be valid JSON" (verified, below). It is in-process callers who reach it, and for them the branch was
doing real harm in the other direction: json.dumps raises TypeError where str() succeeds, so
a state serialize_state cannot render — a set, a datetime — passed the gate on its
repr() and failed later inside serialize_state, which _systemone_inner reports as a 500 "inference failed".

Was the old behaviour deliberate? Three reasons it is not

Worth settling, because one of the four cases makes the gate stricter and one makes it
looser, and a reviewer is right to ask whether the strict one was the intent. It was not, on the
evidence in the repository rather than on my reading of it:

1. The gate states its own purpose, and it is not "bound the repr". The comment above the
constants:

Guardrails for unauthenticated remote input. The state is tokenized once per question and
collated into one tensor, so an unbounded body can OOM the worker; the single-worker pool means
one large request would also starve /health.

What is tokenized is serialize_state(state). The gate measures str(state). Those are different
strings for every dict and list, so the guardrail is not measuring the quantity it exists to bound.

2. The line's own shape is an approximation, not a policy.

state_len = len(state) if isinstance(state, str) else len(str(state))

The string branch is exact — serialize_state returns a string unchanged, so len(state) is
precisely what the tokenizer receives. The container branch reaches for str(). A deliberate
decision to bound reprs would not have got the string case exactly right and then approximated the
other one; this is the shape of "close enough for dicts".

3. No existing test distinguishes the two measurements. The only dict case upstream asserts is
tests/test_example_server_limits.py:144:

{"body": "A" * (MAX_STATE_CHARS + 1)}   ->   413

Every character is A, where repr and JSON agree exactly at one character each — so that
assertion passes under either measurement and pins neither. If repr-length were the intended
contract, nothing in the suite says so.

And the argument that settles it: a deliberate cheap upper bound would be conservative. This one
is not. It over-counts a zero-width space by 6x and under-counts a double quote by 2x, so 99 988
characters of real tokenizer work pass a 50 000-character cap. A proxy that is an over-estimate for
some inputs and an under-estimate for others is not a coherent limit under any reading — it is
simply the wrong string. The provenance is consistent with that: the line arrived in 71d235f,
"serve: timing-safe auth, request limits, safe errors, validated port (#250)", a hardening change
whose goal was bounding request work.

So this PR does not relax the gate. It refuses strictly more of what can actually hurt the
worker — the " body goes from 200 to 413 — and stops refusing payloads that never could, like the
zero-width-space body at a sixth of the limit. The 413-to-200 direction costs no protection, because
a repr is never what reaches the tokenizer.

The fix

The measurement moves into _state_length(), which measures what will be tokenized and raises
400 "'state' must be JSON-serializable" for a state that has no such text. examples/server.py
borrows that helper through the same getattr(_laya_serve, ...) shape it already uses for the
bounds, instead of restating the measurement — its docstring claimed it measured "exactly as
laya.serve measures it — len(v) for a string, len(str(v)) for a dict or list", which is the
line being fixed.

Two consequences on the demo side that come with it: its fallback (for an installed laya that
predates _state_length) raises the same 400 rather than letting a bare TypeError out of
_check_request_limits, which /predict calls outside its own try; and POST /gui's handler now
branches on the status, because a 400 rendered under its "Request too large" heading with a
character-limit hint would describe the wrong problem.

MAX_STATE_CHARS keeps its documented value of 50 000. The point is that it now means 50 000 —
and docs/http-api.md's limits table now says of what, since "50,000 characters" read against a
dict state was exactly the ambiguity the gate resolved the wrong way.

Two deliberate choices:

  • json.dumps inline rather than importing serialize_state. laya.common imports torch at
    module level and import laya.serve must not — tests/test_lazy_import.py asserts exactly
    that, and it still passes. The two are pinned to each other by a test instead.
  • except (TypeError, ValueError, RecursionError), not except Exception. Those are what
    json.dumps raises on caller data (unserializable type, over-long integer or circular
    reference, nesting deeper than the stack). A broader catch would swallow a bug in the server as
    if it were the client's fault, which is the reasoning the neighbouring json.loads catch
    already uses.

Cost of measuring it properly

Measuring the state correctly costs more than measuring it wrongly, and on one path it cost more
than the work it protects. json.dumps runs before the gate decides, so its bound is
MAX_BODY_BYTES, not MAX_STATE_CHARS: a 2 MiB body — the largest the streaming cap admits,
and one the gate then rejects — took 5.69 ms to serialize against 3.46 ms for the str()
it replaces. That +2.24 ms is 101% of the 2.22 ms json.loads spent parsing the same body, so
the fix on its own roughly doubled the CPU an oversized rejected request costs. An earlier revision
of this description called that "not a new cost class", which was too generous.

So the gate now refuses a state it can prove is oversized without serializing it.
_provably_over_state_limit sums the raw lengths of the string values, which is a lower bound on
the JSON length: escaping never shortens a string; keys, separators and brackets only add; numbers,
booleans and nulls contribute at least one character each and are counted as zero. Every one of those
makes it an under-estimate, which is the safe direction — True means certainly over, False means
"not proven" and falls through to the exact json.dumps, so no verdict changes. The walk is
bounded to 64 values so it can never become the expensive step, and dict values are pushed with a
C-level extend rather than iterated in Python (summing key lengths in a Python loop made a
2000-key state 1.66× slower for no gain, since ignoring keys keeps the bound valid).

Best of 40 after 6 warm-up iterations, CPython 3.12 on a 10-core arm64 laptop, against upstream:

state upstream str() this PR ratio
refused, one 2 MiB string 3.461 ms 0.0005 ms 0.00×
refused, 2 MiB over 40 values 3.376 ms 0.0010 ms 0.00×
refused, 2 MiB over 60 000 values 7.131 ms 7.525 ms 1.06×
refused, 2 MiB over 200 000 values 15.148 ms 14.881 ms 0.98×
accepted, at the 50 000 ceiling 0.0798 ms 0.1338 ms 1.68×
accepted, typical ticket 0.0011 ms 0.0032 ms 2.92×

Reported in full rather than by the best row, because the probe is not a blanket win. It
eliminates the worst rejection case — the cheapest one to construct, a single large string — and
does nothing for a payload of many small values: that is not provably over inside a 64-value
budget, so it still pays a full dump at 1.06× upstream. That is tolerable only because str() walked
that shape in full too, which is why the gap there is +0.4 ms rather than +2.2 ms. And accepted
requests remain slower than upstream in ratio — 1.68× and 2.92× — which in absolute terms is
+0.054 ms and +0.002 ms, against 11.6 ms to tokenize that same state once, and a forward pass in
the hundreds of milliseconds. I am not quoting a single figure for the forward pass: router.predict
on a state at the ceiling measured 143 ms, 257 ms and 261 ms in different runs here and 1040–1301 ms
under load, because several jobs share this CPU. The 0.133 ms and 5.68 ms figures are the stable
ones — they reproduced in a separate process to three significant figures.

The refusal reports a length it measured. The probe returns its bound rather than a boolean,
because _check_request_limits formats that number into the 413. Returning cap + 1 would answer
state too large (50001 > 50000 chars) for a 60 012-character state — a count nothing measured, and
the same flaw this gate removes from the except path it replaces. The bound is string content only,
keys and structure ignored, so it understates and the message can never overstate: that state is now
answered state too large (60000 > 50000 chars), short by the 12 structural characters.

Why there is no matching upper bound, which would let an accepted state skip the dump entirely.
It is not soundly computable here, and the failure mode is the one this gate exists to stop: an upper
bound that is too low admits an oversized state. Two attempts were wrong —
6*chars + 8/node gives 198 for a 20-float state that serializes to 507, and 32/node still gives
110 for {"v": 10**500}, which serializes to 508 because one integer node is 501 characters. A sound
version has to model int digit counts, float reprs up to 23 characters and separator expansion, which
is reimplementing the encoder inside the gate whose purpose is to agree with the encoder. The lower
bound carries no such hazard, because string content only ever adds — which is why only that half
exists.

On reusing the serialized text instead. There is no seam: Router.predict → Router.route →
Agent.system_one → build_sequence each take the state object and none accepts pre-serialized
text, so encoding once would mean a new parameter on three public methods — and per AGENTS.md a
public API change has to come with tests/test_hooks_api.py. For 0.054 ms that is not a trade worth
making in a fix to a size gate. json.dumps is also faster than str() on a deeply nested state,
so the change is not uniformly a cost even before the probe.

Behaviour at the HTTP layer, before and after

Through fastapi.testclient.TestClient against create_app(router=...) with the real english
checkpoint
behind it — same script, run against this branch and against upstream/main:

request before after
{"body": '"' * 49988} (str 50 000, json 99 988) 200 413 state too large (99988 > 50000 chars)
{"body": "\u200b" * 8332} (str 50 004, json 8 344) 413 state too large (50004 > 50000 chars) 200
{"body": "a" * 49988} (serializes to exactly 50 000) 200 200
{"body": "a" * 49989} (serializes to 50 001) 413 state too large (50001 > 50000 chars) 413 state too large (50001 > 50000 chars)
"A" * 50000 (string state) 200 200
"A" * 50001 (string state) 413 state too large (50001 > 50000 chars) 413 state too large (50001 > 50000 chars)
body with a 4 301-digit integer state 400 request body must be valid JSON 400 request body must be valid JSON
_check_request_limits(10 ** 4300, …) in-process 413 state too large (50001 > 50000 chars) 400 'state' must be JSON-serializable
_check_request_limits({1, 2}, …) in-process accepted, then a 500 inference failed from serialize_state 400 'state' must be JSON-serializable

The first row is the vulnerability: a request that the server accepted and tokenized at 99 988
characters, twice the limit it documents. The second is its mirror — a 8 344-character state, a
sixth of the limit, that the server refused as too large.

What was loaded, and the end-to-end result

Real weights, not a stub:

  • checkpoint english — convaiinnovations/laya root, ModernBERT-large, 421M params — from a
    local directory, HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1, Router(models={"english": …}, max_loaded=1), config read from the checkpoint's own rl_agent_config.json.
  • the router was injected into create_app(router=…), so every case above went through the real
    POST /v1/systemone path: body cap, JSON parse, _check_request_limits, routing, tokenization,
    forward pass.
  • routing came back {"model": "english", "reason": "English Latin text"} on every answered
    case, so inference really ran on those weights.

Ordinary traffic is untouched. 8 cases — string, dict, list, a dict with mixed-type fields, a
dict whose values contain both " and ', a nested dict, a dict state serializing to exactly
50 000, and a 50 000-character string state — were run on both trees and the full response
compared: answers (choice, probabilities, confidence, answer_confidence, action), usage
and routing. 8 of 8 byte-identical, 0 mismatches. For example, the dict state returned
{"choice": "billing", "probabilities": {"billing": 0.962, "tech": 0.038}, "confidence": 0.7668}
and {"input_tokens": 110} on both.

The control that makes that zero mean something: the same harness shows the two divergent states
flipping 200 → 413 and 413 → 200 on the same run, so it does reach the changed line.

I also confirmed the claim about the old failure mode on real weights rather than assuming it:
router.predict({1, 2}, questions) raises TypeError: Object of type set is not JSON serializable, which is what _systemone_inner turns into a 500.

Tests

tests/test_serve.py — already run by ci.yml's pytest step on both the Linux and Windows lanes,
so no CI wiring is added — gains 15 checks, all through TestClient (11 for the measurement, 4 for the lower-bound probe):

  • the 99 988-character state is refused, with the count the tokenizer would have seen, and the
    router is asserted never to have been called;
  • the 8 344-character state is answered 200 and reaches predict unchanged;
  • 4 parametrized cases pin what must not change: a dict state serializing to exactly 50 000 and
    a 50 000-character string state are 200; one character over each is 413;
  • _state_length is pinned against laya.common.serialize_state itself over 9 states, including
    all three divergent ones (", the zero-width space and the non-BMP character) — this is what keeps serve.py's inline json.dumps from drifting, since it
    cannot import the function;
  • an unserializable state is a 400, asserted on the whole detail string so a character count
    cannot creep back into it;
  • and a body with a 4 301-digit integer is shown to be a 400 from the JSON parser, which is why
    the 400 above is not an HTTP-reachable case.

tests/test_example_server_limits.py's existing serve-vs-demo parity list gains the divergent
state. Its previous cases all used states whose repr and JSON happen to be the same length, so
the list could not have caught this.

Of the 9, 4 change behaviour and 5 are pins: the four parametrized cases use states whose
repr and JSON are the same length on purpose, so they are green on upstream/main too — they
exist to prove the fix does not move the limit for ordinary states — and the over-long-integer case
documents that the body parser rejects it first.

Revert check

laya/serve.py put back to upstream/main with the tests left as they are — the change is
committed, so this is git show upstream/main:laya/serve.py > laya/serve.py, run, then restore:

python -m pytest tests/test_serve.py -q
4 failed, 56 passed, 1 skipped

FAILED test_the_gate_counts_the_json_the_tokenizer_will_see
FAILED test_a_state_a_sixth_of_the_limit_is_not_refused_for_its_repr
FAILED test_the_gate_agrees_with_the_function_that_feeds_the_tokenizer
FAILED test_an_unserializable_state_is_a_400_not_a_count_nothing_measured

python tests/test_example_server_limits.py
66 passed, 1 failed
  FAIL parity with laya.serve: a state whose repr is half its JSON -- serve=accepted demo=refused

So 5 checks go red on revert (4 in the pytest suite plus the parity case), and the other 5 new
checks are the ones that must pass either way. The parity failure is the drift guard working: with
only one of the two surfaces fixed, they disagree.

A Windows-only CI failure this PR caused, and fixed

Worth recording because Linux cannot see it. The first push failed tests (windows) while all four
Linux tests (py3.10–3.13) jobs passed. pytest builds each parametrized case's node id out of the
parameter value, so "A" * MAX_STATE_CHARS produced a 50 084-character node id, and pytest
writes that id into PYTEST_CURRENT_TEST. A single Windows environment variable cannot exceed
32 767 characters, so every case in test_the_declared_limit_is_the_limit_for_ordinary_states
errored at teardown:

tests/test_serve.py::test_the_declared_limit_is_the_limit_for_ordinary_states[AAAA...]
E   ValueError: the environment variable is longer than 32767 characters
<frozen os>:685: ValueError

The four cases now carry explicit short ids — no assertion changed, and the longest node id in the
file drops from 50 084 to 112 characters. b"[" * 100000 elsewhere in this file already carried an
id for exactly this reason.

And because a Windows-only failure is a slow way to learn this a second time, the constraint is now
asserted in the test itself rather than left to the platform: each case checks
len(request.node.nodeid) < 32767, which fails on any platform. Verified by removing the ids
again — 2 of the 4 cases go red on Linux.

Gates

Run with HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 USE_TF=0 USE_TORCH=1 TOKENIZERS_PARALLELISM=false, CPython 3.12.2:

gate result
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed
python -m compileall -q laya/ tests/ clean
ci.yml script suites (all 55 in the "Routing and language-detection tests" list) all pass, 0 failed (test_mcp self-skips: mcp extra not installed here)
pytest tests/test_serve.py tests/test_router_batch.py tests/test_predict_batch.py tests/test_system_one_lang.py tests/test_audit_regressions.py tests/test_truncation_direction.py tests/test_compile.py -q 124 passed, 2 skipped
python tests/test_serve.py-lane suites of note: test_lazy_import.py all lazy-import tests passed (so import laya.serve still pulls in no torch)
python tests/test_example_server_limits.py 69 passed, 0 failed (66 before this PR)
python tests/test_email.py 120 passed, 0 failed
zensical build --strict --clean (for the docs/http-api.md line) No issues found, no griffe: lines

docs/http-api.md's 400 row is deliberately left alone: the new 400 is not reachable over HTTP
(shown above), so listing it would document a response the server cannot return, and the Limits
section's "Every one of them is a 413" stays true.

The longest Python line added is 105 characters (an f-string in the /gui handler, whose
indentation moved), against the project's 120. The one docs/http-api.md table cell is 166, which
is in keeping with the 148-, 152- and 171-character rows already in that file. examples/server.py
has one pre-existing E501 that is not in this diff and was left alone.

Independent re-verification

Re-derived from scratch rather than taken from the earlier run.

The gate is wrong in both directions — arithmetically, and then through the HTTP surface with a
stub router:

state len(str()) serialized upstream this PR
{"body": '"' * 49988} 50 000 99 988 200 accepted — nearly 2× the documented limit 413 99988 > 50000
{"body": "\u200b" * 8332} 50 004 8 344 413 50004 > 50000 — a sixth of the limit, refused 200 accepted
{"body": "x" * 1000} — 1 012 accepted accepted
{"body": "x" * 60000} — 60 012 refused refused, same message

The constraint that shapes the implementation holds. json.dumps is inlined rather than importing
serialize_state because laya.common imports torch at module level and import laya.serve must not.
Verified directly: after import laya.serve, neither torch nor laya.common is in sys.modules, and
tests/test_lazy_import.py passes.

The cost quoted in _state_length's docstring reproduces exactly (best of 200 after 20 warm-ups):

                        docstring    measured     str() claimed / measured
gate ceiling (~50 000)  0.133 ms     0.133 ms     0.080 ms / 0.080 ms
2 MiB body              5.68 ms      5.673 ms     3.46 ms / 3.456 ms

A vacuous check found and fixed, and then a comment that credited the wrong fix. The demo-server
parity loop asserted only that laya.serve and examples/server.py agree, never what they agree on.
Reverting either file alone does break it — but reverting both left all 67 checks green, because two
surfaces measuring str(state) agree perfectly on accepting a 99 988-character state. Re-measured on
the suite as it now stands, that hole is closed: reverting laya/serve.py and
examples/server.py together gives 67 passed, 2 failed of 69, the two reds being the absolute
assertions added for exactly this case. The count is the same 67 because those are the same checks
that stayed green before; what changed is that two more now do not.

Worse, they cannot diverge in the case that matters: examples/server.py binds _state_length from
laya.serve by getattr, so a regression inside that helper moves both surfaces together. Measured
by regressing _state_length to the str() measurement: 0 of the parity checks fail, 4 of the
absolute ones do.
The verdict is now asserted for each surface, and the comment says which check is
the real coverage — the previous one credited the parity case, so anyone later pruning the "redundant"
absolute assertions would have removed the only protection.

Two tests were measuring the environment rather than the change.
test_an_unserializable_state_is_a_400… and test_an_oversized_integer_state_never_reaches_the_gate
both leaned on CPython's int→str digit guard sitting at its default 4300, which an operator can switch
off. Verified: PYTHONINTMAXSTRDIGITS=0 turns both red on an otherwise untouched tree, while
_state_length stays correct in both configurations (a set is a 400 either way; 10 ** 4300 is a 400
with the guard and a legitimate 4301-character state without it). A fixture now pins the guard for those
two tests and restores it afterwards, so they pass under either setting.

Adjacent open work — checked, not duplicated

Ten open pull requests touch laya/serve.py (#281, #387, #408, #454, #638, #642, #643, #656,
#667, #670). I grepped every one of those diffs, plus #654 which touches examples/server.py, for
added or removed lines mentioning MAX_STATE_CHARS, len(str(, state_len or serialize_state:
none of them changes the measurement. #387 only reads MAX_STATE_CHARS in a new test, and
#454 only names it in comments. I then read the diffs of the three that come closest:

  • feat(serve): add GET /v1/models and normalise dict state / structured instructions to a fixed field order #667 "add GET /v1/models and normalise dict state / structured instructions to a fixed
    field order" — adds _normalise_state / _normalise_questions and calls them on the line
    before _check_request_limits. It does not touch len(str(state)) or MAX_STATE_CHARS; the
    two changes are adjacent in the function and independent in effect (_normalise_state reorders
    fields, which does not change either length).
  • fix(common): report state truncation from the token budget that applies #668 "report state truncation from the token budget that applies" — not one of the ten
    above: it touches laya/common.py, laya/agent.py and laya/onnx_agent.py rather than
    laya/serve.py. It is here because it is the other change in flight that concerns state length.
    It is about the token budget after serialization and never touches the HTTP gate.
  • fix(serve): a lone surrogate in the body was a 500 instead of a 400 #454 "a lone surrogate in the body was a 500 instead of a 400" — the nearest in spirit, a
    body-validation fix that turns a client mistake into a 400 rather than a 500. It does not
    cover this: its _has_lone_surrogate walk only inspects str values, it runs after
    _check_request_limits, and an over-long integer is not a string, so on the old code path the
    413 fired before that walk ever saw the body. The two changes stack cleanly.

#387's batch endpoint routes every state through _check_request_limits, so it inherits this fix
rather than conflicting with it. I have not merged the other branches together to test the stack;
this PR touches only the state-length line, the two tests around it and one docs cell.

Integration with the other four open PRs

Five PRs are in flight from this fork, all parented on upstream/main @ 9d95567: #687
(state-prefix tokenization), #688 (ONNX batch-dynamic export), #689 (predict_long window room), #690
(router pinning controls) and #691 (serve state-size gate). They were merged one at a time into a tree
reset to upstream/main:

merge fix/serve-state-size-gate        conflicts=0
merge fix/onnx-export-dynamic-batch    conflicts=0
merge fix/router-pinning-controls      conflicts=0
merge perf/state-prefix-tokenization   conflicts=0
merge fix/predict-long-window-room     conflicts=0
                                       -> 20 files, +2752/-139

All 20 ordered pairs were also trial-merged with git merge-tree: 0 conflicts, so no ordering
of the five produces one.

This PR shares no file with any of the other four. It touches laya/serve.py,
examples/server.py, docs/http-api.md and two suites; laya/serve.py imports nothing the other
PRs change, and import laya.serve still pulls in neither torch nor laya.common on the merged
tree.

On the merged tree: ruff and compileall clean, 61 script-style suites from ci.yml green, the
ci.yml pytest lane 139 passed / 2 skipped, and every one of the nine touched suites at the same
count it reaches on its own branch. Two differential checks cover the shared files: build_sequence
over 768 configurations, 0 differences against upstream/main, and encode_state_head over
10 485 comparisons, 0 differences — with the added-token gate forced to pass as a control, which
produces 8, so the zero is a result rather than an absence of checking.

This PR's own claim on the merged tree, through the HTTP surface with a stub router:

{"body": '"' * 49988}      (serializes to 99 988)   413  state too large (99988 > 50000 chars)
{"body": "\u200b" * 8332}  (str 50 004, json 8 344)   200  accepted
{"body": "x" * 1000}                                200  accepted
{"body": "x" * 60000}                               413  state too large (60012 > 50000 chars)

Files

 docs/http-api.md                    |   2 +-
 examples/server.py                  |  41 ++++++++++---
 laya/serve.py                       |  51 ++++++++++++++--
 tests/test_example_server_limits.py |   4 ++
 tests/test_serve.py                 | 112 ++++++++++++++++++++++++++++++++++++
 5 files changed, 198 insertions(+), 12 deletions(-)

examples/server.py is in the diff because its _check_request_limits docstring declared that it
measured the state "exactly as laya.serve measures it — len(v) for a string, len(str(v)) for a
dict or list". Fixing one surface and not the other would have left that claim false and the CI
parity check asserting nothing, so the measurement is now shared rather than restated. Happy to
split it out if you would rather keep this PR to laya/serve.py.

@keemsisi
keemsisi marked this pull request as draft September 28, 2026 14:45
`_check_request_limits` gated `MAX_STATE_CHARS` on `len(str(state))`, but what gets tokenized is
`serialize_state(state)` -- `json.dumps(state, ensure_ascii=False)` for a dict or list. The two are
different lengths, in both directions, so the documented 50000-character limit was neither a
ceiling nor a floor. Both numbers below are from requests made against the route with the english
checkpoint behind it:

- `repr` quotes a value with `'` and leaves a `"` inside it one character, where JSON escapes it
  to two. `{"body": '"' * 49988}` has a `len(str())` of exactly 50000, so it passed the gate, and
  serialized to 99988 -- the limit admitting very nearly twice what it says.
- `repr` renders a zero-width space as the six characters `\u200b` where `ensure_ascii=False`
  writes the one character it is, so `{"body": "\u200b" * 8332}` -- 8344 characters once
  serialized, a sixth of the limit -- was refused with "state too large (50004 > 50000 chars)".

The measurement moves into `_state_length`, which examples/server.py now borrows instead of
restating, so the two surfaces cannot disagree about what a state measures. docs/http-api.md's
limits table now says which text the 50,000 characters are counted on.

A state `json.dumps` cannot render is now a 400 rather than a 413 quoting a count nothing
measured: the old `except Exception` set `state_len = MAX_STATE_CHARS + 1`, so an integer of over
4300 digits -- which `str()` itself refuses, CPython's int-to-str guard -- was answered "state
too large (50001 > 50000 chars)" for a state of a few kilobytes. No HTTP request reaches that
case, because `json.loads` builds only JSON types and refuses that integer first, which
`_systemone_inner` already answers 400. In-process callers do reach it, and there a state
`json.dumps` refuses used to pass the gate on its `str()` and then fail inside `serialize_state`,
reported as a 500 "inference failed".

Cost: the dump runs before the gate decides, so its bound is MAX_BODY_BYTES rather than
MAX_STATE_CHARS. At the gate's own ceiling it is 0.133 ms against 0.080 ms for the `str()` it
replaces; on a 2 MiB body, the largest the streaming cap allows and one this then refuses, 5.68 ms
against 3.46 ms -- a request rejected either way, whose `str()` was already being walked in full.
Best of 200 after 20 warm-up iterations, CPython 3.12 on a 10-core arm64 laptop. The forward pass
for a state at the gate's ceiling is three orders of magnitude slower on the same machine, so the
serialized text is not worth threading through `predict()` to be encoded only once.

tests/test_serve.py gains 9 checks over the HTTP surface. 4 of them go red when laya/serve.py
alone is reverted; the other 5 are pins on what must not change -- a state sitting exactly on the
limit, string states, and the fact that an over-long integer never reaches the gate at all. The
examples/server.py parity list gains a state whose repr and JSON differ, which goes red if either
surface returns to measuring `str(state)`.

Review found two things in the tests, both fixed.

The demo-server parity loop asserted only that laya.serve and examples/server.py agree, never what they
agree on -- and they cannot disagree in the case that matters, because examples/server.py binds
`_state_length` from laya.serve by `getattr`, so a regression inside that helper moves both surfaces
together. Measured by regressing it to the `str()` measurement: 0 of the parity checks fail and 4 of the
absolute ones do. The verdict is now asserted for each surface, and the comment names which check is the
real coverage -- the previous one credited the parity case, so anyone later pruning the "redundant"
absolute assertions would have removed the only protection.

Two tests were measuring the environment rather than the change: both leaned on CPython's int-to-str
digit guard sitting at its default 4300, which an operator can switch off. `PYTHONINTMAXSTRDIGITS=0`
turned both red on an otherwise untouched tree while `_state_length` stayed correct in either
configuration -- a set is a 400 both ways; 10 ** 4300 is a 400 with the guard and a legitimate
4301-character state without it. A fixture pins the guard for those two and restores it afterwards.

Re-verified from scratch: the gate is wrong in both directions, arithmetically and through the HTTP
surface. `{"body": '"' * 49988}` has a len(str()) of exactly 50000 and serializes to 99988, so upstream
answers 200 -- nearly twice the documented limit -- where this refuses with 413 "99988 > 50000". And
`{"body": "\u200b" * 8342}` serializes to 8354, a sixth of the limit, where upstream refuses it with
"50064 > 50000". Ordinary states are unchanged, same message and all. `import laya.serve` still pulls in
neither torch nor laya.common, which is why json.dumps is inlined rather than importing
serialize_state. The costs quoted in `_state_length`'s docstring reproduce exactly: 0.133 ms against
0.080 ms at the gate ceiling and 5.673 ms against 3.456 ms on a 2 MiB body.
`tests (windows)` failed on this branch while all four Linux jobs passed. pytest builds each
parametrized case's node id out of the parameter value, so a 50 000-character state produces a
50 084-character node id, and pytest writes that id into the PYTEST_CURRENT_TEST environment
variable. Windows caps a single environment variable at 32 767 characters, so every case in
`test_the_declared_limit_is_the_limit_for_ordinary_states` errored at teardown with
`ValueError: the environment variable is longer than 32767 characters`. Linux has no comparable
per-variable limit, which is why only one of the five jobs saw it.

The four cases now carry explicit short ids, which changes no assertion: longest node id in the
file goes from 50 084 to 112 characters. `b"[" * 100000` in this file already carried an id for the
same reason.

Also asserts the constraint in the test itself, because a Windows-only failure is a slow way to
learn this: the case checks `len(request.node.nodeid) < 32767`, which fails on any platform.
Verified by dropping the ids again -- 2 of the 4 cases go red on Linux.

tests/test_serve.py: 60 passed, 1 skipped. The full Windows lane: 120 passed, 2 skipped.
@keemsisi
keemsisi marked this pull request as ready for review September 29, 2026 03:52
@keemsisi
keemsisi marked this pull request as draft September 29, 2026 03:54
Measuring the state properly costs more than measuring it wrongly, and on the path that matters
most it cost more than the work it protects. `json.dumps` runs before the gate decides, so its
bound is MAX_BODY_BYTES rather than MAX_STATE_CHARS: a 2 MiB body -- the largest the streaming cap
admits, and one the gate then rejects -- took 5.69 ms to serialize against 3.46 ms for the `str()`
this replaces. That +2.24 ms is 101% of the 2.22 ms `json.loads` spent parsing the same body, so
the change roughly doubled the CPU an oversized rejected request costs.

`_provably_over_state_limit` refuses such a state without serializing it. The raw lengths of the
string values are a lower bound on the JSON length: escaping never shortens, and keys, separators
and brackets only add, while numbers, booleans and nulls are counted as zero. So True means
certainly over, and False means "not proven" and falls through to the exact `json.dumps`, leaving
every verdict unchanged. The walk is bounded to 64 values, and dict values are pushed with a
C-level `extend` rather than iterated in Python -- summing key lengths in a Python loop made a
2000-key state 1.66x slower for no gain, since ignoring keys keeps the bound valid.

Measured, best of 40 after 6 warm-up iterations, against upstream's `str()`:

                                    upstream   this PR    ratio
  refused, one 2 MiB string         3.461 ms   0.0005 ms  0.00x
  refused, 2 MiB over 40 values     3.376 ms   0.0010 ms  0.00x
  refused, 2 MiB over 60k values    7.131 ms   7.525 ms   1.06x
  refused, 2 MiB over 200k values  15.148 ms  14.881 ms   0.98x
  accepted, at the 50 000 ceiling   0.0798 ms  0.1338 ms  1.68x
  accepted, typical ticket          0.0011 ms  0.0032 ms  2.92x

Reported in full rather than by the best row: the probe eliminates the worst rejection case and
does nothing for a payload of many small values, which is not provably over inside its budget and
still pays a full dump -- 1.06x upstream, which is tolerable only because `str()` walked that shape
in full too. Accepted requests remain 1.68x and 2.92x upstream in ratio, +0.054 ms and +0.002 ms in
absolute terms, against 11.6 ms to tokenize the same state once.

Three checks, each verified to go red on its own mutation: the oversized state is refused with no
call to `json.dumps` (red when the probe is removed); the probe's verdict matches the encoder's over
153 shapes straddling the cap, including the escape-heavy values this gate exists for, and under the
cap the exact length is still returned (red when the probe refuses at half the cap, and when the
early-out reports a number nothing measured); and the budget is bounded, with a 200 000-value state
falling through instead of being refused on a partial sum (red when the budget is removed).

tests/test_serve.py: 63 passed, 1 skipped. Windows pytest lane: 123 passed, 2 skipped. All 60
ci.yml script suites pass. ruff and compileall clean.
…ap + 1

The previous commit returned `MAX_STATE_CHARS + 1` when the probe refused, and
`_check_request_limits` formats that straight into the 413. So a 60 012-character state was answered
`state too large (50001 > 50000 chars)` -- a count nothing had measured, which is the same flaw this
branch removes from the `except` path it replaces, reintroduced two commits later.

`_state_length_lower_bound_over` now returns the bound itself rather than a bool. It is a measured
number and it understates -- string content only, keys and structure ignored -- so the message is a
true statement about the state and can never overstate its length. The 60 012-character state is now
answered `state too large (60000 > 50000 chars)`, short by the 12 structural characters.

A check pins it across three oversized shapes: the reported number must exceed the cap, must not be
`cap + 1`, and must be <= the true serialized length.

Also recorded here because it decided the design: a two-sided bound, where a cheap UPPER bound would
let an accepted state skip the dump entirely, is not sound to build. Two attempts failed --
6*chars + 8/node underestimates a state of 20 floats (198 against 507), and 32/node still
underestimates one big integer (110 against 508, since 10**500 is 501 characters from a single
node). A sound upper bound has to model int digit counts, float reprs and separators, which is
reimplementing the encoder inside the gate whose purpose is to agree with the encoder -- and an
upper bound that is too low ADMITS an oversized state, the exact failure this gate exists to stop.
The lower bound has no such hazard: string content only ever adds.

tests/test_serve.py: 64 passed, 1 skipped. Windows pytest lane: 124 passed, 2 skipped. All 60
ci.yml script suites pass. ruff, compileall and test_lazy_import clean. Four mutations verified red:
reporting cap + 1, disabling the probe, making it refuse at half the cap, and removing its budget.
… three wrong claims

From an adversarial review of the probe added in 3f67b7a. No HIGH findings: 20 000 fuzzed
JSON-reachable states produced no overstatement and no verdict disagreement with the encoder, and no
state was found that passes the gate and then raises in `serialize_state`. Three MEDIUM findings were
real and are fixed here.

1. The walk read `item.values()` through `isinstance`, so a `dict` subclass overriding `values()`
   broke the lower bound in the unsafe direction -- measured, a 13-character state refused as
   `60000 > 50000`, which is precisely the "never claims a length larger than the state really has"
   property the docstring promises. Cycles were affected too: `{"self": <cycle>, "big": <60k>}`
   answered 400 or 413 depending on dict insertion order. The walk now matches EXACT `dict`, `list`
   and `tuple` and hands anything else to the encoder, which decides the size or raises into the 400.
   No HTTP request could reach this -- `json.loads` builds only exact types -- but the 400 branch
   exists for in-process callers, whose docstring names a set and a circular reference.

2. Nothing covered the probe's list/tuple branch: deleting it left both suites green, because every
   check that could notice fell through to `json.dumps` and got the same verdict. A conversation LIST
   is this library's dominant state shape, so the optimisation could have been lost silently. The
   refuses-without-serializing test now runs over a dict, a bare list, a conversation list and a
   tuple. Verified: deleting the branch now turns it red, as does restoring the `isinstance` form.

3. `assert reported != cap + 1` was a latent false failure, not a guard: `{"body": "a" * 50001}`
   legitimately sums to exactly `cap + 1`. The property wanted is `cap < reported <= exact`, which
   the next assertion already makes; the fixtures now include that state and a list.

LOW findings, all corrected rather than argued:
- `_provably_over_state_limit` appeared in three places and does not exist; the symbol is
  `_state_length_lower_bound_over`. One was the docstring of the test guarding the probe.
- The budget comment claimed the walk "can never cost more than one" value. It bounds values
  *examined*, not work: `stack.extend` pushes elements uncharged, so a 2 MiB body of many small
  values costs ~0.43 ms before falling through to the dump it was going to pay anyway. Measured
  figures are now in the comment, with the note that an accepted state is capped at MAX_STATE_CHARS.
- `test_example_server_limits.py` claimed a regression fails "4" of the absolute checks. There are
  two, and reverting `_state_length` two different ways fails exactly those two.
- The 413 carries the exact length when the encoder ran and the understated bound when it did not.
  Both truthful, neither overstating, now said in the docstring.
- `examples/server.py` claimed the two surfaces "cannot disagree about what a state measures". True
  of the measurement, but the demo checks state size before the option budgets and `laya.serve`
  checks it after, so a request violating both gets a different detail from each. The ordering is
  pre-existing and unchanged; the docstring now says so.
- Two bare asserts in a file whose convention is `ok(label, cond, detail)` would have abandoned the
  ~30 checks after them instead of recording one failure; they are `ok()` calls now.
- The counting-dumps test patched an attribute on `serve_mod.json`, which IS the stdlib module, so
  it replaced `json.dumps` process-wide for the test's duration. Now scoped to the module binding.

tests/test_serve.py: 65 passed, 1 skipped. tests/test_example_server_limits.py: 71 passed, 0 failed.
Windows pytest lane: 125 passed, 2 skipped. All 60 ci.yml script suites pass. ruff clean on every
touched file (the tree's 14 pre-existing hits are identical on 9d95567), compileall clean. Re-audited
after the fixes: the int guard, a set and an arbitrary object still give 400; the subclass now
measures 13; the gate still equals `len(serialize_state(state))` for accepted states; 3 586 fuzzed
states, 0 wrong verdicts, 0 overstatements.
@NandhaKishorM
NandhaKishorM merged commit 5a49907 into NandhaKishorM:main Sep 29, 2026
22 of 23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @keemsisi! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants