Repository navigation
fix(serve): measure the state limit on the text the tokenizer receives - #691
Merged
NandhaKishorM merged 5 commits intoSep 29, 2026
Merged
Conversation
keemsisi
marked this pull request as draft
September 28, 2026 14:45
`_check_request_limits` gated `MAX_STATE_CHARS` on `len(str(state))`, but what gets tokenized is
`serialize_state(state)` -- `json.dumps(state, ensure_ascii=False)` for a dict or list. The two are
different lengths, in both directions, so the documented 50000-character limit was neither a
ceiling nor a floor. Both numbers below are from requests made against the route with the english
checkpoint behind it:
- `repr` quotes a value with `'` and leaves a `"` inside it one character, where JSON escapes it
to two. `{"body": '"' * 49988}` has a `len(str())` of exactly 50000, so it passed the gate, and
serialized to 99988 -- the limit admitting very nearly twice what it says.
- `repr` renders a zero-width space as the six characters `\u200b` where `ensure_ascii=False`
writes the one character it is, so `{"body": "\u200b" * 8332}` -- 8344 characters once
serialized, a sixth of the limit -- was refused with "state too large (50004 > 50000 chars)".
The measurement moves into `_state_length`, which examples/server.py now borrows instead of
restating, so the two surfaces cannot disagree about what a state measures. docs/http-api.md's
limits table now says which text the 50,000 characters are counted on.
A state `json.dumps` cannot render is now a 400 rather than a 413 quoting a count nothing
measured: the old `except Exception` set `state_len = MAX_STATE_CHARS + 1`, so an integer of over
4300 digits -- which `str()` itself refuses, CPython's int-to-str guard -- was answered "state
too large (50001 > 50000 chars)" for a state of a few kilobytes. No HTTP request reaches that
case, because `json.loads` builds only JSON types and refuses that integer first, which
`_systemone_inner` already answers 400. In-process callers do reach it, and there a state
`json.dumps` refuses used to pass the gate on its `str()` and then fail inside `serialize_state`,
reported as a 500 "inference failed".
Cost: the dump runs before the gate decides, so its bound is MAX_BODY_BYTES rather than
MAX_STATE_CHARS. At the gate's own ceiling it is 0.133 ms against 0.080 ms for the `str()` it
replaces; on a 2 MiB body, the largest the streaming cap allows and one this then refuses, 5.68 ms
against 3.46 ms -- a request rejected either way, whose `str()` was already being walked in full.
Best of 200 after 20 warm-up iterations, CPython 3.12 on a 10-core arm64 laptop. The forward pass
for a state at the gate's ceiling is three orders of magnitude slower on the same machine, so the
serialized text is not worth threading through `predict()` to be encoded only once.
tests/test_serve.py gains 9 checks over the HTTP surface. 4 of them go red when laya/serve.py
alone is reverted; the other 5 are pins on what must not change -- a state sitting exactly on the
limit, string states, and the fact that an over-long integer never reaches the gate at all. The
examples/server.py parity list gains a state whose repr and JSON differ, which goes red if either
surface returns to measuring `str(state)`.
Review found two things in the tests, both fixed.
The demo-server parity loop asserted only that laya.serve and examples/server.py agree, never what they
agree on -- and they cannot disagree in the case that matters, because examples/server.py binds
`_state_length` from laya.serve by `getattr`, so a regression inside that helper moves both surfaces
together. Measured by regressing it to the `str()` measurement: 0 of the parity checks fail and 4 of the
absolute ones do. The verdict is now asserted for each surface, and the comment names which check is the
real coverage -- the previous one credited the parity case, so anyone later pruning the "redundant"
absolute assertions would have removed the only protection.
Two tests were measuring the environment rather than the change: both leaned on CPython's int-to-str
digit guard sitting at its default 4300, which an operator can switch off. `PYTHONINTMAXSTRDIGITS=0`
turned both red on an otherwise untouched tree while `_state_length` stayed correct in either
configuration -- a set is a 400 both ways; 10 ** 4300 is a 400 with the guard and a legitimate
4301-character state without it. A fixture pins the guard for those two and restores it afterwards.
Re-verified from scratch: the gate is wrong in both directions, arithmetically and through the HTTP
surface. `{"body": '"' * 49988}` has a len(str()) of exactly 50000 and serializes to 99988, so upstream
answers 200 -- nearly twice the documented limit -- where this refuses with 413 "99988 > 50000". And
`{"body": "\u200b" * 8342}` serializes to 8354, a sixth of the limit, where upstream refuses it with
"50064 > 50000". Ordinary states are unchanged, same message and all. `import laya.serve` still pulls in
neither torch nor laya.common, which is why json.dumps is inlined rather than importing
serialize_state. The costs quoted in `_state_length`'s docstring reproduce exactly: 0.133 ms against
0.080 ms at the gate ceiling and 5.673 ms against 3.456 ms on a 2 MiB body.
keemsisi
force-pushed
the
fix/serve-state-size-gate
branch
from
September 28, 2026 23:07
75f63f2 to
d8038b8
Compare
This was referenced Sep 28, 2026
Open
`tests (windows)` failed on this branch while all four Linux jobs passed. pytest builds each parametrized case's node id out of the parameter value, so a 50 000-character state produces a 50 084-character node id, and pytest writes that id into the PYTEST_CURRENT_TEST environment variable. Windows caps a single environment variable at 32 767 characters, so every case in `test_the_declared_limit_is_the_limit_for_ordinary_states` errored at teardown with `ValueError: the environment variable is longer than 32767 characters`. Linux has no comparable per-variable limit, which is why only one of the five jobs saw it. The four cases now carry explicit short ids, which changes no assertion: longest node id in the file goes from 50 084 to 112 characters. `b"[" * 100000` in this file already carried an id for the same reason. Also asserts the constraint in the test itself, because a Windows-only failure is a slow way to learn this: the case checks `len(request.node.nodeid) < 32767`, which fails on any platform. Verified by dropping the ids again -- 2 of the 4 cases go red on Linux. tests/test_serve.py: 60 passed, 1 skipped. The full Windows lane: 120 passed, 2 skipped.
keemsisi
marked this pull request as ready for review
September 29, 2026 03:52
keemsisi
marked this pull request as draft
September 29, 2026 03:54
Measuring the state properly costs more than measuring it wrongly, and on the path that matters
most it cost more than the work it protects. `json.dumps` runs before the gate decides, so its
bound is MAX_BODY_BYTES rather than MAX_STATE_CHARS: a 2 MiB body -- the largest the streaming cap
admits, and one the gate then rejects -- took 5.69 ms to serialize against 3.46 ms for the `str()`
this replaces. That +2.24 ms is 101% of the 2.22 ms `json.loads` spent parsing the same body, so
the change roughly doubled the CPU an oversized rejected request costs.
`_provably_over_state_limit` refuses such a state without serializing it. The raw lengths of the
string values are a lower bound on the JSON length: escaping never shortens, and keys, separators
and brackets only add, while numbers, booleans and nulls are counted as zero. So True means
certainly over, and False means "not proven" and falls through to the exact `json.dumps`, leaving
every verdict unchanged. The walk is bounded to 64 values, and dict values are pushed with a
C-level `extend` rather than iterated in Python -- summing key lengths in a Python loop made a
2000-key state 1.66x slower for no gain, since ignoring keys keeps the bound valid.
Measured, best of 40 after 6 warm-up iterations, against upstream's `str()`:
upstream this PR ratio
refused, one 2 MiB string 3.461 ms 0.0005 ms 0.00x
refused, 2 MiB over 40 values 3.376 ms 0.0010 ms 0.00x
refused, 2 MiB over 60k values 7.131 ms 7.525 ms 1.06x
refused, 2 MiB over 200k values 15.148 ms 14.881 ms 0.98x
accepted, at the 50 000 ceiling 0.0798 ms 0.1338 ms 1.68x
accepted, typical ticket 0.0011 ms 0.0032 ms 2.92x
Reported in full rather than by the best row: the probe eliminates the worst rejection case and
does nothing for a payload of many small values, which is not provably over inside its budget and
still pays a full dump -- 1.06x upstream, which is tolerable only because `str()` walked that shape
in full too. Accepted requests remain 1.68x and 2.92x upstream in ratio, +0.054 ms and +0.002 ms in
absolute terms, against 11.6 ms to tokenize the same state once.
Three checks, each verified to go red on its own mutation: the oversized state is refused with no
call to `json.dumps` (red when the probe is removed); the probe's verdict matches the encoder's over
153 shapes straddling the cap, including the escape-heavy values this gate exists for, and under the
cap the exact length is still returned (red when the probe refuses at half the cap, and when the
early-out reports a number nothing measured); and the budget is bounded, with a 200 000-value state
falling through instead of being refused on a partial sum (red when the budget is removed).
tests/test_serve.py: 63 passed, 1 skipped. Windows pytest lane: 123 passed, 2 skipped. All 60
ci.yml script suites pass. ruff and compileall clean.
…ap + 1 The previous commit returned `MAX_STATE_CHARS + 1` when the probe refused, and `_check_request_limits` formats that straight into the 413. So a 60 012-character state was answered `state too large (50001 > 50000 chars)` -- a count nothing had measured, which is the same flaw this branch removes from the `except` path it replaces, reintroduced two commits later. `_state_length_lower_bound_over` now returns the bound itself rather than a bool. It is a measured number and it understates -- string content only, keys and structure ignored -- so the message is a true statement about the state and can never overstate its length. The 60 012-character state is now answered `state too large (60000 > 50000 chars)`, short by the 12 structural characters. A check pins it across three oversized shapes: the reported number must exceed the cap, must not be `cap + 1`, and must be <= the true serialized length. Also recorded here because it decided the design: a two-sided bound, where a cheap UPPER bound would let an accepted state skip the dump entirely, is not sound to build. Two attempts failed -- 6*chars + 8/node underestimates a state of 20 floats (198 against 507), and 32/node still underestimates one big integer (110 against 508, since 10**500 is 501 characters from a single node). A sound upper bound has to model int digit counts, float reprs and separators, which is reimplementing the encoder inside the gate whose purpose is to agree with the encoder -- and an upper bound that is too low ADMITS an oversized state, the exact failure this gate exists to stop. The lower bound has no such hazard: string content only ever adds. tests/test_serve.py: 64 passed, 1 skipped. Windows pytest lane: 124 passed, 2 skipped. All 60 ci.yml script suites pass. ruff, compileall and test_lazy_import clean. Four mutations verified red: reporting cap + 1, disabling the probe, making it refuse at half the cap, and removing its budget.
4 tasks done
This was referenced Sep 29, 2026
keemsisi
marked this pull request as ready for review
September 29, 2026 12:17
… three wrong claims From an adversarial review of the probe added in 3f67b7a. No HIGH findings: 20 000 fuzzed JSON-reachable states produced no overstatement and no verdict disagreement with the encoder, and no state was found that passes the gate and then raises in `serialize_state`. Three MEDIUM findings were real and are fixed here. 1. The walk read `item.values()` through `isinstance`, so a `dict` subclass overriding `values()` broke the lower bound in the unsafe direction -- measured, a 13-character state refused as `60000 > 50000`, which is precisely the "never claims a length larger than the state really has" property the docstring promises. Cycles were affected too: `{"self": <cycle>, "big": <60k>}` answered 400 or 413 depending on dict insertion order. The walk now matches EXACT `dict`, `list` and `tuple` and hands anything else to the encoder, which decides the size or raises into the 400. No HTTP request could reach this -- `json.loads` builds only exact types -- but the 400 branch exists for in-process callers, whose docstring names a set and a circular reference. 2. Nothing covered the probe's list/tuple branch: deleting it left both suites green, because every check that could notice fell through to `json.dumps` and got the same verdict. A conversation LIST is this library's dominant state shape, so the optimisation could have been lost silently. The refuses-without-serializing test now runs over a dict, a bare list, a conversation list and a tuple. Verified: deleting the branch now turns it red, as does restoring the `isinstance` form. 3. `assert reported != cap + 1` was a latent false failure, not a guard: `{"body": "a" * 50001}` legitimately sums to exactly `cap + 1`. The property wanted is `cap < reported <= exact`, which the next assertion already makes; the fixtures now include that state and a list. LOW findings, all corrected rather than argued: - `_provably_over_state_limit` appeared in three places and does not exist; the symbol is `_state_length_lower_bound_over`. One was the docstring of the test guarding the probe. - The budget comment claimed the walk "can never cost more than one" value. It bounds values *examined*, not work: `stack.extend` pushes elements uncharged, so a 2 MiB body of many small values costs ~0.43 ms before falling through to the dump it was going to pay anyway. Measured figures are now in the comment, with the note that an accepted state is capped at MAX_STATE_CHARS. - `test_example_server_limits.py` claimed a regression fails "4" of the absolute checks. There are two, and reverting `_state_length` two different ways fails exactly those two. - The 413 carries the exact length when the encoder ran and the understated bound when it did not. Both truthful, neither overstating, now said in the docstring. - `examples/server.py` claimed the two surfaces "cannot disagree about what a state measures". True of the measurement, but the demo checks state size before the option budgets and `laya.serve` checks it after, so a request violating both gets a different detail from each. The ordering is pre-existing and unchanged; the docstring now says so. - Two bare asserts in a file whose convention is `ok(label, cond, detail)` would have abandoned the ~30 checks after them instead of recording one failure; they are `ok()` calls now. - The counting-dumps test patched an attribute on `serve_mod.json`, which IS the stdlib module, so it replaced `json.dumps` process-wide for the test's duration. Now scoped to the module binding. tests/test_serve.py: 65 passed, 1 skipped. tests/test_example_server_limits.py: 71 passed, 0 failed. Windows pytest lane: 125 passed, 2 skipped. All 60 ci.yml script suites pass. ruff clean on every touched file (the tree's 14 pre-existing hits are identical on 9d95567), compileall clean. Re-audited after the fixes: the int guard, a set and an arbitrary object still give 400; the subclass now measures 13; the gate still equals `len(serialize_state(state))` for accepted states; 3 586 fuzzed states, 0 wrong verdicts, 0 overstatements.
4 tasks done
Owner
|
Passed checks and merged. Thank you @keemsisi! It ships in 0.3.22. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In short:
laya/serve.py'sMAX_STATE_CHARS = 50000gate measureslen(str(state)), but thetext that reaches the tokenizer is
serialize_state(state)—json.dumps(state, ensure_ascii=False)for a dict or list. Those are different lengths in both directions, so overHTTP the declared 50 000-character limit admitted a state that tokenized as 99 988 characters,
and refused a legitimate state that tokenized as 8 344. The fix measures the serialized form,
which costs 0.133 ms at the largest state the gate admits — three orders of magnitude under
the forward pass that state then gets. The same lines also answered
413 "state too large (50001 > 50000 chars)"for a state nothing had measured; that becomes a400.This is an observable behaviour change, not only a hardening. A state that serializes above
50 000 characters but whose
reprwas under it used to be answered200and is now413; a statewhose
reprwas over but which serializes under is now answered200where it used to be413.Nothing else about the route moves, and the docs table now states the contract explicitly.
The bug
laya/serve.py:249:laya/common.py:build_sequencetokenizesserialize_state(state). For a string statelen(state)is alreadyexact, so this is only about dicts and lists — and for those,
reprand JSON disagree.What a request can actually carry
Every key in a state that arrives over HTTP is a
str, becausejson.loadsmakes it one, so thedivergence has to come from the values. Measured on a
{"a": <char> * 1000}state:len(str)len(json)x"\x00'reprpicks'quotes for a value that has no apostrophe in it, so a"inside stays onecharacter; JSON escapes it to
\"and makes it two. Going the other way,represcapes azero-width space to the six characters
\u200band a non-BMP character to the ten characters\U000e0001, whileensure_ascii=Falsewrites the one character each of them is. So the twomeasurements diverge by up to 2x upward and 10x downward on states a client can send.
Both directions are reachable with a body far under the existing 2 MiB
MAX_BODY_BYTEScap:Those two bodies are 50 032 and 96 037 bytes as a default JSON encoder writes them
(
ensure_ascii=True, so the zero-width space goes out as the six bytes\u200band U+E0001 as asurrogate pair) — both far inside the 2 MiB cap. The non-BMP case is the worst false refusal of the
four; I checked that
json.loadsreally does hand back U+E0001 from that pair, that the fixed gatemeasures it as 8 009 exactly as
serialize_statedoes, and that it now answers 200.Non-string keys, for callers that are not the HTTP route
reprleaves a non-string key bare where JSON must quote it, so a dict keyed on integersdiverges by two characters per key. This cannot arrive over HTTP —
json.loadsreturns stringkeys — but
_check_request_limitsis also whatexamples/server.pymirrors, and an in-processcaller can build such a state. Measured on
{i: 0 for i in range(n)}:len(str)len(json)At 5 601 keys that state's
str()is 49 299 characters, which passes the 50 000 gate, and itsserialization is 60 501 — 11 202 characters more than the gate measured, and 10 501 past the
declared limit.
The
except Exceptionstr()on an integer of more than 4 300 digits raisesValueError(CPython's own int-to-strguard), so this answered
413 "state too large (50001 > 50000 chars)"— a count nothing hadmeasured, for a state of a few kilobytes. It is not HTTP-reachable:
json.loadsrefuses thatinteger itself, and
_systemone_inneralready answers that400 "request body must be valid JSON"(verified, below). It is in-process callers who reach it, and for them the branch wasdoing real harm in the other direction:
json.dumpsraisesTypeErrorwherestr()succeeds, soa state
serialize_statecannot render — aset, adatetime— passed the gate on itsrepr()and failed later insideserialize_state, which_systemone_innerreports as a500 "inference failed".Was the old behaviour deliberate? Three reasons it is not
Worth settling, because one of the four cases makes the gate stricter and one makes it
looser, and a reviewer is right to ask whether the strict one was the intent. It was not, on the
evidence in the repository rather than on my reading of it:
1. The gate states its own purpose, and it is not "bound the repr". The comment above the
constants:
What is tokenized is
serialize_state(state). The gate measuresstr(state). Those are differentstrings for every dict and list, so the guardrail is not measuring the quantity it exists to bound.
2. The line's own shape is an approximation, not a policy.
The string branch is exact —
serialize_statereturns a string unchanged, solen(state)isprecisely what the tokenizer receives. The container branch reaches for
str(). A deliberatedecision to bound reprs would not have got the string case exactly right and then approximated the
other one; this is the shape of "close enough for dicts".
3. No existing test distinguishes the two measurements. The only dict case upstream asserts is
tests/test_example_server_limits.py:144:{"body": "A" * (MAX_STATE_CHARS + 1)} -> 413Every character is
A, wherereprand JSON agree exactly at one character each — so thatassertion passes under either measurement and pins neither. If repr-length were the intended
contract, nothing in the suite says so.
And the argument that settles it: a deliberate cheap upper bound would be conservative. This one
is not. It over-counts a zero-width space by 6x and under-counts a double quote by 2x, so 99 988
characters of real tokenizer work pass a 50 000-character cap. A proxy that is an over-estimate for
some inputs and an under-estimate for others is not a coherent limit under any reading — it is
simply the wrong string. The provenance is consistent with that: the line arrived in
71d235f,"serve: timing-safe auth, request limits, safe errors, validated port (#250)", a hardening change
whose goal was bounding request work.
So this PR does not relax the gate. It refuses strictly more of what can actually hurt the
worker — the
"body goes from 200 to 413 — and stops refusing payloads that never could, like thezero-width-space body at a sixth of the limit. The 413-to-200 direction costs no protection, because
a
repris never what reaches the tokenizer.The fix
The measurement moves into
_state_length(), which measures what will be tokenized and raises400 "'state' must be JSON-serializable"for a state that has no such text.examples/server.pyborrows that helper through the same
getattr(_laya_serve, ...)shape it already uses for thebounds, instead of restating the measurement — its docstring claimed it measured "exactly as
laya.serve measures it —
len(v)for a string,len(str(v))for a dict or list", which is theline being fixed.
Two consequences on the demo side that come with it: its fallback (for an installed laya that
predates
_state_length) raises the same 400 rather than letting a bareTypeErrorout of_check_request_limits, which/predictcalls outside its owntry; andPOST /gui's handler nowbranches on the status, because a 400 rendered under its "Request too large" heading with a
character-limit hint would describe the wrong problem.
MAX_STATE_CHARSkeeps its documented value of 50 000. The point is that it now means 50 000 —and
docs/http-api.md's limits table now says of what, since "50,000 characters" read against adict state was exactly the ambiguity the gate resolved the wrong way.
Two deliberate choices:
json.dumpsinline rather than importingserialize_state.laya.commonimports torch atmodule level and
import laya.servemust not —tests/test_lazy_import.pyasserts exactlythat, and it still passes. The two are pinned to each other by a test instead.
except (TypeError, ValueError, RecursionError), notexcept Exception. Those are whatjson.dumpsraises on caller data (unserializable type, over-long integer or circularreference, nesting deeper than the stack). A broader catch would swallow a bug in the server as
if it were the client's fault, which is the reasoning the neighbouring
json.loadscatchalready uses.
Cost of measuring it properly
Measuring the state correctly costs more than measuring it wrongly, and on one path it cost more
than the work it protects.
json.dumpsruns before the gate decides, so its bound isMAX_BODY_BYTES, notMAX_STATE_CHARS: a 2 MiB body — the largest the streaming cap admits,and one the gate then rejects — took 5.69 ms to serialize against 3.46 ms for the
str()it replaces. That
+2.24 msis 101% of the 2.22 msjson.loadsspent parsing the same body, sothe fix on its own roughly doubled the CPU an oversized rejected request costs. An earlier revision
of this description called that "not a new cost class", which was too generous.
So the gate now refuses a state it can prove is oversized without serializing it.
_provably_over_state_limitsums the raw lengths of the string values, which is a lower bound onthe JSON length: escaping never shortens a string; keys, separators and brackets only add; numbers,
booleans and nulls contribute at least one character each and are counted as zero. Every one of those
makes it an under-estimate, which is the safe direction —
Truemeans certainly over,Falsemeans"not proven" and falls through to the exact
json.dumps, so no verdict changes. The walk isbounded to 64 values so it can never become the expensive step, and dict values are pushed with a
C-level
extendrather than iterated in Python (summing key lengths in a Python loop made a2000-key state 1.66× slower for no gain, since ignoring keys keeps the bound valid).
Best of 40 after 6 warm-up iterations, CPython 3.12 on a 10-core arm64 laptop, against upstream:
str()Reported in full rather than by the best row, because the probe is not a blanket win. It
eliminates the worst rejection case — the cheapest one to construct, a single large string — and
does nothing for a payload of many small values: that is not provably over inside a 64-value
budget, so it still pays a full dump at 1.06× upstream. That is tolerable only because
str()walkedthat shape in full too, which is why the gap there is +0.4 ms rather than +2.2 ms. And accepted
requests remain slower than upstream in ratio — 1.68× and 2.92× — which in absolute terms is
+0.054 ms and +0.002 ms, against 11.6 ms to tokenize that same state once, and a forward pass in
the hundreds of milliseconds. I am not quoting a single figure for the forward pass:
router.predicton a state at the ceiling measured 143 ms, 257 ms and 261 ms in different runs here and 1040–1301 ms
under load, because several jobs share this CPU. The 0.133 ms and 5.68 ms figures are the stable
ones — they reproduced in a separate process to three significant figures.
The refusal reports a length it measured. The probe returns its bound rather than a boolean,
because
_check_request_limitsformats that number into the 413. Returningcap + 1would answerstate too large (50001 > 50000 chars)for a 60 012-character state — a count nothing measured, andthe same flaw this gate removes from the
exceptpath it replaces. The bound is string content only,keys and structure ignored, so it understates and the message can never overstate: that state is now
answered
state too large (60000 > 50000 chars), short by the 12 structural characters.Why there is no matching upper bound, which would let an accepted state skip the dump entirely.
It is not soundly computable here, and the failure mode is the one this gate exists to stop: an upper
bound that is too low admits an oversized state. Two attempts were wrong —
6*chars + 8/nodegives 198 for a 20-float state that serializes to 507, and32/nodestill gives110 for
{"v": 10**500}, which serializes to 508 because one integer node is 501 characters. A soundversion has to model int digit counts, float reprs up to 23 characters and separator expansion, which
is reimplementing the encoder inside the gate whose purpose is to agree with the encoder. The lower
bound carries no such hazard, because string content only ever adds — which is why only that half
exists.
On reusing the serialized text instead. There is no seam:
Router.predict→Router.route→Agent.system_one→build_sequenceeach take the state object and none accepts pre-serializedtext, so encoding once would mean a new parameter on three public methods — and per
AGENTS.mdapublic API change has to come with
tests/test_hooks_api.py. For 0.054 ms that is not a trade worthmaking in a fix to a size gate.
json.dumpsis also faster thanstr()on a deeply nested state,so the change is not uniformly a cost even before the probe.
Behaviour at the HTTP layer, before and after
Through
fastapi.testclient.TestClientagainstcreate_app(router=...)with the real englishcheckpoint behind it — same script, run against this branch and against
upstream/main:{"body": '"' * 49988}(str 50 000, json 99 988)state too large (99988 > 50000 chars){"body": "\u200b" * 8332}(str 50 004, json 8 344)state too large (50004 > 50000 chars){"body": "a" * 49988}(serializes to exactly 50 000){"body": "a" * 49989}(serializes to 50 001)state too large (50001 > 50000 chars)state too large (50001 > 50000 chars)"A" * 50000(string state)"A" * 50001(string state)state too large (50001 > 50000 chars)state too large (50001 > 50000 chars)request body must be valid JSONrequest body must be valid JSON_check_request_limits(10 ** 4300, …)in-processstate too large (50001 > 50000 chars)'state' must be JSON-serializable_check_request_limits({1, 2}, …)in-processinference failedfromserialize_state'state' must be JSON-serializableThe first row is the vulnerability: a request that the server accepted and tokenized at 99 988
characters, twice the limit it documents. The second is its mirror — a 8 344-character state, a
sixth of the limit, that the server refused as too large.
What was loaded, and the end-to-end result
Real weights, not a stub:
english—convaiinnovations/layaroot, ModernBERT-large, 421M params — from alocal directory,
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1,Router(models={"english": …}, max_loaded=1), config read from the checkpoint's ownrl_agent_config.json.create_app(router=…), so every case above went through the realPOST /v1/systemonepath: body cap, JSON parse,_check_request_limits, routing, tokenization,forward pass.
routingcame back{"model": "english", "reason": "English Latin text"}on every answeredcase, so inference really ran on those weights.
Ordinary traffic is untouched. 8 cases — string, dict, list, a dict with mixed-type fields, a
dict whose values contain both
"and', a nested dict, a dict state serializing to exactly50 000, and a 50 000-character string state — were run on both trees and the full response
compared:
answers(choice, probabilities, confidence,answer_confidence,action),usageand
routing. 8 of 8 byte-identical, 0 mismatches. For example, the dict state returned{"choice": "billing", "probabilities": {"billing": 0.962, "tech": 0.038}, "confidence": 0.7668}and
{"input_tokens": 110}on both.The control that makes that zero mean something: the same harness shows the two divergent states
flipping 200 → 413 and 413 → 200 on the same run, so it does reach the changed line.
I also confirmed the claim about the old failure mode on real weights rather than assuming it:
router.predict({1, 2}, questions)raisesTypeError: Object of type set is not JSON serializable, which is what_systemone_innerturns into a 500.Tests
tests/test_serve.py— already run byci.yml's pytest step on both the Linux and Windows lanes,so no CI wiring is added — gains 15 checks, all through
TestClient(11 for the measurement, 4 for the lower-bound probe):router is asserted never to have been called;
predictunchanged;a 50 000-character string state are 200; one character over each is 413;
_state_lengthis pinned againstlaya.common.serialize_stateitself over 9 states, includingall three divergent ones (
", the zero-width space and the non-BMP character) — this is what keeps serve.py's inlinejson.dumpsfrom drifting, since itcannot import the function;
cannot creep back into it;
the 400 above is not an HTTP-reachable case.
tests/test_example_server_limits.py's existing serve-vs-demo parity list gains the divergentstate. Its previous cases all used states whose
reprand JSON happen to be the same length, sothe list could not have caught this.
Of the 9, 4 change behaviour and 5 are pins: the four parametrized cases use states whose
reprand JSON are the same length on purpose, so they are green onupstream/maintoo — theyexist to prove the fix does not move the limit for ordinary states — and the over-long-integer case
documents that the body parser rejects it first.
Revert check
laya/serve.pyput back toupstream/mainwith the tests left as they are — the change iscommitted, so this is
git show upstream/main:laya/serve.py > laya/serve.py, run, then restore:So 5 checks go red on revert (4 in the pytest suite plus the parity case), and the other 5 new
checks are the ones that must pass either way. The parity failure is the drift guard working: with
only one of the two surfaces fixed, they disagree.
A Windows-only CI failure this PR caused, and fixed
Worth recording because Linux cannot see it. The first push failed
tests (windows)while all fourLinux
tests (py3.10–3.13)jobs passed. pytest builds each parametrized case's node id out of theparameter value, so
"A" * MAX_STATE_CHARSproduced a 50 084-character node id, and pytestwrites that id into
PYTEST_CURRENT_TEST. A single Windows environment variable cannot exceed32 767 characters, so every case in
test_the_declared_limit_is_the_limit_for_ordinary_stateserrored at teardown:
The four cases now carry explicit short ids — no assertion changed, and the longest node id in the
file drops from 50 084 to 112 characters.
b"[" * 100000elsewhere in this file already carried anid for exactly this reason.
And because a Windows-only failure is a slow way to learn this a second time, the constraint is now
asserted in the test itself rather than left to the platform: each case checks
len(request.node.nodeid) < 32767, which fails on any platform. Verified by removing the idsagain — 2 of the 4 cases go red on Linux.
Gates
Run with
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 USE_TF=0 USE_TORCH=1 TOKENIZERS_PARALLELISM=false, CPython 3.12.2:ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120python -m compileall -q laya/ tests/ci.ymlscript suites (all 55 in the "Routing and language-detection tests" list)test_mcpself-skips:mcpextra not installed here)pytest tests/test_serve.py tests/test_router_batch.py tests/test_predict_batch.py tests/test_system_one_lang.py tests/test_audit_regressions.py tests/test_truncation_direction.py tests/test_compile.py -qpython tests/test_serve.py-lane suites of note:test_lazy_import.pyimport laya.servestill pulls in no torch)python tests/test_example_server_limits.pypython tests/test_email.pyzensical build --strict --clean(for thedocs/http-api.mdline)No issues found, nogriffe:linesdocs/http-api.md's400row is deliberately left alone: the new 400 is not reachable over HTTP(shown above), so listing it would document a response the server cannot return, and the Limits
section's "Every one of them is a
413" stays true.The longest Python line added is 105 characters (an f-string in the
/guihandler, whoseindentation moved), against the project's 120. The one
docs/http-api.mdtable cell is 166, whichis in keeping with the 148-, 152- and 171-character rows already in that file.
examples/server.pyhas one pre-existing
E501that is not in this diff and was left alone.Independent re-verification
Re-derived from scratch rather than taken from the earlier run.
The gate is wrong in both directions — arithmetically, and then through the HTTP surface with a
stub router:
len(str()){"body": '"' * 49988}99988 > 50000{"body": "\u200b" * 8332}50004 > 50000— a sixth of the limit, refused{"body": "x" * 1000}{"body": "x" * 60000}The constraint that shapes the implementation holds.
json.dumpsis inlined rather than importingserialize_statebecauselaya.commonimports torch at module level andimport laya.servemust not.Verified directly: after
import laya.serve, neithertorchnorlaya.commonis insys.modules, andtests/test_lazy_import.pypasses.The cost quoted in
_state_length's docstring reproduces exactly (best of 200 after 20 warm-ups):A vacuous check found and fixed, and then a comment that credited the wrong fix. The demo-server
parity loop asserted only that
laya.serveandexamples/server.pyagree, never what they agree on.Reverting either file alone does break it — but reverting both left all 67 checks green, because two
surfaces measuring
str(state)agree perfectly on accepting a 99 988-character state. Re-measured onthe suite as it now stands, that hole is closed: reverting
laya/serve.pyandexamples/server.pytogether gives 67 passed, 2 failed of 69, the two reds being the absoluteassertions added for exactly this case. The count is the same 67 because those are the same checks
that stayed green before; what changed is that two more now do not.
Worse, they cannot diverge in the case that matters:
examples/server.pybinds_state_lengthfromlaya.servebygetattr, so a regression inside that helper moves both surfaces together. Measuredby regressing
_state_lengthto thestr()measurement: 0 of the parity checks fail, 4 of theabsolute ones do. The verdict is now asserted for each surface, and the comment says which check is
the real coverage — the previous one credited the parity case, so anyone later pruning the "redundant"
absolute assertions would have removed the only protection.
Two tests were measuring the environment rather than the change.
test_an_unserializable_state_is_a_400…andtest_an_oversized_integer_state_never_reaches_the_gateboth leaned on CPython's int→str digit guard sitting at its default 4300, which an operator can switch
off. Verified:
PYTHONINTMAXSTRDIGITS=0turns both red on an otherwise untouched tree, while_state_lengthstays correct in both configurations (asetis a 400 either way;10 ** 4300is a 400with the guard and a legitimate 4301-character state without it). A fixture now pins the guard for those
two tests and restores it afterwards, so they pass under either setting.
Adjacent open work — checked, not duplicated
Ten open pull requests touch
laya/serve.py(#281, #387, #408, #454, #638, #642, #643, #656,#667, #670). I grepped every one of those diffs, plus #654 which touches
examples/server.py, foradded or removed lines mentioning
MAX_STATE_CHARS,len(str(,state_lenorserialize_state:none of them changes the measurement. #387 only reads
MAX_STATE_CHARSin a new test, and#454 only names it in comments. I then read the diffs of the three that come closest:
GET /v1/modelsand normalise dict state / structured instructions to a fixedfield order" — adds
_normalise_state/_normalise_questionsand calls them on the linebefore
_check_request_limits. It does not touchlen(str(state))orMAX_STATE_CHARS; thetwo changes are adjacent in the function and independent in effect (
_normalise_statereordersfields, which does not change either length).
above: it touches
laya/common.py,laya/agent.pyandlaya/onnx_agent.pyrather thanlaya/serve.py. It is here because it is the other change in flight that concerns state length.It is about the token budget after serialization and never touches the HTTP gate.
body-validation fix that turns a client mistake into a 400 rather than a 500. It does not
cover this: its
_has_lone_surrogatewalk only inspectsstrvalues, it runs after_check_request_limits, and an over-long integer is not a string, so on the old code path the413 fired before that walk ever saw the body. The two changes stack cleanly.
#387's batch endpoint routes every state through
_check_request_limits, so it inherits this fixrather than conflicting with it. I have not merged the other branches together to test the stack;
this PR touches only the state-length line, the two tests around it and one docs cell.
Integration with the other four open PRs
Five PRs are in flight from this fork, all parented on
upstream/main@9d95567: #687(state-prefix tokenization), #688 (ONNX batch-dynamic export), #689 (predict_long window room), #690
(router pinning controls) and #691 (serve state-size gate). They were merged one at a time into a tree
reset to
upstream/main:All 20 ordered pairs were also trial-merged with
git merge-tree: 0 conflicts, so no orderingof the five produces one.
This PR shares no file with any of the other four. It touches
laya/serve.py,examples/server.py,docs/http-api.mdand two suites;laya/serve.pyimports nothing the otherPRs change, and
import laya.servestill pulls in neithertorchnorlaya.commonon the mergedtree.
On the merged tree:
ruffandcompileallclean, 61 script-style suites fromci.ymlgreen, theci.ymlpytest lane 139 passed / 2 skipped, and every one of the nine touched suites at the samecount it reaches on its own branch. Two differential checks cover the shared files:
build_sequenceover 768 configurations, 0 differences against
upstream/main, andencode_state_headover10 485 comparisons, 0 differences — with the added-token gate forced to pass as a control, which
produces 8, so the zero is a result rather than an absence of checking.
This PR's own claim on the merged tree, through the HTTP surface with a stub router:
Files
examples/server.pyis in the diff because its_check_request_limitsdocstring declared that itmeasured the state "exactly as laya.serve measures it —
len(v)for a string,len(str(v))for adict or list". Fixing one surface and not the other would have left that claim false and the CI
parity check asserting nothing, so the measurement is now shared rather than restated. Happy to
split it out if you would rather keep this PR to
laya/serve.py.