Skip to content

fix(dotnet): parse and re-emit JSON without recursion, so depth cannot overflow the stack - #1346

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
thread3d:fix/dotnet-json-iterative
Oct 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
thread3d:fix/dotnet-json-iterative

Conversation

@thread3d

Copy link
Copy Markdown
Contributor

What

laya-dotnet/src/Laya/PythonJsonReader.cs parses, re-emits and walks JSON without recursion: the parser keeps open containers in a Stack<Frame>, the re-emitter a List<WriteFrame> of container enumerators, and IsAlteredByReEmission its own explicit stack. One new test pins the property on a 256 KB thread.

Why

MaxDepth mirrors CPython (2997 levels, which Python 3.12 loads) and is checked before every descent, but reaching it costs about two managed frames per JSON level. On a thread with the platform-default 1 MB stack the stack is exhausted at roughly half that depth, and .NET cannot catch a stack overflow — so a document CPython accepts did not produce a catchable error here, it aborted the process. The port's own suite showed it: running PythonJsonErrorTests killed the test host with

Stack overflow. Repeat 1156 times: at Laya.PythonJsonReader+State.Array(Int32) / State.Value(Int32)
Abort trap: 6

which is why those tests could only ever pass on a runner that happened to hand out a larger stack. A downloaded tokenizer_config.json is exactly the untrusted document this parses.

How it was verified

Built for net8.0 in a scratch copy, because the project pins SDK 10.0.100 / net10.0, which this machine does not have (local-only shims for the three .NET 9 APIs it uses; nothing shimmed is part of the patch):

dotnet build tests/Laya.Tests/Laya.Tests.csproj          # 0 warnings, 0 errors
dotnet tests/Laya.Tests/bin/Debug/net8.0/Laya.Tests.dll  # 2670 passed, 0 failed, 752 skipped
  • The whole suite now runs to completion, including the 85 PythonJsonErrorTests that used to abort it (before: exit 134).
  • All pinned CPython error texts and positions, 2997-loads / 2998-rejected, and the re-emission at 10/65/900/2990 levels pass unchanged.
  • New DeepNestingNeedsNoStackProportionalToItsDepth parses and re-emits a 2990-level document on a 256 KB thread; it aborted the host before this change and passes now, so the fix is what is being tested rather than the runner's stack.
  • A local differential check (2000 generated documents) parsed and re-emitted each document and compared the tree against System.Text.Json structurally — identical kinds, values, member order and strings. Number spelling is deliberately not compared: laya follows Python's spellings (-1.25e-7 re-emits as -1.25E-07 through Utf8JsonWriter), which is pre-existing and unchanged.

No workflow change is needed: dotnet.yml filters only the model-backed classes, so PythonJsonErrorTests is already in the lane — it just could not survive it.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (branch cut from upstream/main at 0.4.14)
  • Tests pass locally
  • Docs or examples updated when the public API changed (no public API change; XML docs explain the walks)

…t overflow the stack

MaxDepth mirrors CPython (2997 levels, which Python 3.12 loads) and is checked before
every descent, but reaching it costs about two managed frames per JSON level: on a thread
with the platform-default 1 MB stack the stack runs out at roughly half that depth, and
.NET cannot catch a stack overflow. So a document CPython accepts did not produce an error
here -- it aborted the process. The port's own suite demonstrated it: running
PythonJsonErrorTests killed the test host with "Stack overflow. Repeat N times: at
State.Array/State.Value" and exit 134, which is why those tests could only ever pass on a
runner that happened to hand out a larger stack.

Three walks recursed over the same depth, so all three now carry an explicit stack:

  * State.Value/Object/Array -> one loop with a Stack<Frame> of open containers. The depth
    limit reads the open-container count, which is the number the old `depth` parameter
    carried, so 2997 levels still load and 2998 still raise. Tokens, scalar types, error
    texts and their exact positions, last-value-wins duplicate keys and the "{} / []"
    fast paths are unchanged.
  * Write (the re-emission ParseDocument performs for tokenizer_config.json) -> a
    List<WriteFrame> of container enumerators; Utf8JsonWriter already keeps its own depth
    counter and does not recurse.
  * IsAlteredByReEmission -> the same explicit-stack walk.

Verified by building the port for net8.0 in a scratch copy (it pins SDK 10.0.100, which
this machine does not have; local-only shims for the .NET 9 APIs it uses):

  * the whole suite now runs to completion: 2670 passed, 0 failed, 752 skipped (the ONNX
    artifact classes) -- the same 85 tests that used to abort PythonJsonErrorTests included;
  * the 85 pinned Python error texts, 2997-loads / 2998-rejected and the re-emission at
    10/65/900/2990 levels all pass;
  * new test: parse + re-emit of a 2990-level document on a 256 KB thread, which aborted
    the host before this change and now passes, so the property is the fix rather than the
    runner's stack;
  * a local differential check (2000 generated documents) compared every parsed and
    re-emitted tree against System.Text.Json structurally: identical kinds, values, member
    order and strings. Number spelling is deliberately not compared -- laya follows
    Python's spellings (-1.25e-7 re-emits as -1.25E-07 through Utf8JsonWriter), which is
    pre-existing behaviour and unchanged here.
thread3d added a commit to thread3d/laya that referenced this pull request Oct 24, 2026
The fork's own release: upstream 0.4.14 plus this fork's patch set and the seventeen fixes
verified in this checkout (each proposed upstream as its own PR, NandhaKishorM#1331-NandhaKishorM#1346).

Declared in the three places tests/test_packaging.py holds equal - pyproject.toml,
laya/__init__.py and nix/package.nix - and in the routing golden meta, which records
laya.__version__ and regenerates byte-identically at this commit. README's release notes
rotate to 0.4.15.

Gates: canonical suite 89 invocations with only the documented langchain-extra failure,
ruff/compileall, packaging (173 checks), doc tables (181), env docs (67), citations (22);
TypeScript 1237 vitest tests, TS client 32, Rust 27 integration targets, .NET whole suite
2670 on a net8 retarget (the port pins SDK 10.0.100, absent here). The sdist and wheel built
from this commit install and report 0.4.15.
@NandhaKishorM
NandhaKishorM merged commit e42f5ce into NandhaKishorM:main Oct 24, 2026
46 of 50 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity
(#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343),
fail-closed input handling in evals and the integrations (#1317, #1318,
#1319, #1320, #1321, #1322, #1326, #1335), exact and faster script
detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328,
#1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344,
#1345), CI (#1323, #1347, #1348).

Verified before tagging: all 92 suites pass, 610 tests. ruff clean on
both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean.
Version agrees across pyproject.toml, laya.__version__ and nix/package.nix.
README carries 0.4.15 only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@thread3d
thread3d deleted the fix/dotnet-json-iterative branch October 28, 2026 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants