Repository navigation
fix(evals): a gate policy nested too deep for json is a usage error, not a crash - #1318
Merged
NandhaKishorM merged 3 commits intoOct 24, 2026
Merged
Conversation
…st that it is a usage error
…the flag The mutation harness for this change found that dropping `OSError` from the `except` tuple still exited 2 -- `main` has its own `FileNotFoundError` arm -- so nothing in the suite proved the loader catches it. It is worth pinning because the two messages differ where it matters: the loader says which flag pointed at the missing file, and `main`'s fallback names only the path.
NandhaKishorM
added a commit
that referenced
this pull request
Oct 24, 2026
33 PRs. Server hardening (#1331, #1332, #1333), TypeScript parity (#1330, #1336, #1337, #1338, #1339, #1340, #1341, #1342, #1343), fail-closed input handling in evals and the integrations (#1317, #1318, #1319, #1320, #1321, #1322, #1326, #1335), exact and faster script detection (#1334, #1349), JVM and .NET resource handling (#1327, #1328, #1329, #1346), port goldens and clipping (#1324 excluded, #1325, #1344, #1345), CI (#1323, #1347, #1348). Verified before tagging: all 92 suites pass, 610 tests. ruff clean on both lists. laya-ts 1247 tests across 44 files, tsc --noEmit clean. Version agrees across pyproject.toml, laya.__version__ and nix/package.nix. README carries 0.4.15 only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya/_eval_policy.py::load_policyread the caller's gate policy file with:RecursionErroris now in that tuple, with a docstring naming the exit-code rule it protects. Nosignature change, no new error type, no new message format -- a too-deep policy now produces the same
invalid gate policy '<path>': <reason>refusal the other three shapes already produced.+10/-2inlaya/_eval_policy.py(one word in the tuple, plus the docstring) and+20/-2intests/test_evals.pyacross three commits (fix(evals):, then twotest(evals):).git diff --numstat 0a05a450 HEADis exactly those two files.load_policyhas two production call sites --laya/evals_cli.pyrun --gate-policyandcompare --gate-policy-- and both are reached before a checkpoint loads, so the reachable surface isthose two flags. Grep of
laya/,tests/anddocs/forload_policyoutside this file returns thedefinition, those two CLI sites and the tests.
Why
CPython's
jsondoes not report a nesting depth it will not walk as aValueError. It raisesRecursionError-- aRuntimeError, so it passes straight through a handler written forJSONDecodeError. The caller's file is unreadable, the CLI's own usage-error handler never sees it,and
laya-evalsexits 1, the code it keeps for "a threshold or a baseline tolerance failed".Measured by running the CLI in a child process, printing which
laya/evals_cli.pyit imported beforeanswering (
laya-bench/probe_evals_policy_depth.py; base witnesslaya-bench/probe_evals_policy_depth.BASE.log14:23Z against a pristine0a05a450worktree, branchwitness
laya-bench/probe_evals_policy_depth.BRANCH.log14:23Z against this tree). The policy file is"[" * 100000 + "]" * 100000:0a05a450run --gate-policy <too deep>RecursionError: Stack overflow (used 8144 kB) while decoding a JSON array from a unicode stringlaya-evals: invalid gate policy '<path>': Stack overflow (used 8144 kB) …compare --gate-policy <too deep>run --gate-policy {not json}run --gate-policy <absent path>run --gate-policy <valid, unsatisfiable>FAIL: slice gate rule 1 (language=en, noul_accuracy): metric or slice is missingload_policy("<too deep>")in-processRecursionErrorEvalErrorThe two changed rows are the bug; the four others are the controls that show nothing else moved. The
valid-policy row is the one a reviewer should notice: a genuine quality failure still exits 1 with the
same
FAIL:line, so this does not convert failures into usage errors.Why a CI job cannot tell the two apart: exit 1 is the code "the model regressed" already prints, and a
traceback on stderr from a job that reads only the code looks like a scored run that failed its gate. A
policy file truncated mid-write by a crashed generator, or an editor that dumped a deeply-merged
rulesarray, produces that.docs/evals.md:497-498states the rule this restores, verbatim:and
docs/evals.md:28-29gives the same contract for the whole CLI. The_load_reportreader inlaya/evals_cli.py:294and the agreement-run readers atlaya/evals_cli.py:590already catchRecursionErrorfor exactly this reason; this makes the third caller-supplied JSON door in the harnessagree with the two that were fixed.
The tests add two things to
test_cli_rejects_invalid_policy_json_and_version, which already walked{not jsonand aversion: 2document and asserted only the exit code:Tracebackon stderr, that the refusal still startswith
laya-evals:, and that it says gate policy -- so the message says which flag broke;OSErrorwas already in the tuple, andmainhas its ownFileNotFoundErrorarm, so this arm passes onmaintoo; it is here because the mutation harnessfound that nothing proved the loader rather than
mainwas answering, and the two messages differin the one part that matters (which of the CLI's several paths was missing).
How it was verified
The lane CI runs.
tests/test_evals.pyis not inscripts/test_suites.py; CI runs it as its ownstep,
.github/workflows/ci.yml:174=python -m pytest tests/test_evals.py tests/test_evals_agreement.py -q. That exact command, offline,on this tree: 264 passed (
laya-bench/ci_step_evals_policy_branch_head7b7f19d6.log, at this PR'sfinal head;
laya-bench/ci_step_evals_policy_branch_head702595cb.log14:23Z is the same count at thefix commit) against 264 passed on a pristine
0a05a450worktree(
laya-bench/ci_step_evals_ctrl0a05.log, 14:18Z). The count does not move, and that is expected:both new arms live inside an existing test function, so collection cannot show them. The revert arm
below is the proof they execute.
Revert arm. branch tests +
git show 0a05a450:laya/_eval_policy.py->1 failed, 263 passed (
laya-bench/revert_arm_evals_policy_head7b7f19d6.log, 14:41Z), and the failureis named:
tests/test_evals.py::test_cli_rejects_invalid_policy_json_and_version, withE RecursionError: Stack overflow (used 8144 kB) while decoding a JSON array from a unicode stringasthe line that fails it. Base file sha256
9788b90c673f, branch729d64e76b90; restored byte-for-byte(post-restore sha256
729d64e76b90,git status --porcelainempty). The other 263 -- everypre-existing gate-policy arm, both
runandcompare-- pass on base, so nothing a caller could relyon changed.
Mutation harness (
laya-bench/mutate_evals_policy_depth.py; first passlaya-bench/mutate_evals_policy_depth.log14:26Z, second passlaya-bench/mutate_evals_policy_depth.pass2.log14:38Z). Predictions written into the harness beforethe first run; baseline
185 passed, each mutant a full literal blockcompile()-checked, filerestored and sha256-verified between mutants.
RecursionErrordroppedRecursionErrorreplaced byRuntimeErrorjson.JSONDecodeErrordropped{not jsonescapesOSErrordropped"invalid gate policy"kindValueError, notEvalErrormain's handler misses itPass 2 result:
7 mutants, 5 killed, 2 survived, 1 predicted-count mismatch. All three first-passsurvivors are disclosed rather than dropped:
OSErrorstill exited 2, becausemain(
laya/evals_cli.py) has its own narrowedFileNotFoundErrorarm -- so the surviving mutant was anequivalence at exit-code level, not a gap in my change. It is not equivalent at message level:
mainprints only the OS error, so the caller loses which flag pointed at the missing file. Test arm2 above pins that, and m4 then fails on exactly one test (
predicted 0 failing, got 1-- themismatch is the pin working, disclosed as such).
issubclass(RecursionError, RuntimeError)is
Truein CPython 3.14, soRuntimeErrorin the tuple catches every inputRecursionErrorcatches and the refusal is byte-identical. No distinguishing input exists. I kept
RecursionErrorbecause it says which failure is being handled; the mutant survives because there isnothing for a test to see.
ledger, and it isn't.
Gate.
LAYA_TREE=$TREE HF_HUB_OFFLINE=1 $VENV -B laya-bench/ci_gate.pyat this branch's head ->laya-bench/gate_evals_policy_head7b7f19d6.log, diffed against the pristine0a05a450control(
laya-bench/gate_ctrl0a05.log, 14:10Z) withlaya-bench/diff_gate_logs.py. The shared 92-suite liststructurally cannot see this change (
tests/test_evals.pyis not in it), which is why the CI step andthe revert arm above are the primary witnesses; the gate is here to show the change breaks nothing that
CI does run. The gate at the preceding head
792b548eread 81 invocations,all 92 suites passed,ci_gate exit=0(laya-bench/gate_evals_policy_head792b548e.log, 14:29Z, withtree=andlaya_from=both naming this tree), and its diff against the control isinvocations: branch=81 control=81,identical: 80 differing: 1(
laya-bench/gate_evals_policy_diff_head792b548e.txt, 14:29Z). The one differing invocation istests/test_mcp.py, where the same set members print in a different order(
{'routing', 'latency_ms', 'answers'}vs{'answers', 'latency_ms', 'routing'}) — pre-existingnondeterminism in a suite this PR does not touch, and both tails read
all mcp tests passed. Nothinggained or lost a failure.
Docs. No page moves.
docs/evals.md:497-498already states the rule, and the docstring added herecites it in prose rather than publishing new contract text.
docs/reference/*.mdhas no griffedirective naming
laya._eval_policy(grep of the::: layadirectives returnslaya.confidencesymbols), so the new docstring paragraph is not published. The strict docs build is not run:
/tmp/docsvenv(the zensical venv used for prior doc sweeps) no longer exists, andgit diff --numstatcovers only
laya/_eval_policy.pyandtests/test_evals.py, which is the narrower proof.Lint, compile, secrets.
/opt/homebrew/bin/ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120->All checks passed!.$VENV -m compileall -q laya/ tests/-> rc=0. The+lines of
git diff 0a05a450 HEAD(30 lines) scanned forapi_key|secret|token|password|passwd|credential|authorization|bearer |private key|ghp_|gho_|sk-…|xox[baprs]-|AKIA…:0 matches, rc=1.
Duplicate guard. 30 open PRs at 14:43Z (
gh pr list --repo NandhaKishorM/laya --state open --limit 200 --json files),mainstill0a05a450. Filtering every open PR's file list forlaya/_eval_policy.py,laya/evals.py,laya/evals_cli.py,laya/evidence.pyandtests/test_evals.pyreturns one hit -- my own #1317, which is a different unit -- and no othereditor of
laya/_eval_policy.py.Cross-PR conflict, measured not assumed. #1317, this PR and the dataset-reader PR all insert into
tests/test_evals.py, sogit merge-tree --write-treewas run for all six orderings of the threebranch pairs: every one exits 0 with no conflicted path.
laya/_eval_policy.pyhas one editor inthe stack: this PR.
Cross-port check. Grep of
laya-java,laya-go,laya-dotnet,laya-ts,laya-rustfor agate-policy JSON reader returns 0 files, so no port fixture pins the current escape.
Notes for the reviewer
The depth limit itself is untouched. This does not raise or lower how deep
jsonwill go, and does notadd a
max_depthknob; CPython's scanner still answers at ~8 MB of stack, and its message is what therefusal now carries.
laya/evals.py::Dataset.from_jsonlhas the same shape of hole for the datasetfile and is deliberately not in this PR (it needs its own reader-level tests, and #1317/#this already
share
tests/test_evals.py).main'sFileNotFoundErrorarm stays as it is: it is the second door for a path that was neveropened, and the two arms differ in that
load_policynames which flag asked for the file.No new dependency, no hosted service, no public API signature change, no change for any policy that
jsoncan read.Checklist
main(0a05a450;git rev-parse refs/remotes/upstream/main=0a05a450)docs/evals.md:497-498already documents the exit-code rule this restores, so nothing needed editing)Fixes #