Skip to content

fix(common): a temperature wider than a double falls back, not crashes - #1433

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:py-temperature-overflow-fallback
Oct 31, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:py-temperature-overflow-fallback

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya/common.py::clamp_temperature and ::temperature_rejected now read OverflowError the way
they already read TypeError and ValueError: as "this does not convert to a number". The clamp
returns its documented neutral 1.0 and the predicate reports the entry, instead of letting the
exception escape into the loader.

 except (TypeError, ValueError):          ->   except (TypeError, ValueError, OverflowError):

Three test files carry it, one per entry point the guard has:

  • tests/test_router.py — four rows on the pinned clamp table (the two wide integers, the smallest
    power of two float() refuses, and its convertible neighbour, which must keep clamping).
  • tests/test_temperature_loading.py — a new test_json_integer_wider_than_a_float_uses_neutral_fallback_and_warn
    that builds a checkpoint in a tmpdir and loads it through laya.load(), plus four rows on the
    temperature_rejected predicate table.
  • tests/test_calibrate.py — a wide-int-load/… block on the calibration-file path, where this
    function's own docstring promise is the one being broken (below).

Why

JSON's grammar has no double limit, and json.load keeps arbitrary precision — so a temperature
written out as a 401-digit integer literal loads as an int and float() answers it with
OverflowError: int too large to convert to float. Both guards caught (TypeError, ValueError)
only, so the documented fallback ("falling back to 1.0 if it is not a number") did not happen: the
load died after the checkpoint had already been read off disk.

Measured with one probe, run against both trees with laya.__file__ printed as the witness
(…/laya-maincheck/laya/__init__.py | 0.4.19 and …/laya-u13/laya/__init__.py | 0.4.19):

input in a temperature field 55a19f7b (base) this branch
10**400 — 401 digits, what json.load produces OverflowError: int too large to convert to float 1.0
-(10**400) OverflowError 1.0
2**1024 — smallest power of two float() refuses OverflowError 1.0
2**1023 — the neighbour float() can hold 5.0 (clamped) 5.0 (unchanged)
Fraction(10**400, 1) OverflowError: integer division result too large for a float 1.0
10**5000 OverflowError 1.0
float("inf"), nan, "abc", [], True, None 1.0 1.0 (unchanged)
9223372036854775807 — int64 max, converts fine 5.0 5.0 (unchanged)

temperature_rejected(raw, 1.0) answered OverflowError at base for the five values the probe ran
through it — 10**400, -(10**400), 2**1024, 10**5000, Fraction(10**400, 1) — and now
answers True, so the loader names the entry instead of raising on the way to its own warning;
every other row of that table is unchanged. And
resolve_lang_temperatures({"en": {"temperature": [10**400, 1, 1]}}, …) — where a per-language
override is resolved — moved from OverflowError: int too large to convert to float to

{'en': {'temperature': [1.0, 1.0, 1.0], 'temperature_by_options': {}}}

plus the loader's own RuntimeWarning, which names the value it replaced digit for digit.

This is a broken documented promise, not a hypothetical one. apply_calibration_payload says at
laya/calibrate.py:743-745: "Each temperature is passed through clamp_temperature, so a
non-numeric or out-of-range entry cannot crash a later forward the way an unclamped zero used to."
Measured against that sentence, four payloads through the same function:

payload["temperature"][0] base this branch
9.0 — out of range installed [5.0, 1.0, 1.0], 1 warning same
"abc" — non-numeric installed [1.0, 1.0, 1.0], 1 warning same
inf — what JSON's 1e400 parses to installed [1.0, 1.0, 1.0], 1 warning same
10**400 — an integer of the same magnitude raises OverflowError installed [1.0, 1.0, 1.0], 1 warning

Three of the four arms of that promise hold at base; the fourth — the same number, spelled as an
integer instead of with an exponent — crashes Agent.load_calibration() and
ONNXAgent.load_calibration() instead of warning. That is the inconsistency worth closing: the
spelling of a JSON number decides whether a calibration file is forgiven or fatal.

Three supporting observations:

  • This is the remaining half of work already merged here. fix(confidence): a probability too wide for a double is no number, not a crash #1405 ("a probability too wide for a
    double is no number, not a crash") and fix(confidence): a JSON integer wider than a double is refused, not crashed, by every confidence reader #1413 ("a JSON integer wider than a double is refused, not
    crashed, by every confidence reader") made the confidence readers answer OverflowError as a
    non-number. The temperature readers in common.py were never reached by that change, and they
    sit on the same json.load path from the same file format.
  • The rest of the package already reads OverflowError this way. confidence._finite
    (laya/confidence.py:15,24,85-86), hooks.validate_timeout (laya/hooks.py:275-277),
    calibrate.apply_calibration_payload (laya/calibrate.py:780) and _eval_policy
    (laya/_eval_policy.py:20) all catch it as "does not read as a number". These two guards in
    common.py were the last ones in the temperature path that did not.
  • The port already agrees, which makes the Python core the divergent side.
    laya-ts/src/common.ts:563-570 answers 1.0 for all four of JSON.parse("1e400"),
    JSON.parse("1" + "0".repeat(400)), 10n ** 400n and Infinity — measured against the built
    dist/.

Scope is bounded honestly. temperature_rejected's promise is that the predicate never raises for
a value the clamp replaced. The one value still outside that is a value %r cannot print — past
CPython's 4300-digit integer-to-string limit — and it cannot reach a loader at all, because
json.load rejects a 5001-digit literal itself, also measured:
ValueError: Exceeds the limit (4300 digits) for integer string conversion: value has 5001 digits.
So the fix closes the whole reachable domain and introduces no new failure mode in the warning path.

How it was verified

python tests/test_router.py                           # 871 -> 875 checks, 0 failed
python -m unittest tests.test_temperature_loading -v  # Ran 10 ... OK  ->  Ran 11 ... OK
python tests/test_calibrate.py                        # 375 -> 378 checks, 0 failed
python tests/test_onnx_batch.py                       # 67 passed, 0 failed
python tests/test_onnx_lang_parity.py                 # 20 passed, 0 failed
python tests/test_calibration_persistence.py          # Ran 7 tests ... OK
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/
python laya-java/scripts/gen_fixtures.py --check
python scripts/test_suites.py                         # the CI/release gate list: all 93 suites passed

Every number above is from a fresh run of the base tree (55a19f7b, a pristine worktree at
upstream/main) and this branch; the gate wrapper prints laya_from= before running, and each run
answered from the tree it claims.

gen_fixtures --check reports lang_detect.json and python_json.json as stale — verified as
exactly the same two at base 55a19f7b, so inherited and untouched here. No port fixture pins the
old crash: the fixture generators feed ordinary floats to clamp_temperature, and a fixture holding
a wide integer would have crashed itself while being written.

Mutation arms — each reverts one except line of laya/common.py, runs the three suites, and
counts an aborted suite as a KILL (the crash is what the suite is there to catch):

arm test_router.py test_temperature_loading.py test_calibrate.py
neither (base source) ABORT OverflowError ABORT OverflowError ABORT OverflowError
clamp_temperature only 875 passed, 0 failed ABORT OverflowError ABORT OverflowError
temperature_rejected only ABORT OverflowError ABORT OverflowError ABORT OverflowError
both (the branch) 875 passed, 0 failed Ran 11 / OK 378 passed, 0 failed

The middle row is the one that matters for coverage: with only the clamp fixed, test_router.py
goes green while the two load tests still abort, so the load-path tests are genuinely reaching the
predicate and the three test additions are not each other's duplicates. The harness snapshots
laya/common.py (56370 bytes, sha256 89b9db7ad50c16f4…), asserts the restore is byte-identical,
and re-runs all three suites from disk afterwards — the tree that reported the first row is the tree
committed.

Gate output
tree=/Users/aashish/github contribution /GIT/laya-u13
laya_from=/Users/aashish/github contribution /GIT/laya-u13/laya/__init__.py
gate=scripts/test_suites.py (the list CI and the release gate share)

===== tests/test_router.py =====
875 passed, 0 failed

===== tests/test_calibrate.py =====
378 passed, 0 failed

[... every remaining suite prints its own 0-failed line or OK ...]

785 passed, 1 skipped, 6 warnings in 28.28s

all 93 suites passed
ci_gate exit=0

Every failure count the gate printed reads zero — 53 lines of the form N passed, 0 failed and 16
unittest blocks ending in OK — and the run exits 0. The 785 passed, 1 skipped line is the
pytest-discovery block. The skips the log names are this machine missing checkpoint files, not test
failures: SKIP: english: no tokenizer under /Users/aashish/laya_models and multilingual: no checkpoint files, skipped. The gate was re-run after the test_calibrate.py addition, so the tree
it measured is the tree in the commit.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

The public API did not change: no new symbol, no signature change, and no documented promise is
rewritten. clamp_temperature's docstring already says "falling back to 1.0 if it is not a number"
and apply_calibration_payload's already says a non-numeric entry "cannot crash a later forward" —
this makes those two sentences true rather than adding a third.

Fixes #

…her unreadable one

`clamp_temperature` and `temperature_rejected` caught `TypeError` and `ValueError`, so a JSON
integer literal of arbitrary precision -- `10**400`, which is what `json.load` hands a loader
for a 401-digit temperature -- reached `float()` and escaped as `OverflowError`. The documented
neutral 1.0 never happened: the load died after the checkpoint had already been read off disk,
and `apply_calibration_payload` broke its own promise (laya/calibrate.py:743-745) that "a
non-numeric or out-of-range entry cannot crash a later forward" for exactly that one spelling,
while `9.0`, `"abc"` and `1e400` (which JSON parses as a float, so it arrives as `inf`) were all
forgiven.

`confidence._finite`, `hooks.validate_timeout` and `calibrate` all already read this exception
as "does not convert to a number"; these were the last two handlers in the temperature path
that did not.

Verified on base 55a19f7 against this branch: tests/test_router.py 871 -> 875 checks,
tests/test_temperature_loading.py Ran 10 -> Ran 11 (both OK), tests/test_calibrate.py
375 -> 378 checks, and the shared CI/release gate list (scripts/test_suites.py) reports all 93
suites passed. Three mutation arms -- each reverts one `except` line and counts an aborted
suite as a KILL -- all kill, with the tree restored byte-identical and re-run from disk.
@NandhaKishorM
NandhaKishorM merged commit 113692d into NandhaKishorM:main Oct 31, 2026
64 of 65 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged, and shipping in 0.4.20.

The argument that persuaded me is the one in your "broken documented promise" section: apply_calibration_payload's docstring already promises that a non-numeric or out-of-range entry "cannot crash a later forward", and three of the four arms of that promise held while the fourth, the same magnitude written as an integer instead of with an exponent, raised. The spelling of a JSON number deciding whether a calibration file is forgiven or fatal is the kind of inconsistency worth a two-line fix.

I ran the four arms myself:

payload["temperature"][0] result with this merged
9.0, out of range installed [5.0, 1.0, 1.0], 1 warning
"abc", non-numeric installed [1.0, 1.0, 1.0], 1 warning
inf, what JSON's 1e400 parses to installed [1.0, 1.0, 1.0], 1 warning
10**400, the same number as an integer installed [1.0, 1.0, 1.0], 1 warning

The fourth row raised OverflowError before. And the guards themselves: 10**400, -(10**400), 2**1024, 10**5000 and Fraction(10**400, 1) all clamp to 1.0 and all report True from temperature_rejected.

The controls are what make this safe to take, and I checked all nine. 2**1023 still clamps to 5.0, because it is the neighbour float() can hold and a fix that swallowed it would be a silent behaviour change on valid input. 9223372036854775807 still clamps to 5.0. inf, nan, "abc", [], True and None still give 1.0, and 2.5 still passes through as 2.5.

Your mutation table's middle row is the one that earns the three separate test files: with only clamp_temperature fixed, test_router.py goes green while both load-path suites still abort, so the additions are reaching the predicate rather than restating each other. That is the right way to show three test files are not one test three times.

scripts/test_suites.py passes here with this merged: all 93 suites, 791 pytest-collected tests, exit 0, and tests/test_temperature_loading.py goes from 10 to 11 as you said.

The scope note about %r and CPython's 4300-digit integer-to-string limit is appreciated, and correct that json.load refuses a 5001-digit literal first, so the unreachable case stays unreachable.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Follow-up, since this shipped and then showed up in the .NET lane. Nothing here is a problem with your change, and I am not asking you to do anything.

The .NET lane re-records its goldens from the Python code at each commit, and three rows of laya-dotnet/tests/Laya.Tests/golden/routing/calibration_probe.json existed specifically to pin the old crash: "a temperature that is a huge integer", "a bucket temperature that is a huge integer", and "a huge integer after an identity warning". All three recorded OverflowError: int too large to convert to float. With this merged they record a clamp warning and an installed [1.0, 1.0, 1.0] instead, so the C# port's own pre-check, which still throws, now disagrees with the regenerated golden. 3 failures out of 2557, tracked in #1434.

This is the lane working as designed. It is continue-on-error on purpose, and the workflow header says why: Python is the product and the ports follow it. I mention it only because it is a useful answer to a question your PR raised and answered carefully, "no port fixture pins the old crash". That was true of the fixture generators you checked, laya-java/scripts/gen_fixtures.py, which do feed ordinary floats to clamp_temperature. The .NET probe is generated by a different tool, laya-dotnet/tools/regen_golden.py, and that one did pin it. Worth knowing for next time: grep -rl "int too large to convert to float" --include='*.json' finds it, and the .NET probe is the only fixture in the repo that carries these cases.

It also turned up one thing your scope note did not cover, which I have filed as #1435 rather than expand your PR after the fact. binning_map values in the same file, validated in the same function, still go through [float(v) for v in values] with nothing catching OverflowError:

value in binning_map['choice:2'].values result at 0.4.20
"abc", None, [] ValueError: ... values must be numbers, got [...]
inf, what JSON's 1e400 parses to ValueError: ... values must be finite, got [...]
10**400 OverflowError: int too large to convert to float

That is your own argument one key over: the same number written two ways, one forgiven and one fatal. The fix there is not the same as yours though, which is why I did not just extend this PR. A temperature has a documented neutral fallback so clamping to 1.0 and warning is right; a binning-map value has no neutral stand-in and that path already refuses every unreadable value, so the integer spelling should produce the same named ValueError rather than a fallback.

Your diff was right to stop where it did. Shipped in 0.4.20 and the controls all hold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants