Repository navigation
fix(common): a temperature wider than a double falls back, not crashes - #1433
NandhaKishorM merged 1 commit into
Conversation
…her unreadable one `clamp_temperature` and `temperature_rejected` caught `TypeError` and `ValueError`, so a JSON integer literal of arbitrary precision -- `10**400`, which is what `json.load` hands a loader for a 401-digit temperature -- reached `float()` and escaped as `OverflowError`. The documented neutral 1.0 never happened: the load died after the checkpoint had already been read off disk, and `apply_calibration_payload` broke its own promise (laya/calibrate.py:743-745) that "a non-numeric or out-of-range entry cannot crash a later forward" for exactly that one spelling, while `9.0`, `"abc"` and `1e400` (which JSON parses as a float, so it arrives as `inf`) were all forgiven. `confidence._finite`, `hooks.validate_timeout` and `calibrate` all already read this exception as "does not convert to a number"; these were the last two handlers in the temperature path that did not. Verified on base 55a19f7 against this branch: tests/test_router.py 871 -> 875 checks, tests/test_temperature_loading.py Ran 10 -> Ran 11 (both OK), tests/test_calibrate.py 375 -> 378 checks, and the shared CI/release gate list (scripts/test_suites.py) reports all 93 suites passed. Three mutation arms -- each reverts one `except` line and counts an aborted suite as a KILL -- all kill, with the tree restored byte-identical and re-run from disk.
|
Merged, and shipping in 0.4.20. The argument that persuaded me is the one in your "broken documented promise" section: I ran the four arms myself:
The fourth row raised The controls are what make this safe to take, and I checked all nine. Your mutation table's middle row is the one that earns the three separate test files: with only
The scope note about |
|
Follow-up, since this shipped and then showed up in the .NET lane. Nothing here is a problem with your change, and I am not asking you to do anything. The This is the lane working as designed. It is It also turned up one thing your scope note did not cover, which I have filed as #1435 rather than expand your PR after the fact.
That is your own argument one key over: the same number written two ways, one forgiven and one fatal. The fix there is not the same as yours though, which is why I did not just extend this PR. A temperature has a documented neutral fallback so clamping to 1.0 and warning is right; a binning-map value has no neutral stand-in and that path already refuses every unreadable value, so the integer spelling should produce the same named Your diff was right to stop where it did. Shipped in 0.4.20 and the controls all hold. |
What
laya/common.py::clamp_temperatureand::temperature_rejectednow readOverflowErrorthe waythey already read
TypeErrorandValueError: as "this does not convert to a number". The clampreturns its documented neutral
1.0and the predicate reports the entry, instead of letting theexception escape into the loader.
Three test files carry it, one per entry point the guard has:
tests/test_router.py— four rows on the pinned clamp table (the two wide integers, the smallestpower of two
float()refuses, and its convertible neighbour, which must keep clamping).tests/test_temperature_loading.py— a newtest_json_integer_wider_than_a_float_uses_neutral_fallback_and_warnthat builds a checkpoint in a tmpdir and loads it through
laya.load(), plus four rows on thetemperature_rejectedpredicate table.tests/test_calibrate.py— awide-int-load/…block on the calibration-file path, where thisfunction's own docstring promise is the one being broken (below).
Why
JSON's grammar has no double limit, and
json.loadkeeps arbitrary precision — so a temperaturewritten out as a 401-digit integer literal loads as an
intandfloat()answers it withOverflowError: int too large to convert to float. Both guards caught(TypeError, ValueError)only, so the documented fallback ("falling back to 1.0 if it is not a number") did not happen: the
load died after the checkpoint had already been read off disk.
Measured with one probe, run against both trees with
laya.__file__printed as the witness(
…/laya-maincheck/laya/__init__.py | 0.4.19and…/laya-u13/laya/__init__.py | 0.4.19):55a19f7b(base)10**400— 401 digits, whatjson.loadproducesOverflowError: int too large to convert to float1.0-(10**400)OverflowError1.02**1024— smallest power of twofloat()refusesOverflowError1.02**1023— the neighbourfloat()can hold5.0(clamped)5.0(unchanged)Fraction(10**400, 1)OverflowError: integer division result too large for a float1.010**5000OverflowError1.0float("inf"),nan,"abc",[],True,None1.01.0(unchanged)9223372036854775807— int64 max, converts fine5.05.0(unchanged)temperature_rejected(raw, 1.0)answeredOverflowErrorat base for the five values the probe ranthrough it —
10**400,-(10**400),2**1024,10**5000,Fraction(10**400, 1)— and nowanswers
True, so the loader names the entry instead of raising on the way to its own warning;every other row of that table is unchanged. And
resolve_lang_temperatures({"en": {"temperature": [10**400, 1, 1]}}, …)— where a per-languageoverride is resolved — moved from
OverflowError: int too large to convert to floattoplus the loader's own
RuntimeWarning, which names the value it replaced digit for digit.This is a broken documented promise, not a hypothetical one.
apply_calibration_payloadsays atlaya/calibrate.py:743-745: "Each temperature is passed throughclamp_temperature, so anon-numeric or out-of-range entry cannot crash a later forward the way an unclamped zero used to."
Measured against that sentence, four payloads through the same function:
payload["temperature"][0]9.0— out of range[5.0, 1.0, 1.0], 1 warning"abc"— non-numeric[1.0, 1.0, 1.0], 1 warninginf— what JSON's1e400parses to[1.0, 1.0, 1.0], 1 warning10**400— an integer of the same magnitudeOverflowError[1.0, 1.0, 1.0], 1 warningThree of the four arms of that promise hold at base; the fourth — the same number, spelled as an
integer instead of with an exponent — crashes
Agent.load_calibration()andONNXAgent.load_calibration()instead of warning. That is the inconsistency worth closing: thespelling of a JSON number decides whether a calibration file is forgiven or fatal.
Three supporting observations:
double is no number, not a crash") and fix(confidence): a JSON integer wider than a double is refused, not crashed, by every confidence reader #1413 ("a JSON integer wider than a double is refused, not
crashed, by every confidence reader") made the confidence readers answer
OverflowErroras anon-number. The temperature readers in
common.pywere never reached by that change, and theysit on the same
json.loadpath from the same file format.OverflowErrorthis way.confidence._finite(
laya/confidence.py:15,24,85-86),hooks.validate_timeout(laya/hooks.py:275-277),calibrate.apply_calibration_payload(laya/calibrate.py:780) and_eval_policy(
laya/_eval_policy.py:20) all catch it as "does not read as a number". These two guards incommon.pywere the last ones in the temperature path that did not.laya-ts/src/common.ts:563-570answers1.0for all four ofJSON.parse("1e400"),JSON.parse("1" + "0".repeat(400)),10n ** 400nandInfinity— measured against the builtdist/.Scope is bounded honestly.
temperature_rejected's promise is that the predicate never raises fora value the clamp replaced. The one value still outside that is a value
%rcannot print — pastCPython's 4300-digit integer-to-string limit — and it cannot reach a loader at all, because
json.loadrejects a 5001-digit literal itself, also measured:ValueError: Exceeds the limit (4300 digits) for integer string conversion: value has 5001 digits.So the fix closes the whole reachable domain and introduces no new failure mode in the warning path.
How it was verified
Every number above is from a fresh run of the base tree (
55a19f7b, a pristine worktree atupstream/main) and this branch; the gate wrapper prints
laya_from=before running, and each runanswered from the tree it claims.
gen_fixtures --checkreportslang_detect.jsonandpython_json.jsonas stale — verified asexactly the same two at base
55a19f7b, so inherited and untouched here. No port fixture pins theold crash: the fixture generators feed ordinary floats to
clamp_temperature, and a fixture holdinga wide integer would have crashed itself while being written.
Mutation arms — each reverts one
exceptline oflaya/common.py, runs the three suites, andcounts an aborted suite as a KILL (the crash is what the suite is there to catch):
test_router.pytest_temperature_loading.pytest_calibrate.pyOverflowErrorOverflowErrorOverflowErrorclamp_temperatureonly875 passed, 0 failedOverflowErrorOverflowErrortemperature_rejectedonlyOverflowErrorOverflowErrorOverflowError875 passed, 0 failedRan 11 / OK378 passed, 0 failedThe middle row is the one that matters for coverage: with only the clamp fixed,
test_router.pygoes green while the two load tests still abort, so the load-path tests are genuinely reaching the
predicate and the three test additions are not each other's duplicates. The harness snapshots
laya/common.py(56370 bytes, sha25689b9db7ad50c16f4…), asserts the restore is byte-identical,and re-runs all three suites from disk afterwards — the tree that reported the first row is the tree
committed.
Gate output
Every failure count the gate printed reads zero — 53 lines of the form
N passed, 0 failedand 16unittest blocks ending in
OK— and the run exits 0. The785 passed, 1 skippedline is thepytest-discovery block. The skips the log names are this machine missing checkpoint files, not test
failures:
SKIP: english: no tokenizer under /Users/aashish/laya_modelsandmultilingual: no checkpoint files, skipped. The gate was re-run after thetest_calibrate.pyaddition, so the treeit measured is the tree in the commit.
Checklist
mainThe public API did not change: no new symbol, no signature change, and no documented promise is
rewritten.
clamp_temperature's docstring already says "falling back to 1.0 if it is not a number"and
apply_calibration_payload's already says a non-numeric entry "cannot crash a later forward" —this makes those two sentences true rather than adding a third.
Fixes #