Skip to content

fix(calibrate): a target_error that cannot be compared is refused by name - #1404

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:calibrate-target-error-comparable
Oct 28, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:calibrate-target-error-comparable

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

fit_abstention_thresholds now reports a non-comparable target_error through the ValueError it already raises for a bool, instead of letting the interpreter's TypeError escape. One try around the existing comparison; no new message, no new import.

Why

laya/calibrate.py:485 read:

if isinstance(target_error, bool) or not 0.0 <= target_error <= 1.0:

A str (a "0.1" read out of JSON), a None (a key that was not there), a complex, a list or a dict is not bool and not comparable, so the <= raises before the if completes. Measured against the suite's own _small records:

'0.1'   -> TypeError: '<=' not supported between instances of 'float' and 'str'
None    -> TypeError: '<=' not supported between instances of 'float' and 'NoneType'
1j      -> TypeError: '<=' not supported between instances of 'float' and 'complex'
[0.1]   -> TypeError: '<=' not supported between instances of 'float' and 'list'
{}      -> TypeError: '<=' not supported between instances of 'float' and 'dict'
nan     -> ValueError: target_error must be in [0.0, 1.0], got nan

The parameter is a documented float on a public fitting entry point that laya.train reaches from a config file, and the very next line runs min_bucket_n through _positive_int (:114), which has refused a string and a None by name for a while — so the two arguments one line apart disagreed about whether a bad shape is worth naming. After the change all five rows above print target_error must be in [0.0, 1.0], got '0.1' and so on.

The comparison is run and its TypeError turned into a refusal, rather than the range test being replaced by an isinstance check, because an isinstance list is what would break a working caller: not isinstance(target_error, Real) refuses Decimal("0.10") — decimal.Decimal is registered as numbers.Number, not numbers.Real — and a Decimal compares against 0.0/1.0 perfectly well and is only ever used for rate <= target_error (:435). My first draft used Real and the new Decimal case caught it; the case stays in the suite so the simpler-looking version cannot be swapped back in later.

How it was verified

LAYA_WT=$PWD .venv/bin/python -B tests/test_conformal_abstention.py   # 86 passed, 0 failed
LAYA_WT=$PWD .venv/bin/python -B tests/test_train.py                  # Ran 58 tests ... OK
ruff check laya/calibrate.py tests/test_conformal_abstention.py --select=E9,F63,F7,F82,F401,F811 --line-length=120

The pristine b9fe351 worktree gives 80 passed, 0 failed for the same suite, so the six added checks are the only difference. ruff check laya/ (the CI gate) passes; the 14 errors ruff check tests/ prints are pre-existing on base and outside the gate.

The mutation arm reverts only the guard and keeps the tests, which fails exactly the five refusals and quotes both sides — and shows the Decimal case passing in both arms, which is what it is for:

81 passed, 5 failed
  FAIL target_error/'0.1' is refused by name: got "raised TypeError: '<=' not supported between instances of 'float' and 'str'" want "target_error must be in [0.0, 1.0], got '0.1'"
  FAIL target_error/None is refused by name: got "raised TypeError: ... 'NoneType'" want 'target_error must be in [0.0, 1.0], got None'
  FAIL target_error/1j, [0.1], {'t': 0.1}: same, for complex, list and dict

Nothing is fixture-pinned: grep -rn "target_error must be" over laya-java/ laya-go/ laya-rust/ laya-dotnet/ laya-ts/ returns nothing, and gen_fixtures.py --check reports the same two families stale as a pristine base worktree (lang_detect.json, python_json.json — the Linux/macOS regeneration documented at gen_fixtures.py:1922), so this diff moves no fixture.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

…ion, not a dict key

`chosen_key not in lookup` raised a bare interpreter TypeError for a `choice` the
remote body sent as a JSON array or object, so neither the RuntimeError nor the
fallback_tool path the string case takes was ever reached. agno, crewai and
llamaindex already refuse a non-string `choice` at the same boundary.
…ed to a range

`0.0 <= target_error <= 1.0` raised the interpreter's own TypeError for a "0.1"
from a config file or a None from a missing key, naming neither the parameter nor
the caller's mistake. Run the comparison and report its failure through the
message the bool arm already uses, so a value that does compare -- a Decimal,
which numbers.Real does not cover -- keeps fitting.
@NandhaKishorM
NandhaKishorM merged commit e157bc4 into NandhaKishorM:main Oct 28, 2026
52 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 28, 2026
24 PRs. Vietnamese kept off the English checkpoint (#1392), deciding-window
attribution preserved through end hooks (#1369), RemoteRouter identifying
itself with a configurable User-Agent (#1396), ten more inputs refused where
they enter (#1372, #1394, #1397, #1398, #1400, #1401, #1402, #1403, #1404,
#1405, #1406), per-language temperatures and per-call budgets in .NET and
Rust (#1370, #1371), .NET calibration files (#1384), the Go router lifecycle,
SHA-256 pins and routed inference (#1374, #1379, #1395), a C++ foundation
(#1399, #1407), and parity-lane sharding that replaces the budget headroom
added for 0.4.16 (#1373, #1375).

Verified before tagging: all 92 suites pass, 640 tests, up from 612. ruff
clean on both lists. laya-ts 1251 tests across 44 files, tsc --noEmit clean.
Every generated-artifact gate clean on CPython 3.12 / unicodedata 15.0.0:
Java 20 families match laya (9 unverified), UnicodeTables matches CPython
15.0.0, LanguageTables matches laya/lang.py, three Go tables_gen.go match,
both laya-cpp .inc files match. Version agrees across pyproject.toml,
laya.__version__ and nix/package.nix. README carries 0.4.17 only.

Security audit of the merge diff, 362766 added lines over 480 files: zero
obfuscation, zero secrets, zero boundary hits, zero binaries, no mode or
symlink changes, 20 of 20 actions SHA-pinned including the new cpp.yml, npm
install hooks unchanged, and no third-party host in any added line
(example.com 90, github.com 2, download.pytorch.org 1, docs.rs 1). The 12
dynamic-exec hits are the C++ test harness invoking its own checker through
sys.executable, the Go sample generator reading laya's own example by AST at
development time, and one comment in rust.yml.

Held: #1364 (unchanged, it moves a cross-port contract), #1381 (a #[doc(hidden)]
but pub Rust hook that would ship as public API able to replace the forward
pass), #1386 (route_batch switched from self.route to self._route, silently
bypassing subclass overrides the docstring promises), and nine stacked port
PRs that conflict with each other and need linearizing (#1377, #1378, #1380,
#1382, #1383, #1385, #1387, #1388, #1389, #1390).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. The reasoning I want to single out is why you ran the comparison and caught TypeError rather than replacing it with an isinstance allowlist: a Decimal does compare and numbers.Real does not cover it, so an allowlist would have refused a value that works today. That is the harder of the two options to get right and the one that does not quietly narrow the accepted types.

The failure being fixed is worth stating plainly for anyone reading later. '<=' not supported between instances of 'float' and 'str' names neither the parameter nor the caller's mistake, and min_bucket_n one line below has always named itself, so the inconsistency was visible in the same function. A "0.1" out of a config file and a None from a missing key are both realistic sources.

Keeping the isinstance(target_error, bool) guard ahead of the range test still matters for the reason the existing comment gives, since False reads as the cut that abstains on everything and True gates nothing, and both fit a map without complaint.

Shipped in 0.4.17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants