Skip to content

docs(calibrate): an empty bin rewrites its region to the bin's centre - #1124

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-binning-empty-bin-centre
Oct 12, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:fix-binning-empty-bin-centre

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

One clause of fit_binning_map's docstring, corrected, and the behaviour it describes pinned by 17
new checks (240 → 257 in tests/test_calibrate.py).

The sentence read:

An empty bin (a confidence range the calibration set never produced) maps to its own midpoint, i.e.
leaves that region unchanged
, so an unseen value is never recalibrated to a fabricated 0.

The first clause is right and the last is right; the middle one is not. values.append((b + 0.5) / bins) replaces every confidence in that region with one number, so a value that lands in an untouched
bin is rewritten and moves. The new text says what the centre does, bounds the move, and names the
identity case the sentence had conflated it with — a bucket the map has no entry for, which
apply_binning_map genuinely does return untouched.

No behaviour changes.

Why

The two cases are one level apart and the prose merged them:

  • bucket missing — apply_binning_map(confidence, bucket, map) looks up binning_map.get(bucket),
    finds nothing, returns the confidence as it came in. Already checked
    (binning/apply unknown bucket is identity).
  • bin empty — the bucket is in the map, but no calibration record ever landed in that one bin of
    it. The value comes back as (b + 0.5) / bins, not as what went in.

A reader who took "leaves that region unchanged" at face value reasons wrong about exactly the case
that matters: fit_abstention_thresholds(..., binning_map=) fits a gate cut on the binned scale,
because the runtime rewrites answer_confidence through the map before the gate reads it. An unseen
confidence — one the calibration set never produced, which is what an empty bin means — is the value
whose post-binning number a caller cannot otherwise predict.

Measured, at the shipped default bins=15, on a bucket whose records all land in one bin:

empty-bin values                     [0.0333, 0.1, 0.1667, 0.2333, 0.3, 0.3667, 0.4333, 0.5,
                                      0.5667, 1.0, 0.7, 0.7667, 0.8333, 0.9, 0.9667]
largest move inside an untouched bin  0.033333 (= 1/(2*bins))

So a confidence in an untouched bin moves by up to 0.0333 — small, but the same order as the
target_error=0.10 margin an abstention cut is chosen against, and not zero. The bound is what the
docstring should have given, and now does.

How it was verified

python tests/test_calibrate.py
python scripts/test_suites.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/

tests/test_calibrate.py: 257 passed, 0 failed (240 at base). scripts/test_suites.py:
all 87 suites passed, exit 0. ruff All checks passed!, compileall silent.

The checks build a deterministic map — 200 records all at confidence 0.65, so one bin is filled and
the other nine are empty by construction, no RNG — and then:

mutant what it claims named failures
empty bin → 0.0 a fabricated 0 is harmless 8: an untouched bin holds its centre, exactly the filled bin is not a centre, the four comes back as the centre probes, moves a value by at most half a bin width, no confidence is recalibrated to 0
empty bin → b / bins (left edge) any point of the region will do the same 8, printing different values
apply_binning_map returns the confidence for a centre-valued bin the old prose, made true in code 8: the four comes back as the centre probes and the four is not the value it came in as probes
the shipped docstring restored the sentence was fine 4 doc checks, and only they

The last row is why the docstring is pinned at all: it is the only way this suite distinguishes "the
prose was corrected" from "the prose was rewritten". And the third row is the mutant that would make
the old sentence true — killed, so the four doc checks are not merely enforcing one wording.

  • Diff scanned for keys/tokens: 0 hits.
  • No docs page repeats the claim (grep -rn "binning" docs/ README.md finds no prose copy of it), so
    nothing else goes stale.
  • Precedent followed: tests/test_conformal_abstention.py:230-237 pins check_min_confidence_map's
    docstring to its refusals the same way, with normalized-whitespace inspect.getdoc.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

`fit_binning_map` said an empty bin "maps to its own midpoint, i.e. leaves
that region unchanged". The second clause is false: `values.append((b + 0.5)
/ bins)` replaces every confidence in the region with one number, so a value
in an untouched bin moves by up to half a bin width. The identity case is a
different one -- a bucket the map has no entry for, which `apply_binning_map`
returns untouched -- and the sentence conflated the two.

Say what the centre does (a value stays inside the region it came from and is
never recalibrated to a fabricated 0) and hold the docstring against the
behaviour: an untouched bin's edges, the half-bin bound, the non-zero floor,
and four doc checks that fail if the pass-through claim comes back.
@NandhaKishorM
NandhaKishorM merged commit 2df4c9c into NandhaKishorM:main Oct 12, 2026
40 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. "Maps to its own midpoint, i.e. leaves that region unchanged" was two claims and only the first was true: the value is still rewritten, it just moves by at most half a bin width. What the centre actually buys is that an unseen confidence is never recalibrated to a fabricated 0, and only a bucket the map has no entry for passes through untouched. Getting those two apart matters for anyone reasoning about what a binning map does to a confidence it never saw.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants