Repository navigation
docs(calibrate): an empty bin rewrites its region to the bin's centre - #1124
Merged
NandhaKishorM merged 1 commit intoOct 12, 2026
Merged
Conversation
`fit_binning_map` said an empty bin "maps to its own midpoint, i.e. leaves that region unchanged". The second clause is false: `values.append((b + 0.5) / bins)` replaces every confidence in the region with one number, so a value in an untouched bin moves by up to half a bin width. The identity case is a different one -- a bucket the map has no entry for, which `apply_binning_map` returns untouched -- and the sentence conflated the two. Say what the centre does (a value stays inside the region it came from and is never recalibrated to a fabricated 0) and hold the docstring against the behaviour: an untouched bin's edges, the half-bin bound, the non-zero floor, and four doc checks that fail if the pass-through claim comes back.
4 tasks done
Owner
|
Merged. "Maps to its own midpoint, i.e. leaves that region unchanged" was two claims and only the first was true: the value is still rewritten, it just moves by at most half a bin width. What the centre actually buys is that an unseen confidence is never recalibrated to a fabricated 0, and only a bucket the map has no entry for passes through untouched. Getting those two apart matters for anyone reasoning about what a binning map does to a confidence it never saw. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
One clause of
fit_binning_map's docstring, corrected, and the behaviour it describes pinned by 17new checks (240 → 257 in
tests/test_calibrate.py).The sentence read:
The first clause is right and the last is right; the middle one is not.
values.append((b + 0.5) / bins)replaces every confidence in that region with one number, so a value that lands in an untouchedbin is rewritten and moves. The new text says what the centre does, bounds the move, and names the
identity case the sentence had conflated it with — a bucket the map has no entry for, which
apply_binning_mapgenuinely does return untouched.No behaviour changes.
Why
The two cases are one level apart and the prose merged them:
apply_binning_map(confidence, bucket, map)looks upbinning_map.get(bucket),finds nothing, returns the confidence as it came in. Already checked
(
binning/apply unknown bucket is identity).it. The value comes back as
(b + 0.5) / bins, not as what went in.A reader who took "leaves that region unchanged" at face value reasons wrong about exactly the case
that matters:
fit_abstention_thresholds(..., binning_map=)fits a gate cut on the binned scale,because the runtime rewrites
answer_confidencethrough the map before the gate reads it. An unseenconfidence — one the calibration set never produced, which is what an empty bin means — is the value
whose post-binning number a caller cannot otherwise predict.
Measured, at the shipped default
bins=15, on a bucket whose records all land in one bin:So a confidence in an untouched bin moves by up to 0.0333 — small, but the same order as the
target_error=0.10margin an abstention cut is chosen against, and not zero. The bound is what thedocstring should have given, and now does.
How it was verified
tests/test_calibrate.py: 257 passed, 0 failed (240 at base).scripts/test_suites.py:all 87 suites passed, exit 0. ruff
All checks passed!, compileall silent.The checks build a deterministic map — 200 records all at confidence 0.65, so one bin is filled and
the other nine are empty by construction, no RNG — and then:
0.0an untouched bin holds its centre,exactly the filled bin is not a centre, the fourcomes back as the centreprobes,moves a value by at most half a bin width,no confidence is recalibrated to 0b / bins(left edge)apply_binning_mapreturns the confidence for a centre-valued bincomes back as the centreprobes and the fouris not the value it came in asprobesThe last row is why the docstring is pinned at all: it is the only way this suite distinguishes "the
prose was corrected" from "the prose was rewritten". And the third row is the mutant that would make
the old sentence true — killed, so the four doc checks are not merely enforcing one wording.
grep -rn "binning" docs/ README.mdfinds no prose copy of it), sonothing else goes stale.
tests/test_conformal_abstention.py:230-237pinscheck_min_confidence_map'sdocstring to its refusals the same way, with normalized-whitespace
inspect.getdoc.Checklist
main