Skip to content

feat(calibrate): histogram-binning recalibration for answer_confidence - #871

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
AlKor13:feat/binning-calibration
Oct 2, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
AlKor13:feat/binning-calibration

Conversation

@AlKor13

@AlKor13 AlKor13 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What

Temperature scaling (fit_temperature_map) applies one scalar per temp_bucket — it can only sharpen or soften a bucket uniformly. It cannot fix a bucket whose reliability curve bends the other way, e.g. the pathological choice:11+ bucket the shipped English checkpoint carries (a ~0.10 temperature that turns a 0.24 top probability into a published 0.99).

This adds the standard non-parametric alternative — histogram binning:

  • fit_binning_map(records, temperature, temperature_by_options, *, bins=15, min_bucket_n=200) — splits each bucket's calibrated confidences into bins equal-width bins over [0, 1] and maps a confidence to that bin's empirical accuracy. Returns {bucket: {"bins": N, "values": [...]}}.
  • apply_binning_map(confidence, bucket, binning_map) — looks up the recalibrated confidence for one answer.

Design

  • NumPy only, no new dependency. Isotonic regression would pull in scikit-learn; histogram binning gives the same "fix a non-monotone reliability curve" benefit with the stdlib + NumPy laya already uses.
  • Composes with temperature, doesn't replace it. It consumes the same (qtype, logits, target[, k]) records fit_temperature_map does and scores the calibrated max(p) — so you fit temperatures first, then a binning map on top.
  • Safe edges. Buckets below MIN_BINNING_BUCKET_N are omitted (a binning map splits a bucket across bins, so it needs more per bucket than a scalar). An empty bin maps to its own midpoint — an unseen confidence range is left unchanged, never recalibrated to a fabricated 0. apply_binning_map returns the confidence unchanged for a bucket the map was not fit for.
  • Keyed exactly like temperature_by_options (common.temp_bucket), so it slots into the same per-bucket calibration model the rest of the pipeline uses.

Post-hoc recalibration of the reported answer_confidence (the surface the abstention gate, the per-option-count thresholds, and the selective-classification metrics all read). Wiring it into _decode_answers/the calibration payload as a selectable method is a natural follow-up; this PR keeps the fitter + apply additive and torch-free.

Tests

tests/test_calibrate.py: bucket keying, sub-floor omission, value range, apply edge/unknown-bucket behaviour, and that binning lowers ECE on a bucket where the fitted temperature alone cannot. Full suite: 143 passed. Documented in docs/reference/helpers.md.

Temperature scaling applies one scalar per `temp_bucket`; it cannot fix a bucket
whose reliability curve is not a simple sharpen/soften (the pathological
`choice:11+` the shipped English checkpoint carries is one). Add the non-parametric
alternative: `fit_binning_map` splits each bucket's calibrated confidences into
`bins` equal-width bins over [0, 1] and maps a confidence to its bin's empirical
accuracy; `apply_binning_map` looks that up for one answer.

NumPy only, no new dependency (isotonic would pull in scikit-learn). It consumes
the same `(qtype, logits, target[, k])` records `fit_temperature_map` does and
scores the *calibrated* max(p), so a binning map composes on top of a temperature
map rather than replacing it. Buckets below `MIN_BINNING_BUCKET_N` are omitted; an
empty bin maps to its own midpoint, so an unseen confidence range is left unchanged
rather than recalibrated to a fabricated 0. `apply_binning_map` returns the
confidence unchanged for a bucket the map was not fit for.

tests/test_calibrate.py: bucket keying, sub-floor omission, value range, apply
edge/unknown-bucket behaviour, and that binning lowers ECE on a bucket where the
fitted temperature alone cannot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NandhaKishorM
NandhaKishorM merged commit 98ebcbb into NandhaKishorM:main Oct 2, 2026
25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

The case for this is the one bucket I keep running into: choice:11+ on the shipped English checkpoint ships a temperature of 0.1006, which the runtime clamps to TEMP_MIN = 0.5, so temperature scaling cannot express what that bucket needs at all. A scalar per bucket can only sharpen or soften uniformly, and a reliability curve that bends the other way is exactly what it cannot fix. Histogram binning can.

One integration note, since #853 landed in the same pass and both of you added a function in the same place in laya/calibrate.py: the merge split each new function across two hunks around a shared _iter_records / by_bucket block, so a naive resolution stacked the tail of one onto the other. Both functions now have their own copy of that block, records_from_labeled kept its @torch.no_grad() from #832, and both export sets are in laya/__init__.py and docs/reference/helpers.md.

Verified after that: fit_abstention_thresholds, fit_binning_map and apply_binning_map all import and are in __all__; test_calibrate 192 passed, test_conformal_abstention 20, test_imports (including the mkdocstrings render check) and test_hooks_api 425 pass; full shared test list green; zero decision flips against 0.3.23 on the real checkpoints.

Thank you.

NandhaKishorM pushed a commit that referenced this pull request Oct 3, 2026
…ion payload

#871 added fit_binning_map/apply_binning_map as an additive fitter, with
runtime wiring left as a deliberate follow-up. This adds the thinnest
vertical slice of that follow-up:

- calibration payloads can now carry an optional binning_map
- apply_calibration_payload validates and installs it; a payload without
  the key clears the installed map, the same way missing
  temperature_by_options empties the bucket map
- Agent._decode_answers and ONNXAgent._decode_answers apply the map to
  answer_confidence, keyed by temp_bucket, exactly like temperatures
- Agent.fit_binning() fits a map on top of the agent's temperatures;
  save_calibration writes it back out with the checkpoint identity

Tests: payload round-trip + shape validation + decode-level selectability
in tests/test_calibrate.py; test_batch and test_packaging still pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants