Repository navigation
feat(calibrate): histogram-binning recalibration for answer_confidence - #871
Conversation
Temperature scaling applies one scalar per `temp_bucket`; it cannot fix a bucket whose reliability curve is not a simple sharpen/soften (the pathological `choice:11+` the shipped English checkpoint carries is one). Add the non-parametric alternative: `fit_binning_map` splits each bucket's calibrated confidences into `bins` equal-width bins over [0, 1] and maps a confidence to its bin's empirical accuracy; `apply_binning_map` looks that up for one answer. NumPy only, no new dependency (isotonic would pull in scikit-learn). It consumes the same `(qtype, logits, target[, k])` records `fit_temperature_map` does and scores the *calibrated* max(p), so a binning map composes on top of a temperature map rather than replacing it. Buckets below `MIN_BINNING_BUCKET_N` are omitted; an empty bin maps to its own midpoint, so an unseen confidence range is left unchanged rather than recalibrated to a fabricated 0. `apply_binning_map` returns the confidence unchanged for a bucket the map was not fit for. tests/test_calibrate.py: bucket keying, sub-floor omission, value range, apply edge/unknown-bucket behaviour, and that binning lowers ECE on a bucket where the fitted temperature alone cannot. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Merged for 0.3.24. The case for this is the one bucket I keep running into: One integration note, since #853 landed in the same pass and both of you added a function in the same place in Verified after that: Thank you. |
…ion payload #871 added fit_binning_map/apply_binning_map as an additive fitter, with runtime wiring left as a deliberate follow-up. This adds the thinnest vertical slice of that follow-up: - calibration payloads can now carry an optional binning_map - apply_calibration_payload validates and installs it; a payload without the key clears the installed map, the same way missing temperature_by_options empties the bucket map - Agent._decode_answers and ONNXAgent._decode_answers apply the map to answer_confidence, keyed by temp_bucket, exactly like temperatures - Agent.fit_binning() fits a map on top of the agent's temperatures; save_calibration writes it back out with the checkpoint identity Tests: payload round-trip + shape validation + decode-level selectability in tests/test_calibrate.py; test_batch and test_packaging still pass.
What
Temperature scaling (
fit_temperature_map) applies one scalar pertemp_bucket— it can only sharpen or soften a bucket uniformly. It cannot fix a bucket whose reliability curve bends the other way, e.g. the pathologicalchoice:11+bucket the shipped English checkpoint carries (a ~0.10 temperature that turns a 0.24 top probability into a published 0.99).This adds the standard non-parametric alternative — histogram binning:
fit_binning_map(records, temperature, temperature_by_options, *, bins=15, min_bucket_n=200)— splits each bucket's calibrated confidences intobinsequal-width bins over[0, 1]and maps a confidence to that bin's empirical accuracy. Returns{bucket: {"bins": N, "values": [...]}}.apply_binning_map(confidence, bucket, binning_map)— looks up the recalibrated confidence for one answer.Design
(qtype, logits, target[, k])recordsfit_temperature_mapdoes and scores the calibratedmax(p)— so you fit temperatures first, then a binning map on top.MIN_BINNING_BUCKET_Nare omitted (a binning map splits a bucket across bins, so it needs more per bucket than a scalar). An empty bin maps to its own midpoint — an unseen confidence range is left unchanged, never recalibrated to a fabricated 0.apply_binning_mapreturns the confidence unchanged for a bucket the map was not fit for.temperature_by_options(common.temp_bucket), so it slots into the same per-bucket calibration model the rest of the pipeline uses.Post-hoc recalibration of the reported
answer_confidence(the surface the abstention gate, the per-option-count thresholds, and the selective-classification metrics all read). Wiring it into_decode_answers/the calibration payload as a selectable method is a natural follow-up; this PR keeps the fitter + apply additive and torch-free.Tests
tests/test_calibrate.py: bucket keying, sub-floor omission, value range, apply edge/unknown-bucket behaviour, and that binning lowers ECE on a bucket where the fitted temperature alone cannot. Full suite: 143 passed. Documented indocs/reference/helpers.md.