Repository navigation
fix(calibrate): fit abstention cuts on the 4-decimal confidences the gate reads - #957
Conversation
…gate reads
`_decode_answers` (Agent and ONNXAgent, and the TS/Java/.NET decoders)
reports `answer_confidence` rounded to 4 decimals, after the binning
lookup, and `apply_confidence_gate` compares that value against the cut.
`fit_abstention_thresholds` chose its cut from full-precision
confidences, so the cut could sit above the reported value of its own
cohort. With a binning map the whole cut bin shares one value: seed 2's
cut is bin 5/6 = 0.83333..., the runtime reports 0.8333, and every
answer in that bin abstains.
Round each confidence to 4 decimals after binning, as the runtime does,
before selecting the cut. The fitter then judges exactly the accepted
set the gate produces on the calibration records, so its error check
(conservative margin included) is about the answers that actually pass.
Measured on 40 seeds x 1,200 synthetic 12-option records,
target_error=0.10, real `_decode_answers` + `apply_confidence_gate`:
binned: before 19/40 seeds drop answers the fitter accepted
(2,411 answers, mean 10.6 / max 13.8 coverage points,
mean coverage 42.6% judged vs 37.6% delivered);
after 0/40, 42.6% judged = 42.6% delivered
unbinned: before 13/40 seeds drop one answer and 11/40 admit one
the fitter never judged; after 0/40 either way
f728836 to
ac2b30d
Compare
|
Merged. The binned case is the one that makes this urgent, and it is a bigger effect than the title suggests. A whole bin sharing one value means a cut that falls inside a bin takes the entire bin with it: the fitter judged 532 accepted answers and the gate delivered 382, a 10.6 coverage point mean gap over 19 of 40 seeds. Anyone who fitted a threshold for a target coverage got a materially different one. What I checked hardest was the risk argument, because rounding before selecting a cut could plausibly have weakened the guarantee rather than fixed the scale. Your reasoning holds: rounding to 4 decimals is monotone, Finishing 09033d1 rather than reversing it is the correct framing. The cut was moved onto the binned scale and left off the reported precision, and the docstring already promised both. The 40-seed table with both binned and unbinned arms, and the unbinned rows showing one answer each way rather than being rounded up into a bigger number, is why this reads as measured instead of argued. Confirming the docstring's own seed 7 numbers do not change is a good check to have run. Checking that no port has its own All 81 suites pass on the merged tree, |
What
fit_abstention_thresholdsnow rounds each calibration confidence to 4 decimals, after the optional binning lookup, before it picks the cut. That is the value_decode_answersreports asanswer_confidenceand the valueapply_confidence_gatecompares. The change is one line inlaya/calibrate.py, plus a regression check intests/test_conformal_abstention.py.Why
The runtime does
round(apply_binning_map(ans_raw, ...), 4): it bins at full precision (3214857), then rounds the public field. Without a map it doesround(ans_raw, 4). This is the same inAgent._decode_answersandONNXAgent. The TS (r4), Java (Rounding.round4) and .NET (Calibration.Round4) decoders also report 4 decimals. The gate testsconf < tauagainst that reported number. The fitter chosetaufrom full-precision confidences, so the cut was not always a value the runtime can report:0.8333333333333334, the runtime reports that bin as0.8333 < tau, and all 150 answers in it abstain. The fitter judged 532 accepted answers. The gate delivers 382.Commit 09033d1 put the cut on the binned scale. This finishes that work by putting it on the reported precision too, so the docstring's promise that thresholds and runtime numbers are "on the same scale" holds.
Why the risk check is preserved, not weakened.
_select_abstention_thresholdjudges each distinct level against the whole set{conf >= tau}, ties included. After the fix,confis the 4-decimal value, so on the calibration records that set is exactly the set the gate accepts: rounding to 4 decimals is monotone, andtauis now one of those values. Rounding merges confidences closer than 1e-4 into one tie group. The fitter then judges them together, which matches the gate, because it cannot tell them apart either. Before the fix, the error the fitter checked (with the conservative +1 margin) applied to a set the gate did not reproduce. After the fix, it applies to the set the gate produces. The thresholds are still empirical cuts, not a formal guarantee, as the docstring says.Measured before / after
40 seeds × 1,200 synthetic 12-option records (the generator this suite already uses),
target_error=0.10. Each run goes through the realAgent._decode_answersandapply_confidence_gate:Seed 2 (binned): before, the cut is
0.8333333333333334, the fitter judges 532/1200 at 0.086 error, and the gate delivers 382/1200 at 0.055. After, the cut is0.8333and both are 532/1200 at 0.086. The numbers in thefit_abstention_thresholdsdocstring (seed 7: 9.8% error at ~50% coverage, 94.5% at 25.6%, binned cut 0.8000) do not change.How verified
tests/test_conformal_abstention.py: seed 2 with a binning map, gated through the real decoder. Onmainit fails 2 checks (cut=0.8333333333333334,got 382 want 532). With the fix all 29 checks pass.python tests/test_conformal_abstention.py: 29 passed, 0 failedpython tests/test_calibrate.py: 215 passed, 0 failedpython tests/test_confidence.py: 134 passed, 0 failedpython tests/test_calibration_persistence.py: 7 tests OKruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120: all checks passedpython -m compileall -q laya/ tests/: cleanNo port has its own
fit_abstention_thresholds. The TS, Java and .NET runtimes consume Python-fitted thresholds and already report 4 decimals, so this PR changes only the Python code.