Skip to content

fix(calibrate): fit abstention cuts on the 4-decimal confidences the gate reads - #957

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
sameedkhan17:fix/calibrate-threshold-rounding
Oct 5, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
sameedkhan17:fix/calibrate-threshold-rounding

Conversation

@sameedkhan17

@sameedkhan17 sameedkhan17 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

What

fit_abstention_thresholds now rounds each calibration confidence to 4 decimals, after the optional binning lookup, before it picks the cut. That is the value _decode_answers reports as answer_confidence and the value apply_confidence_gate compares. The change is one line in laya/calibrate.py, plus a regression check in tests/test_conformal_abstention.py.

Why

The runtime does round(apply_binning_map(ans_raw, ...), 4): it bins at full precision (3214857), then rounds the public field. Without a map it does round(ans_raw, 4). This is the same in Agent._decode_answers and ONNXAgent. The TS (r4), Java (Rounding.round4) and .NET (Calibration.Round4) decoders also report 4 decimals. The gate tests conf < tau against that reported number. The fitter chose tau from full-precision confidences, so the cut was not always a value the runtime can report:

  • With a binning map, the whole cut bin is lost. Every answer in a bin shares one value. Seed 2's cut is bin 5/6 = 0.8333333333333334, the runtime reports that bin as 0.8333 < tau, and all 150 answers in it abstain. The fitter judged 532 accepted answers. The gate delivers 382.
  • Without a map, both directions go wrong, but only slightly. The answer at the cut can round below it and be dropped. An answer just below the cut can round up to it and be admitted, and the fitter never counted that answer's error.

Commit 09033d1 put the cut on the binned scale. This finishes that work by putting it on the reported precision too, so the docstring's promise that thresholds and runtime numbers are "on the same scale" holds.

Why the risk check is preserved, not weakened. _select_abstention_threshold judges each distinct level against the whole set {conf >= tau}, ties included. After the fix, conf is the 4-decimal value, so on the calibration records that set is exactly the set the gate accepts: rounding to 4 decimals is monotone, and tau is now one of those values. Rounding merges confidences closer than 1e-4 into one tie group. The fitter then judges them together, which matches the gate, because it cannot tell them apart either. Before the fix, the error the fitter checked (with the conservative +1 margin) applied to a set the gate did not reproduce. After the fix, it applies to the set the gate produces. The thresholds are still empirical cuts, not a formal guarantee, as the docstring says.

Measured before / after

40 seeds × 1,200 synthetic 12-option records (the generator this suite already uses), target_error=0.10. Each run goes through the real Agent._decode_answers and apply_confidence_gate:

seeds dropping fitter-accepted answers seeds admitting unjudged answers mean coverage, fitter vs gate seeds with gate error > 0.10
binned, before 19/40 (2,411 answers; mean 10.6, max 13.8 coverage points) 0/40 42.6% vs 37.6% 0/40
binned, after 0/40 0/40 42.6% vs 42.6% 0/40
unbinned, before 13/40 (1 answer each) 11/40 (1 answer each) 47.7% vs 47.7% 0/40
unbinned, after 0/40 0/40 47.7% vs 47.7% 0/40

Seed 2 (binned): before, the cut is 0.8333333333333334, the fitter judges 532/1200 at 0.086 error, and the gate delivers 382/1200 at 0.055. After, the cut is 0.8333 and both are 532/1200 at 0.086. The numbers in the fit_abstention_thresholds docstring (seed 7: 9.8% error at ~50% coverage, 94.5% at 25.6%, binned cut 0.8000) do not change.

How verified

  • New check in tests/test_conformal_abstention.py: seed 2 with a binning map, gated through the real decoder. On main it fails 2 checks (cut=0.8333333333333334, got 382 want 532). With the fix all 29 checks pass.
  • python tests/test_conformal_abstention.py: 29 passed, 0 failed
  • python tests/test_calibrate.py: 215 passed, 0 failed
  • python tests/test_confidence.py: 134 passed, 0 failed
  • python tests/test_calibration_persistence.py: 7 tests OK
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120: all checks passed
  • python -m compileall -q laya/ tests/: clean

No port has its own fit_abstention_thresholds. The TS, Java and .NET runtimes consume Python-fitted thresholds and already report 4 decimals, so this PR changes only the Python code.

…gate reads

`_decode_answers` (Agent and ONNXAgent, and the TS/Java/.NET decoders)
reports `answer_confidence` rounded to 4 decimals, after the binning
lookup, and `apply_confidence_gate` compares that value against the cut.
`fit_abstention_thresholds` chose its cut from full-precision
confidences, so the cut could sit above the reported value of its own
cohort. With a binning map the whole cut bin shares one value: seed 2's
cut is bin 5/6 = 0.83333..., the runtime reports 0.8333, and every
answer in that bin abstains.

Round each confidence to 4 decimals after binning, as the runtime does,
before selecting the cut. The fitter then judges exactly the accepted
set the gate produces on the calibration records, so its error check
(conservative margin included) is about the answers that actually pass.

Measured on 40 seeds x 1,200 synthetic 12-option records,
target_error=0.10, real `_decode_answers` + `apply_confidence_gate`:

  binned:   before 19/40 seeds drop answers the fitter accepted
            (2,411 answers, mean 10.6 / max 13.8 coverage points,
            mean coverage 42.6% judged vs 37.6% delivered);
            after 0/40, 42.6% judged = 42.6% delivered
  unbinned: before 13/40 seeds drop one answer and 11/40 admit one
            the fitter never judged; after 0/40 either way
@sameedkhan17
sameedkhan17 force-pushed the fix/calibrate-threshold-rounding branch from f728836 to ac2b30d Compare October 5, 2026 13:59
@NandhaKishorM
NandhaKishorM merged commit 6b409f2 into NandhaKishorM:main Oct 5, 2026
0 of 31 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged.

The binned case is the one that makes this urgent, and it is a bigger effect than the title suggests. A whole bin sharing one value means a cut that falls inside a bin takes the entire bin with it: the fitter judged 532 accepted answers and the gate delivered 382, a 10.6 coverage point mean gap over 19 of 40 seeds. Anyone who fitted a threshold for a target coverage got a materially different one.

What I checked hardest was the risk argument, because rounding before selecting a cut could plausibly have weakened the guarantee rather than fixed the scale. Your reasoning holds: rounding to 4 decimals is monotone, tau after the fix is one of the values the runtime can actually report, and _select_abstention_threshold judging each distinct level against the whole {conf >= tau} set means the set the fitter judges is now exactly the set the gate accepts. Merging confidences closer than 1e-4 into one tie group is right rather than a concession, because the gate genuinely cannot tell them apart either. Before the fix the conservative margin was being applied to a set that did not exist downstream, which is the worse of the two situations.

Finishing 09033d1 rather than reversing it is the correct framing. The cut was moved onto the binned scale and left off the reported precision, and the docstring already promised both.

The 40-seed table with both binned and unbinned arms, and the unbinned rows showing one answer each way rather than being rounded up into a bigger number, is why this reads as measured instead of argued. Confirming the docstring's own seed 7 numbers do not change is a good check to have run.

Checking that no port has its own fit_abstention_thresholds, and that TypeScript r4, Java Rounding.round4 and .NET Calibration.Round4 all already report 4 decimals, is what makes this Python-only correctly rather than by omission.

All 81 suites pass on the merged tree, tests/test_calibrate.py and tests/test_conformal_abstention.py included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants