Repository navigation
docs(readme): the abstention cut is not option-count invariant, and the map is the fix - #1098
Conversation
…he map is the fix README's gating section called `answer_confidence` "the calibrated quantity, invariant to the number of options" while README's own comparison section says both confidence values shift with the option count, and `fit_abstention_thresholds` says a single cut does not transfer (NandhaKishorM#394). The committed T4 run settles it: on `laya` the mean of max(p) is 0.7677 over 3-option XNLI against 0.9659 over 20-option MASSIVE intent, and on `laya-multilingual` it runs the other way (0.9152 against 0.8055). Same field, same run, opposite ordering between checkpoints -- so no flat cutoff carries one meaning across shapes, and `flag_low_confidence`'s docstring said the same wrong thing. The per-bucket map is the control that answers this, is implemented in core, is tested, and is documented for the HTTP API -- but it appears nowhere in the section a Python caller reads. Documented it there with the fitter that produces the numbers, and corrected the invariance claim at both sites.
|
Merged in 0.4.3. Both halves of the sentence were wrong and in different ways, which is why this is worth 21 lines. "Invariant to the number of options" is contradicted 253 lines earlier in the same file, by "Both values shift with the number of options". A README that states a property and its negation leaves a reader to guess which paragraph was written later. "Calibrated" is the one I would have flagged first, because the repository already holds that line elsewhere and this section broke it. README's own Calibration section states mean ECE 0.466 as shipped, and Documenting the per-bucket Verified on the merged tree: |
What
Corrects one false sentence in README's abstention section, says the same correction in
laya/confidence.py::flag_low_confidence, and documents the per-bucketmin_confidencemap there for the first time.README's gating section described
answer_confidenceas "the calibrated quantity, invariant to the number of options". It is neither:tests/test_confidence.pyfailsexamples/18_confidence_gating.pyfor writing "the calibrated probability" unqualified. The section that introduces the gate used the wording the suite bans on the example page.laya/calibrate.py::fit_abstention_thresholds, whose first line is "A singlemin_confidencedoes not transfer across option counts (A confidence threshold does not transfer across option counts: at 20 options the gate at 0.85 selects below model accuracy #394)".The map form is implemented in core (
check_min_confidence_map,resolve_min_confidence,apply_confidence_gate), fitted bylaya/calibrate.py::fit_abstention_thresholds, tested intests/test_conformal_abstention.py, and documented indocs/http-api.md— and it appeared nowhere in the section a Python caller reads. The README now carries the example, the twelve-key vocabulary, thedefaultfallthrough, the validation refusal, and the pointer to the fitter.Why
#776's calibration finding is the same complaint from the outside: at a 0.90 accuracy target the advised cut measured 0.8074 for 3-option XNLI and 0.9944 for 20-option MASSIVE. The repo had already answered that — the map is exactly the mechanism — but the most-read page told callers the question could not arise. A reader who trusts the old sentence sets one global threshold, and the sentence's own example (
min_confidence=0.85) then means two different things in one batch.The claim was checkable without a model, from the committed run, so it was checked.
research/results/t4_colab_benchmark.jsonstoresmean_confidenceper suite, computed asfloat(np.max(p))per question — the field the gate reads (research/scripts/laya_benchmark_colab.ipynb, thebench_enginecell). Averaged over the suites of each family:layalaya-multilingualSame field, same run, and the two checkpoints order the option counts in opposite directions — so the fix could not be "say which one is higher", only "say that one cut cannot serve both, and point at the map". The macro accuracies my per-suite means produce also reproduce the artifact's own stored summary exactly (
laya: xnli 0.5436, massive_intent 0.3405), which is what makes the aggregation trustworthy rather than my choice of it.How it was verified
The README text is now pinned against live behavior, no model loaded:
The README claims the map is available on the five listed methods across three classes;
inspect.signatureconfirms all 15 pairs carrymin_confidence, and no path coerces it (grepforfloat(mc/float(min_confidencefinds none), so the dict survives toresolve_min_confidence— including throughlaya.serve, which maps a bad key to a 422.Two disclosures:
zensicalis not installed here, so the docs strict build is the first place the rewordedflag_low_confidencedocstring renders — the edit is prose inside an existing docstring, with no structure change. And README's other half of the contradiction (line 776, "fit and validate any cutoff … at the option counts your workload uses") is the natural place to point at the map too, but #1006's open hunk sits at line 777 on its trailing context, so I left it rather than guarantee a conflict. It is a one-clause follow-up.Ref #394, #776.
Checklist
mainRef #394