Skip to content

docs(readme): the abstention cut is not option-count invariant, and the map is the fix - #1098

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:docs-abstention-option-count
Oct 11, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:docs-abstention-option-count

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Corrects one false sentence in README's abstention section, says the same correction in laya/confidence.py::flag_low_confidence, and documents the per-bucket min_confidence map there for the first time.

README's gating section described answer_confidence as "the calibrated quantity, invariant to the number of options". It is neither:

  • calibrated is a property a fit earns, not a property of the field, and this repo already holds that line — README's own Calibration section states mean ECE 0.466 as shipped, and tests/test_confidence.py fails examples/18_confidence_gating.py for writing "the calibrated probability" unqualified. The section that introduces the gate used the wording the suite bans on the example page.
  • invariant to the number of options is contradicted 253 lines earlier in the same file ("Both values shift with the number of options"), and by laya/calibrate.py::fit_abstention_thresholds, whose first line is "A single min_confidence does not transfer across option counts (A confidence threshold does not transfer across option counts: at 20 options the gate at 0.85 selects below model accuracy #394)".

The map form is implemented in core (check_min_confidence_map, resolve_min_confidence, apply_confidence_gate), fitted by laya/calibrate.py::fit_abstention_thresholds, tested in tests/test_conformal_abstention.py, and documented in docs/http-api.md — and it appeared nowhere in the section a Python caller reads. The README now carries the example, the twelve-key vocabulary, the default fallthrough, the validation refusal, and the pointer to the fitter.

Why

#776's calibration finding is the same complaint from the outside: at a 0.90 accuracy target the advised cut measured 0.8074 for 3-option XNLI and 0.9944 for 20-option MASSIVE. The repo had already answered that — the map is exactly the mechanism — but the most-read page told callers the question could not arise. A reader who trusts the old sentence sets one global threshold, and the sentence's own example (min_confidence=0.85) then means two different things in one batch.

The claim was checkable without a model, from the committed run, so it was checked. research/results/t4_colab_benchmark.json stores mean_confidence per suite, computed as float(np.max(p)) per question — the field the gate reads (research/scripts/laya_benchmark_colab.ipynb, the bench_engine cell). Averaged over the suites of each family:

checkpoint 3-option XNLI 20-option MASSIVE intent
laya 0.7677 0.9659
laya-multilingual 0.9152 0.8055

Same field, same run, and the two checkpoints order the option counts in opposite directions — so the fix could not be "say which one is higher", only "say that one cut cannot serve both, and point at the map". The macro accuracies my per-suite means produce also reproduce the artifact's own stored summary exactly (laya: xnli 0.5436, massive_intent 0.3405), which is what makes the aggregation trustworthy rather than my choice of it.

How it was verified

python -B scripts/test_suites.py           # all 86 suites passed (rc 0)
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/       # rc 0

The README text is now pinned against live behavior, no model loaded:

bucket_keys -> 12  ['choice:11+', 'choice:2', 'choice:3-5', 'choice:6-10', 'noul:11+', 'noul:2', ...]
flat 0.85      -> {'q1': ('passed', 0.85), 'q2': ('abstained', 0.85)}          # 3- and 20-option answers
per-bucket map -> {'q1': ('passed', 0.9), 'q2': ('abstained', 0.98)}           # abstention_threshold is per-bucket
only choice:11+ named -> {'q1': ('passed', 0.0), 'q2': ('abstained', 0.98)}    # the ungated-by-omission case
refused {'choice:4-6': 0.9} -> "min_confidence map keys must be a bucket like 'choice:3-5' or 'default'"
refused {'choice:3-5': 'high'} -> "min_confidence must be a float in [0.0, 1.0]"
refused {} -> "a min_confidence map must be a non-empty dict"

The README claims the map is available on the five listed methods across three classes; inspect.signature confirms all 15 pairs carry min_confidence, and no path coerces it (grep for float(mc / float(min_confidence finds none), so the dict survives to resolve_min_confidence — including through laya.serve, which maps a bad key to a 422.

Two disclosures: zensical is not installed here, so the docs strict build is the first place the reworded flag_low_confidence docstring renders — the edit is prose inside an existing docstring, with no structure change. And README's other half of the contradiction (line 776, "fit and validate any cutoff … at the option counts your workload uses") is the natural place to point at the map too, but #1006's open hunk sits at line 777 on its trailing context, so I left it rather than guarantee a conflict. It is a one-clause follow-up.

Ref #394, #776.

Checklist

  • Focused on one change (one wrong claim corrected, one undocumented control documented — both in the same section)
  • Rebased on the latest main
  • Tests pass locally (all 86 suites, rc 0)
  • Docs or examples updated when the public API changed (no API change; the public control was undocumented in README)

Ref #394

…he map is the fix

README's gating section called `answer_confidence` "the calibrated quantity, invariant
to the number of options" while README's own comparison section says both confidence
values shift with the option count, and `fit_abstention_thresholds` says a single cut
does not transfer (NandhaKishorM#394). The committed T4 run settles it: on `laya` the mean of
max(p) is 0.7677 over 3-option XNLI against 0.9659 over 20-option MASSIVE intent, and
on `laya-multilingual` it runs the other way (0.9152 against 0.8055). Same field, same
run, opposite ordering between checkpoints -- so no flat cutoff carries one meaning
across shapes, and `flag_low_confidence`'s docstring said the same wrong thing.

The per-bucket map is the control that answers this, is implemented in core, is tested,
and is documented for the HTTP API -- but it appears nowhere in the section a Python
caller reads. Documented it there with the fitter that produces the numbers, and
corrected the invariance claim at both sites.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.3. Both halves of the sentence were wrong and in different ways, which is why this is worth 21 lines.

"Invariant to the number of options" is contradicted 253 lines earlier in the same file, by "Both values shift with the number of options". A README that states a property and its negation leaves a reader to guess which paragraph was written later.

"Calibrated" is the one I would have flagged first, because the repository already holds that line elsewhere and this section broke it. README's own Calibration section states mean ECE 0.466 as shipped, and tests/test_confidence.py fails examples/18_confidence_gating.py for writing "the calibrated probability" unqualified. The section introducing the gate was using the exact wording the suite bans on an example page, so the rule existed and the most-read surface was exempt from it by accident.

Documenting the per-bucket min_confidence map in flag_low_confidence is the part that turns a correction into an answer. Saying the cut is not option-count invariant without naming the mechanism that handles it leaves the reader with a problem; naming the map gives them the fix in the same breath.

Verified on the merged tree: tests/test_confidence.py passes, all 87 suites pass, and the 94-row real-weights decision diff is identical to 68804629.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants