Repository navigation
docs: stop presenting a confidence threshold as permission to act - #378
Merged
NandhaKishorM merged 1 commit intoSep 24, 2026
Conversation
`README.md`'s "Automated Confidence Gating" section was the one place the documentation
told a reader to act on a number the same file says not to trust:
```python
if conf >= 0.85:
# High confidence: automated action without human in the loop
route_automatically(dept)
```
Two lines in the same file contradict that as written:
- the Calibration section, forty lines further up: *"Both checkpoints are over-confident as
shipped"*, and *"`laya-multilingual` ships with no fitted temperatures at all, so fit them
before relying on its probabilities"*;
- the bilingual sweep table, where Khmer answers at 0.952 confidence and 0.000 accuracy, and
Bengali keeps 0.895 mean confidence at 0.45 accuracy.
So `0.85` was not a threshold the model supports; it was a number that looked like one. The
replacement keeps the shape of the example — the branch structure and both calls are
unchanged — and moves the two things that were missing into it: the threshold is the reader's
own measurement, and confidence orders decisions rather than establishing that one is right.
Deliberate choices tested:
- **No policy is invented.** The edit does not say what a threshold should be, or what to gate. It removes an unsupported instruction and points at the material that already exists — the Calibration section above it and the fine-tuning notebook's fitting loop — so the guidance now comes from the project rather than from the reader's guess at what `0.85` meant.
- **The example stays runnable in shape.** `THRESHOLD` replaces the literal so the snippet cannot be pasted unexamined, and the comment above `route_automatically` now says to sample the decisions taken automatically, which is the one practice that makes a wrong threshold recoverable.
- **I removed a name I had invented.** A first draft of this paragraph referred to `laya.calibrate`; no such function exists (`laya.__all__` has nothing matching `calib`, and temperature fitting happens in the notebook). It now links the notebook instead. Flagging it because it is exactly the kind of plausible-looking reference that survives review.
- **Both links resolve**: `#calibration` matches the `### Calibration` heading under the repo's anchor rules, and the notebook path exists.
Second, smaller fix in the same commit, same class of problem:
`docs/docker.md` said training was not available yet — *"Training scripts are tracked in NandhaKishorM#4 and
NandhaKishorM#26; training commands can follow once that interface is available."* Fine-tuning has since
shipped: the notebook is in the repo and linked from the README, and the maintainer confirmed
it in NandhaKishorM#26. A reader deciding whether they can bring their own checkpoint was told to wait for
something that already exists. It now names the notebook and keeps NandhaKishorM#4/NandhaKishorM#26 as the place for
open questions about the interface.
```
$ python tests/test_packaging.py 41 passed, 0 failed
$ python tests/test_doc_tables.py 54 passed, 0 failed
```
Known limitation: this does not add the staged-rollout guidance that [NandhaKishorM#367](NandhaKishorM#367)
asks for. That issue offers three mutually exclusive scopes and asks the maintainer to choose
one, so writing it here would mean inventing promotion criteria nobody has endorsed. What this
does is remove the part of the README that argued against the issue's own premise.
Owner
|
Thank you, this is a valuable correction: the old example showed readers acting on a 0.85 cutoff that the calibration section itself says not to trust. The new text points readers to their own held-out data, and the docker.md fix removes a stale note about fine-tuning. Your measurement showing the same threshold flipping from perfect to worse than no gate was a very convincing argument. Merging. |
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 29, 2026
`laya.answer_confidence` is in `laya.__all__`, has a 615-character docstring, and appears in **no page in `docs/`** — while the function it warns you not to confuse it with, `laya.common.confidence_from_probs`, is rendered directly above where it belongs. The reference is generated from the docstrings, so a public name with no directive is simply invisible, and this was the only one in that state: ``` $ python scratch/check_ref_gaps.py === public names with NO directive anywhere === BaseHook -> laya.hooks.BaseHook DOCSTRING 253 chars DecisionResult -> laya.structured.DecisionResult Hook -> laya.hooks.Hook PredictContext -> laya.hooks.PredictContext PredictHook -> typing.Callable answer_confidence -> laya.common.answer_confidence DOCSTRING 615 chars decide -> laya.structured.decide ``` The other six are already covered by hand-written pages and are correctly absent from the generated index — `PredictContext`, `Hook`, `BaseHook` and `PredictHook` have [`hooks/api.md`](../hooks/api.md), which `reference/index.md` links to, and `decide` and `DecisionResult` have [`structured.md`](../structured.md) and are not public names a reader looks up in isolation. `answer_confidence` had neither. It is worth rendering rather than leaving to the reader, because it is the field a caller is supposed to gate on. Its docstring is the only place in the codebase that states the distinction: > This is the quantity temperature scaling fits, and the quantity every calibration figure in this > repository is computed on -- both benchmark harnesses take `conf = max(probs)` before calling > `ece_score`. It is therefore the one confidence with the property the README's gating section > relies on: of the answers returned at confidence c, about c of them are right. > > `confidence_from_probs` below reports a different quantity on a different scale and carries no > such guarantee, so the two must not be compared against the same threshold. That is exactly the material a reader needs when they find `confidence` and `answer_confidence` side by side in one answer object and have to pick, and until now it was reachable only by reading the source. Deliberate choices tested: - **One directive, in the section it belongs to.** It goes with `confidence_from_probs` under "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other. - **No prose added to the page.** The docstring already says it better than a summary would, and the page is generated — an added paragraph would be a second copy to keep in sync. - **The other six names are deliberately not added.** Four would duplicate `hooks/api.md`, which is hand-written precisely because hooks need more than a signature dump; two are covered by `structured.md`. Adding them would move documentation away from where it is maintained, which is the failure mode `reference/index.md` already calls out. ``` $ python -m zensical build --strict --clean # exit 0, no griffe lines answer_confidence rendered: True confidence_from_probs: True $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed ``` Known limitation: this is the generated-reference gap only. It does not add `answer_confidence` to the README's gating section, which is [#378](#378) and the [Questions and answers guide](#418); if those land first, this remains the API-level entry they both point at.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs: stop presenting a confidence threshold as permission to act
README.md's "Automated Confidence Gating" section was the one place the documentationtold a reader to act on a number the same file says not to trust:
Two lines in the same file contradict that as written:
shipped", and "
laya-multilingualships with no fitted temperatures at all, so fit thembefore relying on its probabilities";
Bengali keeps 0.895 mean confidence at 0.45 accuracy.
So
0.85was not a threshold the model supports; it was a number that looked like one. Thereplacement keeps the shape of the example — the branch structure and both calls are
unchanged — and moves the two things that were missing into it: the threshold is the reader's
own measurement, and confidence orders decisions rather than establishing that one is right.
Deliberate choices tested:
0.85meant.THRESHOLDreplaces the literal so the snippet cannot be pasted unexamined, and the comment aboveroute_automaticallynow says to sample the decisions taken automatically, which is the one practice that makes a wrong threshold recoverable.laya.calibrate; no such function exists (laya.__all__has nothing matchingcalib, and temperature fitting happens in the notebook). It now links the notebook instead. Flagging it because it is exactly the kind of plausible-looking reference that survives review.#calibrationmatches the### Calibrationheading under the repo's anchor rules, and the notebook path exists.Since opening this I have measured what a threshold selects, which is the evidence the
replacement text was arguing from. Twelve states whose content makes exactly one option right,
asked as a
choicequestion at two option counts,laya(English), laya 0.3.20, CPU:>= 0.85At 20 options the gate at the old literal selects 75% of cases at 55.6% precision, below the
model's own 58.3% accuracy on the same set — it is worse than not gating. At 3 options the
same number is perfect. That is the concrete form of "a threshold is a policy you choose": it is
not merely uncertified at this option count, it is inverted there. The full measurement, and why
choice:11+cannot be right across the 11-or-more range it covers, is in#394.
The 12 cases are ones I wrote, so those absolute accuracies are not benchmark numbers; what
transfers is the direction, and it agrees with the committed sweep's own Khmer and Bengali rows
above.
Second, smaller fix in the same commit, same class of problem:
docs/docker.mdsaid training was not available yet — "Training scripts are tracked in #4 and#26; training commands can follow once that interface is available." Fine-tuning has since
shipped: the notebook is in the repo and linked from the README, and the maintainer confirmed
it in #26. A reader deciding whether they can bring their own checkpoint was told to wait for
something that already exists. It now names the notebook and keeps #4/#26 as the place for
open questions about the interface.
Known limitation: this does not add the staged-rollout guidance that #367
asks for. That issue offers three mutually exclusive scopes and asks the maintainer to choose
one, so writing it here would mean inventing promotion criteria nobody has endorsed. What this
does is remove the part of the README that argued against the issue's own premise.