Repository navigation
docs(reference): render answer_confidence, which no page documented - #419
NandhaKishorM merged 2 commits into
Conversation
`laya.answer_confidence` is in `laya.__all__`, has a 615-character docstring, and appears in **no page in `docs/`** — while the function it warns you not to confuse it with, `laya.common.confidence_from_probs`, is rendered directly above where it belongs. The reference is generated from the docstrings, so a public name with no directive is simply invisible, and this was the only one in that state: ``` $ python scratch/check_ref_gaps.py === public names with NO directive anywhere === BaseHook -> laya.hooks.BaseHook DOCSTRING 253 chars DecisionResult -> laya.structured.DecisionResult Hook -> laya.hooks.Hook PredictContext -> laya.hooks.PredictContext PredictHook -> typing.Callable answer_confidence -> laya.common.answer_confidence DOCSTRING 615 chars decide -> laya.structured.decide ``` The other six are already covered by hand-written pages and are correctly absent from the generated index — `PredictContext`, `Hook`, `BaseHook` and `PredictHook` have [`hooks/api.md`](../hooks/api.md), which `reference/index.md` links to, and `decide` and `DecisionResult` have [`structured.md`](../structured.md) and are not public names a reader looks up in isolation. `answer_confidence` had neither. It is worth rendering rather than leaving to the reader, because it is the field a caller is supposed to gate on. Its docstring is the only place in the codebase that states the distinction: > This is the quantity temperature scaling fits, and the quantity every calibration figure in this > repository is computed on -- both benchmark harnesses take `conf = max(probs)` before calling > `ece_score`. It is therefore the one confidence with the property the README's gating section > relies on: of the answers returned at confidence c, about c of them are right. > > `confidence_from_probs` below reports a different quantity on a different scale and carries no > such guarantee, so the two must not be compared against the same threshold. That is exactly the material a reader needs when they find `confidence` and `answer_confidence` side by side in one answer object and have to pick, and until now it was reachable only by reading the source. Deliberate choices tested: - **One directive, in the section it belongs to.** It goes with `confidence_from_probs` under "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other. - **No prose added to the page.** The docstring already says it better than a summary would, and the page is generated — an added paragraph would be a second copy to keep in sync. - **The other six names are deliberately not added.** Four would duplicate `hooks/api.md`, which is hand-written precisely because hooks need more than a signature dump; two are covered by `structured.md`. Adding them would move documentation away from where it is maintained, which is the failure mode `reference/index.md` already calls out. ``` $ python -m zensical build --strict --clean # exit 0, no griffe lines answer_confidence rendered: True confidence_from_probs: True $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed ``` Known limitation: this is the generated-reference gap only. It does not add `answer_confidence` to the README's gating section, which is [NandhaKishorM#378](NandhaKishorM#378) and the [Questions and answers guide](NandhaKishorM#418); if those land first, this remains the API-level entry they both point at.
Bruce-Yii
left a comment
There was a problem hiding this comment.
Could you qualify the generated reference before merging? This PR makes the current answer_confidence docstring canonical API guidance, including the unconditional sentence that answers returned at confidence c are correct about c of the time. The function computes max(p); that identifies the calibration quantity, but does not establish that the shipped probabilities are calibrated.
Current main says both checkpoints ship over-confident, laya-multilingual has no fitted temperatures, and confidence does not establish correctness. The synthetic confidence test establishes the property only on data generated to be calibrated, while #394 shows a documented 0.85 gate at 20 options can select below the model's own accuracy.
I would describe this as the calibration target/quantity and condition the c ≈ correctness property on a fit validated for the same checkpoint, question type, option-count regime, and workload. Since #418 repeats the same wording, aligning both would avoid preserving the contradiction in two new guides.
Evidence checked 2026-09-25 05:41:51 (Beijing).
|
Thank you for finding the one public name the reference never rendered, it is the field people should gate on. Before merging, could you update the |
The docstring of `answer_confidence` points at `confidence_from_probs` "below", but the new directive was inserted after that one, so mkdocstrings rendered it above. Move the directive up rather than reword the sentence, which keeps the sibling "See `answer_confidence`" reference reading correctly too. Also record on `answer_confidence` that the "of the answers returned at confidence c, about c of them are right" property is conditional: it holds only after the temperatures have been fitted and validated on held-out data for this checkpoint and this option count. The shipped checkpoints are over-confident -- `choice:11+` is a ~10x sharpener that returns a point mass at 1.0 -- so a threshold applied to them selects below model accuracy (issue NandhaKishorM#394).
`answer_confidence` is the *quantity* temperature scaling fits and every calibration figure in
this repository is computed on. It is not a claim that the number is right. The first commit
of this branch said "calibrated confidence", "calibrated max(p)", "how much should I trust
this answer", and named the helper `calibrated_confidence`. All four overstate it, and the
README is explicit that they should:
Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted
temperatures at all, so fit them before relying on these numbers ... Confidence orders
decisions; it does not establish that a decision is correct.
Reading `answer_confidence` as "about c of the answers returned at c are correct" holds only
after temperatures have been fitted and validated on held-out data for that checkpoint and
question shape. Naming a public helper after a property the shipped checkpoints do not have
reintroduces exactly the ambiguity NandhaKishorM#419 caught: the quantity that calibration targets is not
the same claim as a calibrated model.
What changes:
- `calibrated_confidence(answer)` -> `answer_confidence_value(answer)`. A name that says which
field it reads and asserts nothing about it. Still not in the top-level `__all__`.
- Every docstring and comment that said "calibrated" now says what the number is (`max(p)`, the
probability mass on the reported answer; the quantity temperature scaling fits and ECE
evaluates) and states the condition under which it means c-approximates-correctness. The
`DecisionResult` docstring, `laya/confidence.py`, `docs/structured.md` and the tests all say
the same thing, including that the shipped checkpoints are over-confident.
- The product argument does not change and is now stated more precisely. `answer_confidence` is
the number to filter on **because it is the same quantity `min_confidence` compares against
and the eval and calibration stack measures** -- not because it is more trustworthy. That is a
stronger claim than the old one, because it is checkable: the gate and this field read one
helper, so they cannot disagree about which number is being reported.
- `docs/structured.md` no longer claims the bridge returns "calibrated confidence", and its
example says explicitly that filtering on the right number is not the same as it being a
trustworthy probability, with links to the README's Calibration and Honest limits sections.
- Fixed a dangling `[Calibration](#calibration)` anchor the docs gate caught; that section is in
the README, so it is a cross-doc link now.
The field, the `None` semantics, the no-fallback rule and the `confidence` field's unchanged
value and default all stay as they were -- this is a naming and semantics correction only.
`flag_low_confidence` still has the same signature, semantics and 0.0 no-op.
Unchanged: tests/test_structured.py 114 passed, tests/test_structured_api.py 38,
tests/test_confidence.py 59, tests/test_packaging.py 106, tests/test_structured_docs.py 87,
tests/test_doc_tables.py 109, tests/test_hooks_api.py 387, tests/test_hooks.py 204,
tests/test_langchain.py 207. Full ci.yml suite list, 62 suites, 5 failures -- the same five
that fail identically on unmodified 9d95567 in this environment. ruff, compileall,
git diff --check and zensical build --strict --clean all clean.
|
Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22. |
docs(reference): render
answer_confidence, which no page documentedlaya.answer_confidenceis inlaya.__all__, has a 615-character docstring, and appears inno page in
docs/— while the function it warns you not to confuse it with,laya.common.confidence_from_probs, is rendered directly above where it belongs. The referenceis generated from the docstrings, so a public name with no directive is simply invisible, and
this was the only one in that state:
The other six are already covered by hand-written pages and are correctly absent from the
generated index —
PredictContext,Hook,BaseHookandPredictHookhavehooks/api.md, whichreference/index.mdlinks to, anddecideandDecisionResulthavestructured.mdand are not public names a readerlooks up in isolation.
answer_confidencehad neither.It is worth rendering rather than leaving to the reader, because it is the field a caller is
supposed to gate on. Its docstring is the only place in the codebase that states the distinction:
That is exactly the material a reader needs when they find
confidenceandanswer_confidenceside by side in one answer object and have to pick, and until now it was reachable only by
reading the source.
Deliberate choices tested:
confidence_from_probsunder "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other.hooks/api.md, which is hand-written precisely because hooks need more than a signature dump; two are covered bystructured.md. Adding them would move documentation away from where it is maintained, which is the failure modereference/index.mdalready calls out.Known limitation: this is the generated-reference gap only. It does not add
answer_confidenceto the README's gating section, which is #378
and the Questions and answers guide; if those
land first, this remains the API-level entry they both point at.