Skip to content

docs(reference): render answer_confidence, which no page documented - #419

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
PerryLink:docs-reference-answer-confidence
Sep 29, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
PerryLink:docs-reference-answer-confidence

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

docs(reference): render answer_confidence, which no page documented

laya.answer_confidence is in laya.__all__, has a 615-character docstring, and appears in
no page in docs/ — while the function it warns you not to confuse it with,
laya.common.confidence_from_probs, is rendered directly above where it belongs. The reference
is generated from the docstrings, so a public name with no directive is simply invisible, and
this was the only one in that state:

$ python scratch/check_ref_gaps.py
=== public names with NO directive anywhere ===
  BaseHook               -> laya.hooks.BaseHook         DOCSTRING 253 chars
  DecisionResult         -> laya.structured.DecisionResult
  Hook                   -> laya.hooks.Hook
  PredictContext         -> laya.hooks.PredictContext
  PredictHook            -> typing.Callable
  answer_confidence      -> laya.common.answer_confidence   DOCSTRING 615 chars
  decide                 -> laya.structured.decide

The other six are already covered by hand-written pages and are correctly absent from the
generated index — PredictContext, Hook, BaseHook and PredictHook have
hooks/api.md, which reference/index.md links to, and decide and
DecisionResult have structured.md and are not public names a reader
looks up in isolation. answer_confidence had neither.

It is worth rendering rather than leaving to the reader, because it is the field a caller is
supposed to gate on. Its docstring is the only place in the codebase that states the distinction:

This is the quantity temperature scaling fits, and the quantity every calibration figure in this
repository is computed on -- both benchmark harnesses take conf = max(probs) before calling
ece_score. It is therefore the one confidence with the property the README's gating section
relies on: of the answers returned at confidence c, about c of them are right.

confidence_from_probs below reports a different quantity on a different scale and carries no
such guarantee, so the two must not be compared against the same threshold.

That is exactly the material a reader needs when they find confidence and answer_confidence
side by side in one answer object and have to pick, and until now it was reachable only by
reading the source.

Deliberate choices tested:

  • One directive, in the section it belongs to. It goes with confidence_from_probs under "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other.
  • No prose added to the page. The docstring already says it better than a summary would, and the page is generated — an added paragraph would be a second copy to keep in sync.
  • The other six names are deliberately not added. Four would duplicate hooks/api.md, which is hand-written precisely because hooks need more than a signature dump; two are covered by structured.md. Adding them would move documentation away from where it is maintained, which is the failure mode reference/index.md already calls out.
$ python -m zensical build --strict --clean      # exit 0, no griffe lines
  answer_confidence rendered:  True
  confidence_from_probs:       True
$ python tests/test_packaging.py                 all packaging tests passed
$ python tests/test_doc_tables.py                54 passed, 0 failed

Known limitation: this is the generated-reference gap only. It does not add answer_confidence
to the README's gating section, which is #378
and the Questions and answers guide; if those
land first, this remains the API-level entry they both point at.

`laya.answer_confidence` is in `laya.__all__`, has a 615-character docstring, and appears in
**no page in `docs/`** — while the function it warns you not to confuse it with,
`laya.common.confidence_from_probs`, is rendered directly above where it belongs. The reference
is generated from the docstrings, so a public name with no directive is simply invisible, and
this was the only one in that state:

```
$ python scratch/check_ref_gaps.py
=== public names with NO directive anywhere ===
  BaseHook               -> laya.hooks.BaseHook         DOCSTRING 253 chars
  DecisionResult         -> laya.structured.DecisionResult
  Hook                   -> laya.hooks.Hook
  PredictContext         -> laya.hooks.PredictContext
  PredictHook            -> typing.Callable
  answer_confidence      -> laya.common.answer_confidence   DOCSTRING 615 chars
  decide                 -> laya.structured.decide
```

The other six are already covered by hand-written pages and are correctly absent from the
generated index — `PredictContext`, `Hook`, `BaseHook` and `PredictHook` have
[`hooks/api.md`](../hooks/api.md), which `reference/index.md` links to, and `decide` and
`DecisionResult` have [`structured.md`](../structured.md) and are not public names a reader
looks up in isolation. `answer_confidence` had neither.

It is worth rendering rather than leaving to the reader, because it is the field a caller is
supposed to gate on. Its docstring is the only place in the codebase that states the distinction:

> This is the quantity temperature scaling fits, and the quantity every calibration figure in this
> repository is computed on -- both benchmark harnesses take `conf = max(probs)` before calling
> `ece_score`. It is therefore the one confidence with the property the README's gating section
> relies on: of the answers returned at confidence c, about c of them are right.
>
> `confidence_from_probs` below reports a different quantity on a different scale and carries no
> such guarantee, so the two must not be compared against the same threshold.

That is exactly the material a reader needs when they find `confidence` and `answer_confidence`
side by side in one answer object and have to pick, and until now it was reachable only by
reading the source.

Deliberate choices tested:

- **One directive, in the section it belongs to.** It goes with `confidence_from_probs` under "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other.
- **No prose added to the page.** The docstring already says it better than a summary would, and the page is generated — an added paragraph would be a second copy to keep in sync.
- **The other six names are deliberately not added.** Four would duplicate `hooks/api.md`, which is hand-written precisely because hooks need more than a signature dump; two are covered by `structured.md`. Adding them would move documentation away from where it is maintained, which is the failure mode `reference/index.md` already calls out.

```
$ python -m zensical build --strict --clean      # exit 0, no griffe lines
  answer_confidence rendered:  True
  confidence_from_probs:       True
$ python tests/test_packaging.py                 all packaging tests passed
$ python tests/test_doc_tables.py                54 passed, 0 failed
```

Known limitation: this is the generated-reference gap only. It does not add `answer_confidence`
to the README's gating section, which is [NandhaKishorM#378](NandhaKishorM#378)
and the [Questions and answers guide](NandhaKishorM#418); if those
land first, this remains the API-level entry they both point at.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you qualify the generated reference before merging? This PR makes the current answer_confidence docstring canonical API guidance, including the unconditional sentence that answers returned at confidence c are correct about c of the time. The function computes max(p); that identifies the calibration quantity, but does not establish that the shipped probabilities are calibrated.

Current main says both checkpoints ship over-confident, laya-multilingual has no fitted temperatures, and confidence does not establish correctness. The synthetic confidence test establishes the property only on data generated to be calibrated, while #394 shows a documented 0.85 gate at 20 options can select below the model's own accuracy.

I would describe this as the calibration target/quantity and condition the c ≈ correctness property on a fit validated for the same checkpoint, question type, option-count regime, and workload. Since #418 repeats the same wording, aligning both would avoid preserving the contradiction in two new guides.

Evidence checked 2026-09-25 05:41:51 (Beijing).

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for finding the one public name the reference never rendered, it is the field people should gate on. Before merging, could you update the answer_confidence docstring in laya/common.py so the 'about c of them are right' sentence is conditional: it describes the quantity temperature scaling fits, and it holds only after temperatures are fitted and validated on held-out data for that checkpoint and option count, since the shipped checkpoints are over-confident (see #394). Also, the docstring says confidence_from_probs is 'below', but on the page it now renders above, so please put the new directive before confidence_from_probs or reword that line.

The docstring of `answer_confidence` points at `confidence_from_probs` "below", but
the new directive was inserted after that one, so mkdocstrings rendered it above.
Move the directive up rather than reword the sentence, which keeps the sibling
"See `answer_confidence`" reference reading correctly too.

Also record on `answer_confidence` that the "of the answers returned at confidence
c, about c of them are right" property is conditional: it holds only after the
temperatures have been fitted and validated on held-out data for this checkpoint
and this option count. The shipped checkpoints are over-confident -- `choice:11+`
is a ~10x sharpener that returns a point mass at 1.0 -- so a threshold applied to
them selects below model accuracy (issue NandhaKishorM#394).
Bruce-Yii added a commit to Bruce-Yii/laya that referenced this pull request Sep 28, 2026
`answer_confidence` is the *quantity* temperature scaling fits and every calibration figure in
this repository is computed on. It is not a claim that the number is right. The first commit
of this branch said "calibrated confidence", "calibrated max(p)", "how much should I trust
this answer", and named the helper `calibrated_confidence`. All four overstate it, and the
README is explicit that they should:

    Both checkpoints are over-confident as shipped and `laya-multilingual` has no fitted
    temperatures at all, so fit them before relying on these numbers ... Confidence orders
    decisions; it does not establish that a decision is correct.

Reading `answer_confidence` as "about c of the answers returned at c are correct" holds only
after temperatures have been fitted and validated on held-out data for that checkpoint and
question shape. Naming a public helper after a property the shipped checkpoints do not have
reintroduces exactly the ambiguity NandhaKishorM#419 caught: the quantity that calibration targets is not
the same claim as a calibrated model.

What changes:

- `calibrated_confidence(answer)` -> `answer_confidence_value(answer)`. A name that says which
  field it reads and asserts nothing about it. Still not in the top-level `__all__`.
- Every docstring and comment that said "calibrated" now says what the number is (`max(p)`, the
  probability mass on the reported answer; the quantity temperature scaling fits and ECE
  evaluates) and states the condition under which it means c-approximates-correctness. The
  `DecisionResult` docstring, `laya/confidence.py`, `docs/structured.md` and the tests all say
  the same thing, including that the shipped checkpoints are over-confident.
- The product argument does not change and is now stated more precisely. `answer_confidence` is
  the number to filter on **because it is the same quantity `min_confidence` compares against
  and the eval and calibration stack measures** -- not because it is more trustworthy. That is a
  stronger claim than the old one, because it is checkable: the gate and this field read one
  helper, so they cannot disagree about which number is being reported.
- `docs/structured.md` no longer claims the bridge returns "calibrated confidence", and its
  example says explicitly that filtering on the right number is not the same as it being a
  trustworthy probability, with links to the README's Calibration and Honest limits sections.
- Fixed a dangling `[Calibration](#calibration)` anchor the docs gate caught; that section is in
  the README, so it is a cross-doc link now.

The field, the `None` semantics, the no-fallback rule and the `confidence` field's unchanged
value and default all stay as they were -- this is a naming and semantics correction only.
`flag_low_confidence` still has the same signature, semantics and 0.0 no-op.

Unchanged: tests/test_structured.py 114 passed, tests/test_structured_api.py 38,
tests/test_confidence.py 59, tests/test_packaging.py 106, tests/test_structured_docs.py 87,
tests/test_doc_tables.py 109, tests/test_hooks_api.py 387, tests/test_hooks.py 204,
tests/test_langchain.py 207. Full ci.yml suite list, 62 suites, 5 failures -- the same five
that fail identically on unmodified 9d95567 in this environment. ruff, compileall,
git diff --check and zensical build --strict --clean all clean.
@NandhaKishorM
NandhaKishorM merged commit d6397e7 into NandhaKishorM:main Sep 29, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants