Skip to content

docs: add the Questions and answers guide (#390) - #418

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:docs-answers-guide
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:docs-answers-guide

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

docs: add the Questions and answers guide

Implements guide 2 from #390, which lists
"Questions and answers: choice, score and noul, reading probabilities and confidence
(including the calibrated confidence from #126), confidence gating, and the presets" as one of
the topics still to write. Nobody had claimed it — @emiliano-go said in that thread he was not
working on any of the six guides.

Every printed value in the page was produced by running the code on laya 0.3.20 against the
English checkpoint, so the answer shapes are transcripts rather than reconstructions.

What the page covers

  • The three types, with a full answer object for each, and why a noul question is not a
    two-option choice.
  • Reading confidence, which is the part that is easiest to get wrong.
  • score is an expected value, not the most likely level.
  • Presets: triage_questions, guard_questions, moderation_questions.
  • Reading the options back with render_options, including a caveat about its argument shape.
  • Limits, linking the two issues that measure them.

The part worth reviewing closely

An answer carries two confidence fields, and nothing in the README says they differ:

field what it is calibrated?
answer_confidence max(p) — probability of the answer reported yes
confidence 1 - H(p)/log k — concentration of the whole distribution no

Measured on one state, all three question types:

dept      confidence 0.9339   answer_confidence 0.9881   (choice, k=3)
urgent    confidence 0.8727   answer_confidence 0.8727   (noul,   k=2)
severity  confidence 0.5187   answer_confidence 0.7881   (score,  k=5)

answer_confidence is confirmed to be exactly max(p) for every type, and confidence to be
confidence_from_probs(p, k) for choice and score. For noul they are equal by
construction
— over two options max(p, 1-p) is max(p) — so a noul caller cannot tell from
the value which of the two they have been reading. This is deliberate, not an inconsistency:
laya/agent.py:673-678 says so in as many words —

confidence means one thing for noul (max(p)) and another for choice and score
(normalized entropy), and only the first is the quantity temperature scaling fits and ECE
measures. Rather than change one underneath existing callers, report both: answer_confidence
is the calibrated one, on every question type, so a caller can gate across types on a single
number.

So the guide tells a reader to gate on answer_confidence, and says plainly that the two are not
on the same scale and must not be compared against one number.

score is an expected value, and the page makes that concrete. In the same run,
severity reports score 2.9046 while the most likely level is serious (index 3) at 0.7881.
Rounding score and calling it the label gives the right answer here by luck and will not in
general; legend is documented as the way to map an index back to a label. This is the kind of
thing a reference page states and the docstring does not.

Corrections made while writing it

  • render_options takes the internal shape. I first wrote the example with the public
    {"type": ..., "criteria": ...} form. That raises KeyError: 't' — the function reads
    q["t"] and q["crit"]. The page now shows the short-key form it actually accepts and gives
    render_options(agent._to_internal(question)) for rendering a question as written.
  • The noul example is from a real call, not a plausible-looking number, which is how the
    confidence == answer_confidence coincidence for noul surfaced at all.

Deliberate choices tested:

  • Both new guides are separate PRs. Docs site: structure and the guides still to write #390 says "each would be a separate PR", and routing and questions are independent pages.
  • No nav edit. docs/.nav.yml says an unlisted page appears under Guides, so neither page touches navigation and neither can conflict with another PR over it.
  • The limits section links the issues rather than restating them. #394 measures the confidence overlap at 20 options and #377 the negation failure, so the page points at them instead of freezing numbers that will move when they are fixed.
  • No external link to BENCHMARKS.md. The first version of the routing page linked ../BENCHMARKS.md, which fails zensical build --strict because the file is outside docs/. This page cites it as a GitHub URL.
$ python -m zensical build --strict --clean      # exit 0, no griffe lines
$ python tests/test_packaging.py                 all packaging tests passed
$ python tests/test_doc_tables.py                54 passed, 0 failed

Known limitation: the calibration section points at BENCHMARKS.md and the fine-tuning notebook
rather than reproducing the fitting loop, because the accepted figures there are per option count
and restating them would create a second copy to keep in sync. The other four guides #390 lists
are not covered here.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, the confidence section is the clearest explanation of the two fields we have, and the render_options examples check out exactly. Please change three things: describe answer_confidence as the quantity temperature scaling fits, calibrated only after you fit and validate temperatures on your own held-out data (and add that note to the THRESHOLD snippet), rather than 'calibrated: yes'. Point to the fine-tuning notebook for the fitting loop, since BENCHMARKS.md only has the before and after ECE. And either mark agent._to_internal as an internal helper that may change, or show the short-key conversion instead. It would be great to keep the wording in step with #419.

Implements guide 2 from [NandhaKishorM#390](NandhaKishorM#390), which lists
"Questions and answers: `choice`, `score` and `noul`, reading probabilities and confidence
(including the calibrated confidence from NandhaKishorM#126), confidence gating, and the presets" as one of
the topics still to write. Nobody had claimed it — @emiliano-go said in that thread he was not
working on any of the six guides.

Every printed value in the page was produced by running the code on laya 0.3.20 against the
English checkpoint, so the answer shapes are transcripts rather than reconstructions.

## What the page covers

- **The three types**, with a full answer object for each, and why a `noul` question is not a two-option `choice`.
- **Reading confidence**, which is the part that is easiest to get wrong.
- **`score` is an expected value, not the most likely level.**
- **Presets**: `triage_questions`, `guard_questions`, `moderation_questions`.
- **Reading the options back** with `render_options`.
- **Limits**, linking the two issues that measure them.

## The part worth reviewing closely

An answer carries **two** confidence fields, and they differ:

| field | what it is | gating on it? |
|---|---|---|
| `answer_confidence` | `max(p)` — the probability of the answer reported | **yes, after fitting** |
| `confidence` | `1 - H(p)/log k` — concentration of the whole distribution | no |

`answer_confidence` is confirmed to be exactly `max(p)` for every type, and `confidence` to be
`confidence_from_probs(p, k)` for `choice` and `score`. **For `noul` they are equal by
construction** — over two options `max(p, 1-p)` is `max(p)` — so a `noul` caller cannot tell from
the value which of the two they have been reading. This is deliberate: `laya/agent.py` says so in
as many words, and reports both rather than changing one underneath existing callers.

## Three corrections from the review

**1. `answer_confidence` is not "calibrated: yes".** The table said so, and the sentence under it
stated the calibration property unconditionally. It is the quantity temperature scaling *fits*,
and the property — of the answers returned at confidence *c*, about *c* of them are right — holds
only after temperatures are fitted and validated on held-out data for that checkpoint at that
option count. The shipped checkpoints are over-confident and how far depends on the option count,
so an untuned threshold can select below the model's own accuracy
([NandhaKishorM#394](NandhaKishorM#394)). The row now reads "yes, after fitting"
and the paragraph says what has to happen first.

**2. The `THRESHOLD` snippet carries that note**, and points at the fitting loop in the
fine-tuning notebook — `BENCHMARKS.md` has the before and after ECE but not the loop itself.

**3. `agent._to_internal` is no longer presented as the way to do it.** It is internal and may
change, so the page shows the short, stable mapping inline instead:

```python
def as_internal(q):
    crit = q.get("criteria")
    if q["type"] == "choice" and isinstance(crit, list):
        crit = {c: None for c in crit}
    return {"t": q["type"], "ins": q["instructions"], "crit": crit}
```

Checked against `Agent._to_internal` on five question shapes (a `choice` given as a list, as a
dict, a `score`, a bare `noul`, and a `noul` with criteria): **5 of 5 produce the same internal
question and the same `render_options` output.**

The wording on `answer_confidence` is kept in step with the reference PR, which fixes the same
over-claim in the function's own docstring.

Deliberate choices tested:

- **Both new guides are separate PRs.** NandhaKishorM#390 says "each would be a separate PR", and routing and questions are independent pages.
- **No nav edit.** `docs/.nav.yml` says an unlisted page appears under Guides, so neither page touches navigation and neither can conflict with another PR over it.
- **The limits section links the issues rather than restating them**, so the page does not freeze numbers that will move when those are fixed.
- **No link out of the docs tree.** A first version of the routing page linked `../BENCHMARKS.md`, which fails `zensical build --strict` because the file is outside `docs/`; this page cites it as a GitHub URL.

```
$ zensical build --strict --clean      exit 0, no griffe lines
$ python tests/test_packaging.py       all packaging tests passed
$ python tests/test_doc_tables.py      54 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

Known limitation: the calibration section points at `BENCHMARKS.md` and the notebook rather than
reproducing the fitting loop, because the accepted figures there are per option count and
restating them would create a second copy to keep in sync. The other four guides NandhaKishorM#390 lists are
not covered here.
NandhaKishorM pushed a commit that referenced this pull request Sep 27, 2026
Implements guide 5 from [#390](#390), which asks
for "a general walkthrough of the notebook, fitting calibration temperatures, and pushing a
checkpoint to the Hub, alongside the existing browser-agent example". Nobody has claimed it in
the thread: @emiliano-go is not working on any of the six, and @Bruce-Yii took the command
line + MCP guide (with #415 for staged rollout).

Every number on the page is from committed material -- the README's 0.36/0.35/0.766 and 4-5 h
runtime, the notebook's own cells (slice sizes, hyper-parameters, the 4-6 minute demo run, the
fixed comparison rows) -- or a public issue number. Nothing is estimated, and no training was
run to write it.

## What the page covers

- **The notebook end to end**, as a table: what each step does, the Kaggle settings, and where
  the outputs land.
- **The training recipe**: gold distributions as targets, the GRPO-style policy gradient with
  proper-scoring rewards, the soft cross-entropy term, and the knob table for a 16 GB card.
- **Calibration is part of the run**, the page's centre: the slice is held out *before* the
  shard (fitting on trained items returns a degenerate scale), the fit is per question type and
  clamped, and the inherited `temperature_by_options` is removed in the same write because it
  would take precedence at inference. `tests/test_calibration_persistence.py` is the regression.
- **Evaluating**: the test-split pass, the metric list, the fixed reference rows, and the two
  habits -- hold out your own slices, report calibration next to accuracy.
- **Pushing to the Hub**, and **what to watch** (targets as the ceiling, the small calibration
  slice, primitive limits, config persistence).

Deliberate choices tested:

- **One nav edit.** The Fine-tuning section exists and already holds the browser-agent page, so
  `finetune.md` is listed beside it rather than left to the Guides glob -- #390's rule is that
  an unlisted page shows up under Guides and can be "placed in another section when needed".
- **No link out of the docs tree, and none into pages still in review.** `BENCHMARKS.md` is
  plain text for the same reason as the Routing guide (zensical `--strict` resolves relative
  links), and links to the Questions and answers guide (#418, in review) failed both gates --
  zensical aborted, and `test_packaging.py` named the missing target -- so the primitives'
  limits are linked as issues (#394, #377) instead.
- **It expands the README's Fine-Tuning section rather than copying it**, per #390 ("the
  README summaries kept in place"): the walkthrough, the calibration order and the publish
  cell are notebook-level detail the README does not carry.

```
$ zensical build --strict --clean        exit 0, no griffe lines
$ python tests/test_packaging.py         all packaging tests passed
$ python tests/test_doc_tables.py        54 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: this is the fine-tuning walkthrough only. The other guides #390 lists --
HTTP API (which should wait for #304/#281 to settle), command line and MCP (@Bruce-Yii; #415
landed for staged rollout), benchmarks -- are not covered here.
@NandhaKishorM
NandhaKishorM merged commit 2cacc00 into NandhaKishorM:main Sep 29, 2026
22 checks passed
NandhaKishorM pushed a commit that referenced this pull request Sep 29, 2026
`laya.answer_confidence` is in `laya.__all__`, has a 615-character docstring, and appears in
**no page in `docs/`** — while the function it warns you not to confuse it with,
`laya.common.confidence_from_probs`, is rendered directly above where it belongs. The reference
is generated from the docstrings, so a public name with no directive is simply invisible, and
this was the only one in that state:

```
$ python scratch/check_ref_gaps.py
=== public names with NO directive anywhere ===
  BaseHook               -> laya.hooks.BaseHook         DOCSTRING 253 chars
  DecisionResult         -> laya.structured.DecisionResult
  Hook                   -> laya.hooks.Hook
  PredictContext         -> laya.hooks.PredictContext
  PredictHook            -> typing.Callable
  answer_confidence      -> laya.common.answer_confidence   DOCSTRING 615 chars
  decide                 -> laya.structured.decide
```

The other six are already covered by hand-written pages and are correctly absent from the
generated index — `PredictContext`, `Hook`, `BaseHook` and `PredictHook` have
[`hooks/api.md`](../hooks/api.md), which `reference/index.md` links to, and `decide` and
`DecisionResult` have [`structured.md`](../structured.md) and are not public names a reader
looks up in isolation. `answer_confidence` had neither.

It is worth rendering rather than leaving to the reader, because it is the field a caller is
supposed to gate on. Its docstring is the only place in the codebase that states the distinction:

> This is the quantity temperature scaling fits, and the quantity every calibration figure in this
> repository is computed on -- both benchmark harnesses take `conf = max(probs)` before calling
> `ece_score`. It is therefore the one confidence with the property the README's gating section
> relies on: of the answers returned at confidence c, about c of them are right.
>
> `confidence_from_probs` below reports a different quantity on a different scale and carries no
> such guarantee, so the two must not be compared against the same threshold.

That is exactly the material a reader needs when they find `confidence` and `answer_confidence`
side by side in one answer object and have to pick, and until now it was reachable only by
reading the source.

Deliberate choices tested:

- **One directive, in the section it belongs to.** It goes with `confidence_from_probs` under "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other.
- **No prose added to the page.** The docstring already says it better than a summary would, and the page is generated — an added paragraph would be a second copy to keep in sync.
- **The other six names are deliberately not added.** Four would duplicate `hooks/api.md`, which is hand-written precisely because hooks need more than a signature dump; two are covered by `structured.md`. Adding them would move documentation away from where it is maintained, which is the failure mode `reference/index.md` already calls out.

```
$ python -m zensical build --strict --clean      # exit 0, no griffe lines
  answer_confidence rendered:  True
  confidence_from_probs:       True
$ python tests/test_packaging.py                 all packaging tests passed
$ python tests/test_doc_tables.py                54 passed, 0 failed
```

Known limitation: this is the generated-reference gap only. It does not add `answer_confidence`
to the README's gating section, which is [#378](#378)
and the [Questions and answers guide](#418); if those
land first, this remains the API-level entry they both point at.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants