Repository navigation
docs: add the Questions and answers guide (#390) - #418
Merged
Merged
Conversation
This was referenced Sep 24, 2026
Owner
|
Thank you, the confidence section is the clearest explanation of the two fields we have, and the render_options examples check out exactly. Please change three things: describe |
Implements guide 2 from [NandhaKishorM#390](NandhaKishorM#390), which lists "Questions and answers: `choice`, `score` and `noul`, reading probabilities and confidence (including the calibrated confidence from NandhaKishorM#126), confidence gating, and the presets" as one of the topics still to write. Nobody had claimed it — @emiliano-go said in that thread he was not working on any of the six guides. Every printed value in the page was produced by running the code on laya 0.3.20 against the English checkpoint, so the answer shapes are transcripts rather than reconstructions. ## What the page covers - **The three types**, with a full answer object for each, and why a `noul` question is not a two-option `choice`. - **Reading confidence**, which is the part that is easiest to get wrong. - **`score` is an expected value, not the most likely level.** - **Presets**: `triage_questions`, `guard_questions`, `moderation_questions`. - **Reading the options back** with `render_options`. - **Limits**, linking the two issues that measure them. ## The part worth reviewing closely An answer carries **two** confidence fields, and they differ: | field | what it is | gating on it? | |---|---|---| | `answer_confidence` | `max(p)` — the probability of the answer reported | **yes, after fitting** | | `confidence` | `1 - H(p)/log k` — concentration of the whole distribution | no | `answer_confidence` is confirmed to be exactly `max(p)` for every type, and `confidence` to be `confidence_from_probs(p, k)` for `choice` and `score`. **For `noul` they are equal by construction** — over two options `max(p, 1-p)` is `max(p)` — so a `noul` caller cannot tell from the value which of the two they have been reading. This is deliberate: `laya/agent.py` says so in as many words, and reports both rather than changing one underneath existing callers. ## Three corrections from the review **1. `answer_confidence` is not "calibrated: yes".** The table said so, and the sentence under it stated the calibration property unconditionally. It is the quantity temperature scaling *fits*, and the property — of the answers returned at confidence *c*, about *c* of them are right — holds only after temperatures are fitted and validated on held-out data for that checkpoint at that option count. The shipped checkpoints are over-confident and how far depends on the option count, so an untuned threshold can select below the model's own accuracy ([NandhaKishorM#394](NandhaKishorM#394)). The row now reads "yes, after fitting" and the paragraph says what has to happen first. **2. The `THRESHOLD` snippet carries that note**, and points at the fitting loop in the fine-tuning notebook — `BENCHMARKS.md` has the before and after ECE but not the loop itself. **3. `agent._to_internal` is no longer presented as the way to do it.** It is internal and may change, so the page shows the short, stable mapping inline instead: ```python def as_internal(q): crit = q.get("criteria") if q["type"] == "choice" and isinstance(crit, list): crit = {c: None for c in crit} return {"t": q["type"], "ins": q["instructions"], "crit": crit} ``` Checked against `Agent._to_internal` on five question shapes (a `choice` given as a list, as a dict, a `score`, a bare `noul`, and a `noul` with criteria): **5 of 5 produce the same internal question and the same `render_options` output.** The wording on `answer_confidence` is kept in step with the reference PR, which fixes the same over-claim in the function's own docstring. Deliberate choices tested: - **Both new guides are separate PRs.** NandhaKishorM#390 says "each would be a separate PR", and routing and questions are independent pages. - **No nav edit.** `docs/.nav.yml` says an unlisted page appears under Guides, so neither page touches navigation and neither can conflict with another PR over it. - **The limits section links the issues rather than restating them**, so the page does not freeze numbers that will move when those are fixed. - **No link out of the docs tree.** A first version of the routing page linked `../BENCHMARKS.md`, which fails `zensical build --strict` because the file is outside `docs/`; this page cites it as a GitHub URL. ``` $ zensical build --strict --clean exit 0, no griffe lines $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` Known limitation: the calibration section points at `BENCHMARKS.md` and the notebook rather than reproducing the fitting loop, because the accepted figures there are per option count and restating them would create a second copy to keep in sync. The other four guides NandhaKishorM#390 lists are not covered here.
PerryLink
force-pushed
the
docs-answers-guide
branch
from
September 25, 2026 15:47
2e0807b to
2b1f6c4
Compare
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 27, 2026
Implements guide 5 from [#390](#390), which asks for "a general walkthrough of the notebook, fitting calibration temperatures, and pushing a checkpoint to the Hub, alongside the existing browser-agent example". Nobody has claimed it in the thread: @emiliano-go is not working on any of the six, and @Bruce-Yii took the command line + MCP guide (with #415 for staged rollout). Every number on the page is from committed material -- the README's 0.36/0.35/0.766 and 4-5 h runtime, the notebook's own cells (slice sizes, hyper-parameters, the 4-6 minute demo run, the fixed comparison rows) -- or a public issue number. Nothing is estimated, and no training was run to write it. ## What the page covers - **The notebook end to end**, as a table: what each step does, the Kaggle settings, and where the outputs land. - **The training recipe**: gold distributions as targets, the GRPO-style policy gradient with proper-scoring rewards, the soft cross-entropy term, and the knob table for a 16 GB card. - **Calibration is part of the run**, the page's centre: the slice is held out *before* the shard (fitting on trained items returns a degenerate scale), the fit is per question type and clamped, and the inherited `temperature_by_options` is removed in the same write because it would take precedence at inference. `tests/test_calibration_persistence.py` is the regression. - **Evaluating**: the test-split pass, the metric list, the fixed reference rows, and the two habits -- hold out your own slices, report calibration next to accuracy. - **Pushing to the Hub**, and **what to watch** (targets as the ceiling, the small calibration slice, primitive limits, config persistence). Deliberate choices tested: - **One nav edit.** The Fine-tuning section exists and already holds the browser-agent page, so `finetune.md` is listed beside it rather than left to the Guides glob -- #390's rule is that an unlisted page shows up under Guides and can be "placed in another section when needed". - **No link out of the docs tree, and none into pages still in review.** `BENCHMARKS.md` is plain text for the same reason as the Routing guide (zensical `--strict` resolves relative links), and links to the Questions and answers guide (#418, in review) failed both gates -- zensical aborted, and `test_packaging.py` named the missing target -- so the primitives' limits are linked as issues (#394, #377) instead. - **It expands the README's Fine-Tuning section rather than copying it**, per #390 ("the README summaries kept in place"): the walkthrough, the calibration order and the publish cell are notebook-level detail the README does not carry. ``` $ zensical build --strict --clean exit 0, no griffe lines $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: this is the fine-tuning walkthrough only. The other guides #390 lists -- HTTP API (which should wait for #304/#281 to settle), command line and MCP (@Bruce-Yii; #415 landed for staged rollout), benchmarks -- are not covered here.
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 29, 2026
`laya.answer_confidence` is in `laya.__all__`, has a 615-character docstring, and appears in **no page in `docs/`** — while the function it warns you not to confuse it with, `laya.common.confidence_from_probs`, is rendered directly above where it belongs. The reference is generated from the docstrings, so a public name with no directive is simply invisible, and this was the only one in that state: ``` $ python scratch/check_ref_gaps.py === public names with NO directive anywhere === BaseHook -> laya.hooks.BaseHook DOCSTRING 253 chars DecisionResult -> laya.structured.DecisionResult Hook -> laya.hooks.Hook PredictContext -> laya.hooks.PredictContext PredictHook -> typing.Callable answer_confidence -> laya.common.answer_confidence DOCSTRING 615 chars decide -> laya.structured.decide ``` The other six are already covered by hand-written pages and are correctly absent from the generated index — `PredictContext`, `Hook`, `BaseHook` and `PredictHook` have [`hooks/api.md`](../hooks/api.md), which `reference/index.md` links to, and `decide` and `DecisionResult` have [`structured.md`](../structured.md) and are not public names a reader looks up in isolation. `answer_confidence` had neither. It is worth rendering rather than leaving to the reader, because it is the field a caller is supposed to gate on. Its docstring is the only place in the codebase that states the distinction: > This is the quantity temperature scaling fits, and the quantity every calibration figure in this > repository is computed on -- both benchmark harnesses take `conf = max(probs)` before calling > `ece_score`. It is therefore the one confidence with the property the README's gating section > relies on: of the answers returned at confidence c, about c of them are right. > > `confidence_from_probs` below reports a different quantity on a different scale and carries no > such guarantee, so the two must not be compared against the same threshold. That is exactly the material a reader needs when they find `confidence` and `answer_confidence` side by side in one answer object and have to pick, and until now it was reachable only by reading the source. Deliberate choices tested: - **One directive, in the section it belongs to.** It goes with `confidence_from_probs` under "Calibration and training" rather than in a new section, because the docstring's whole point is the contrast between the two, and a reader arriving at either should see the other. - **No prose added to the page.** The docstring already says it better than a summary would, and the page is generated — an added paragraph would be a second copy to keep in sync. - **The other six names are deliberately not added.** Four would duplicate `hooks/api.md`, which is hand-written precisely because hooks need more than a signature dump; two are covered by `structured.md`. Adding them would move documentation away from where it is maintained, which is the failure mode `reference/index.md` already calls out. ``` $ python -m zensical build --strict --clean # exit 0, no griffe lines answer_confidence rendered: True confidence_from_probs: True $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed ``` Known limitation: this is the generated-reference gap only. It does not add `answer_confidence` to the README's gating section, which is [#378](#378) and the [Questions and answers guide](#418); if those land first, this remains the API-level entry they both point at.
Owner
|
Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs: add the Questions and answers guide
Implements guide 2 from #390, which lists
"Questions and answers:
choice,scoreandnoul, reading probabilities and confidence(including the calibrated confidence from #126), confidence gating, and the presets" as one of
the topics still to write. Nobody had claimed it — @emiliano-go said in that thread he was not
working on any of the six guides.
Every printed value in the page was produced by running the code on laya 0.3.20 against the
English checkpoint, so the answer shapes are transcripts rather than reconstructions.
What the page covers
noulquestion is not atwo-option
choice.scoreis an expected value, not the most likely level.triage_questions,guard_questions,moderation_questions.render_options, including a caveat about its argument shape.The part worth reviewing closely
An answer carries two confidence fields, and nothing in the README says they differ:
answer_confidencemax(p)— probability of the answer reportedconfidence1 - H(p)/log k— concentration of the whole distributionMeasured on one state, all three question types:
answer_confidenceis confirmed to be exactlymax(p)for every type, andconfidenceto beconfidence_from_probs(p, k)forchoiceandscore. Fornoulthey are equal byconstruction — over two options
max(p, 1-p)ismax(p)— so anoulcaller cannot tell fromthe value which of the two they have been reading. This is deliberate, not an inconsistency:
laya/agent.py:673-678says so in as many words —So the guide tells a reader to gate on
answer_confidence, and says plainly that the two are noton the same scale and must not be compared against one number.
scoreis an expected value, and the page makes that concrete. In the same run,severityreportsscore2.9046 while the most likely level isserious(index 3) at 0.7881.Rounding
scoreand calling it the label gives the right answer here by luck and will not ingeneral;
legendis documented as the way to map an index back to a label. This is the kind ofthing a reference page states and the docstring does not.
Corrections made while writing it
render_optionstakes the internal shape. I first wrote the example with the public{"type": ..., "criteria": ...}form. That raisesKeyError: 't'— the function readsq["t"]andq["crit"]. The page now shows the short-key form it actually accepts and givesrender_options(agent._to_internal(question))for rendering a question as written.noulexample is from a real call, not a plausible-looking number, which is how theconfidence == answer_confidencecoincidence fornoulsurfaced at all.Deliberate choices tested:
docs/.nav.ymlsays an unlisted page appears under Guides, so neither page touches navigation and neither can conflict with another PR over it.../BENCHMARKS.md, which failszensical build --strictbecause the file is outsidedocs/. This page cites it as a GitHub URL.Known limitation: the calibration section points at
BENCHMARKS.mdand the fine-tuning notebookrather than reproducing the fitting loop, because the accepted figures there are per option count
and restating them would create a second copy to keep in sync. The other four guides #390 lists
are not covered here.