Repository navigation
docs(examples): example 18 must not call the shipped answer_confidence calibrated - #973
Merged
NandhaKishorM merged 2 commits intoOct 7, 2026
Conversation
…e calibrated The module docstring said the page "branches on Laya's calibrated confidence" and the banner called `answer_confidence` "the calibrated probability Laya puts on the answer it reports". `answer_confidence` is max(p), the field temperature scaling fits and the abstention gate reads, but the README's own Calibration section (line 1586) says the shipped `laya` checkpoint produces mean ECE 0.466 as shipped and 0.081 only after refitting on held-out data, and warns that "Both checkpoints are over-confident as shipped". The loader emits on this very checkpoint: RuntimeWarning: ... choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated. The page now reads back, per question, the temperature this checkpoint actually applies -- replaying `Agent.predict_batch`'s own `temperature_by_options.get(temp_bucket(...), temperature[QTYPES[...]])` lookup on the loaded agent rather than restating a table -- and states the calibration claim as the README's condition. Measured on the shipped English checkpoint against the two questions the example asks: department choice 4 choice:3-5 1.7602 bucket map refund_requested noul 2 noul:2 1.9834 bucket map Both temperatures are meaningfully above 1.0, so unlike a bucket the shipped checkpoint leaves at 1.0 the raw softmax is scaled. But scale alone is not a calibration measurement: the 0.85 threshold below the banner is our policy on the shipped distribution, not a cutoff a held-out fit earned.
Mirror the page-28 idiom in a self-contained page-18 section that pulls
`option_count` and `scale_for` from the example's AST and execs them against a
fabricated agent, injecting `temp_bucket`, `QTYPES` and `math` from
`laya.common` so no weights load:
- `test_page18_helpers_are_live_and_defer_to_core` asserts both helpers are
called at module level, that their only free non-builtin names are the
injected trio, that neither prints, and that `scale_for` carries no
hardcoded `"choice:3-5"` / `"noul:2"` string constant.
- `test_page18_questions_reach_core_bucket_names` literal_evals the page's
own QUESTIONS dict, replays it through both `temp_bucket` and `scale_for`
on a fabricated agent, and pins the shapes to
`department: choice k=4 -> choice:3-5 @ 1.7601518630981445` and
`refund_requested: noul k=2 -> noul:2 @ 1.983399510383606` -- the exact
values this checkpoint publishes.
- `test_page18_drops_the_unconditional_calibration_claim` is witnessed
against an OLD_PAGE_18 literal of main's wording: the ban fires there, not
here; the "over-confident" condition and the `temp_bucket(` lookup are
required here and absent there. It also re-quotes the README's own line
("Both checkpoints are over-confident as shipped" and the "0.466 -> 0.081"
ECE pair) so the page cannot misstate the calibration claim it points to.
Mutation harness `laya-bench/mutate_example18_gate.py`: 6/6 caught, tree
restored True. Suite: `python tests/test_confidence.py` -> 137 passed, 0
failed (up from 134 on main).
This was referenced Oct 6, 2026
Owner
|
Merged in 0.3.29.
Your framing of the distinction is the part I want to quote back: calibrated is a property a fit earns, not a property of the field. That is the sentence the whole batch turns on. Reading back the temperature actually applied to each of the page's two questions, rather than deleting the word, keeps the page useful. |
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/18_confidence_gating.pyis the intro-level gating pattern — the "production pattern from the README" the README's Automated Confidence Gating section points at. Its module docstring says the page "branches on Laya's calibrated confidence" and the banner callsanswer_confidence"the calibrated probability Laya puts on the answer it reports". Both are claims the code doesn't back on this checkpoint.The page now:
layais 0.466 as shipped, 0.081 after refitting one temperature per (question type, option-count) bucket on held-out data, and "Both checkpoints are over-confident as shipped" (README line 1586).agent.temperature_by_optionsviatemp_bucket(QTYPES[...], k)— the same lookupAgent.predict_batchperforms — rather than restating a table.Why
answer_confidenceismax(p).laya/common.py's own docstring says so, andlaya/confidence.py:17-22says the calibration reading "holds only after temperatures have been fitted and validated". The 0.85 threshold this example applies is a policy on the shipped distribution, not a cutoff a held-out measurement earned on this checkpoint. The loader on this exact checkpoint emits:The interesting nuance is that example 18's two buckets are both meaningfully scaled:
choice:3-5 = 1.7601518630981445andnoul:2 = 1.983399510383606. Unlike example 40, where the 6-option intent question hits a bucket the checkpoint ships at1.0000158548355103(i.e. the raw softmax), the answers this page gates on have been through a fitted scale. That is what makes the fix here different from a mechanical copy of #972 — the shipped scale being above 1.0 is necessary for a calibrated probability but not sufficient: the README's own Calibration table says the shipped state produces mean ECE 0.466 onlaya, and that refitting drops it to 0.081. So a threshold at 0.85 on a scaled-but-not-refit distribution is still a policy, not a measurement.Example 41 builds directly on the auto/review/escalate pattern in example 40, and this page is the same production pattern in the README. Teaching it with an unqualified "calibrated" is what downstream examples and every caller following the tutorial inherits.
How it was verified
Relationship to #972
Both PRs target the same defect class — an example that calls
answer_confidence"the calibrated probability" — but each fix is different because the checkpoints' temperatures under each page's questions differ. #972 (example 40) covers a case where the temperature is 1.0 (unscaled); this PR covers a case where the temperature is above 1.0 but the checkpoint's shipped ECE is still 0.466. #972 introduces a page-40 section intests/test_confidence.pythat adds_FakeAgent,_temp_bucket,_QTYPES40,_free_names, and the final print-block. This PR's page-18 section is deliberately self-contained:_FakeAgent18,_temp_bucket18,_QTYPES18,_helpers18andEXAMPLE_18all carry their own names, so the only textual overlap is the anchor "before the print block at end of file". Whichever merges second needs a trivial rebase — the merge is a straight append in a different region, not a rewrite.Checklist
main(a4a8921 release: 0.3.28)