Skip to content

docs(examples): example 18 must not call the shipped answer_confidence calibrated - #973

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-example18-calibrated-claim
Oct 7, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-example18-calibrated-claim

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/18_confidence_gating.py is the intro-level gating pattern — the "production pattern from the README" the README's Automated Confidence Gating section points at. Its module docstring says the page "branches on Laya's calibrated confidence" and the banner calls answer_confidence "the calibrated probability Laya puts on the answer it reports". Both are claims the code doesn't back on this checkpoint.

The page now:

  • Drops the two unconditional phrasings and states the calibration as conditional.
  • Adds the README's own numbers as the condition: mean ECE on laya is 0.466 as shipped, 0.081 after refitting one temperature per (question type, option-count) bucket on held-out data, and "Both checkpoints are over-confident as shipped" (README line 1586).
  • Prints, per question, the temperature this checkpoint actually applies, read back off agent.temperature_by_options via temp_bucket(QTYPES[...], k) — the same lookup Agent.predict_batch performs — rather than restating a table.
   -- what each question's number was actually scaled by --
   question               type    k   bucket       T applied  source
   department             choice  4   choice:3-5   1.7602     bucket map
   refund_requested       noul    2   noul:2       1.9834     bucket map

Why

answer_confidence is max(p). laya/common.py's own docstring says so, and laya/confidence.py:17-22 says the calibration reading "holds only after temperatures have been fitted and validated". The 0.85 threshold this example applies is a policy on the shipped distribution, not a cutoff a held-out measurement earned on this checkpoint. The loader on this exact checkpoint emits:

RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5];
using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.

The interesting nuance is that example 18's two buckets are both meaningfully scaled: choice:3-5 = 1.7601518630981445 and noul:2 = 1.983399510383606. Unlike example 40, where the 6-option intent question hits a bucket the checkpoint ships at 1.0000158548355103 (i.e. the raw softmax), the answers this page gates on have been through a fitted scale. That is what makes the fix here different from a mechanical copy of #972 — the shipped scale being above 1.0 is necessary for a calibrated probability but not sufficient: the README's own Calibration table says the shipped state produces mean ECE 0.466 on laya, and that refitting drops it to 0.081. So a threshold at 0.85 on a scaled-but-not-refit distribution is still a policy, not a measurement.

Example 41 builds directly on the auto/review/escalate pattern in example 40, and this page is the same production pattern in the README. Teaching it with an unqualified "calibrated" is what downstream examples and every caller following the tutorial inherits.

How it was verified

# CI path (script mode; the file has a module-level sys.exit like every other example gate).
python tests/test_confidence.py
# → 137 passed, 0 failed. Baseline on upstream/main a4a8921 in the same venv: 134 passed, 0 failed.
#   All 3 new page-18 tests pass. The page-28 section (8 helpers + 8 tests) is unchanged.

python examples/18_confidence_gating.py
# → exit=0 against the real English checkpoint offline; prints the temperature table above
#   and the six-email gating decisions (3 automated, 3 escalated at threshold 0.85).

ruff check examples/18_confidence_gating.py tests/test_confidence.py \
    --select=E9,F63,F7,F82,F401,F811 --line-length=120
# → All checks passed!

python -m compileall -q laya/ tests/ examples/
# → exit=0

# Mutation harness at laya-bench/mutate_example18_gate.py — seven edits to the example file,
# each byte-restored, run with PYTHONDONTWRITEBYTECODE=1:
#   M1 option_count returns 2 for choice              → test_page18_questions_reach_core_bucket_names
#   M2 scale_for ignores the bucket map, always 1.0   → test_page18_questions_reach_core_bucket_names
#   M3 scale_for hardcodes "noul:2"                   → test_page18_helpers_are_live_and_defer_to_core
#   M4 docstring says "Laya's calibrated confidence"  → test_page18_drops_the_unconditional_calibration_claim
#   M5 banner says "the calibrated probability"       → test_page18_drops_the_unconditional_calibration_claim
#   M6 banner drops the "over-confident" condition    → test_page18_drops_the_unconditional_calibration_claim
# → 6/6 caught, tree restored: True.

Relationship to #972

Both PRs target the same defect class — an example that calls answer_confidence "the calibrated probability" — but each fix is different because the checkpoints' temperatures under each page's questions differ. #972 (example 40) covers a case where the temperature is 1.0 (unscaled); this PR covers a case where the temperature is above 1.0 but the checkpoint's shipped ECE is still 0.466. #972 introduces a page-40 section in tests/test_confidence.py that adds _FakeAgent, _temp_bucket, _QTYPES40, _free_names, and the final print-block. This PR's page-18 section is deliberately self-contained: _FakeAgent18, _temp_bucket18, _QTYPES18, _helpers18 and EXAMPLE_18 all carry their own names, so the only textual overlap is the anchor "before the print block at end of file". Whichever merges second needs a trivial rebase — the merge is a straight append in a different region, not a rewrite.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (a4a8921 release: 0.3.28)
  • Tests pass locally (script mode: 137 passed, 0 failed)
  • Docs or examples updated when the public API changed (no public API change)

…e calibrated

The module docstring said the page "branches on Laya's calibrated confidence" and
the banner called `answer_confidence` "the calibrated probability Laya puts on
the answer it reports". `answer_confidence` is max(p), the field temperature
scaling fits and the abstention gate reads, but the README's own Calibration
section (line 1586) says the shipped `laya` checkpoint produces mean ECE 0.466
as shipped and 0.081 only after refitting on held-out data, and warns that
"Both checkpoints are over-confident as shipped". The loader emits on this very
checkpoint:

   RuntimeWarning: ... choice:11+=0.10058280825614929 -> 0.5.
   Treat confidence from the affected entries as uncalibrated.

The page now reads back, per question, the temperature this checkpoint actually
applies -- replaying `Agent.predict_batch`'s own
`temperature_by_options.get(temp_bucket(...), temperature[QTYPES[...]])` lookup
on the loaded agent rather than restating a table -- and states the calibration
claim as the README's condition. Measured on the shipped English checkpoint
against the two questions the example asks:

   department             choice  4   choice:3-5   1.7602     bucket map
   refund_requested       noul    2   noul:2       1.9834     bucket map

Both temperatures are meaningfully above 1.0, so unlike a bucket the shipped
checkpoint leaves at 1.0 the raw softmax is scaled. But scale alone is not a
calibration measurement: the 0.85 threshold below the banner is our policy on
the shipped distribution, not a cutoff a held-out fit earned.
Mirror the page-28 idiom in a self-contained page-18 section that pulls
`option_count` and `scale_for` from the example's AST and execs them against a
fabricated agent, injecting `temp_bucket`, `QTYPES` and `math` from
`laya.common` so no weights load:

   - `test_page18_helpers_are_live_and_defer_to_core` asserts both helpers are
     called at module level, that their only free non-builtin names are the
     injected trio, that neither prints, and that `scale_for` carries no
     hardcoded `"choice:3-5"` / `"noul:2"` string constant.
   - `test_page18_questions_reach_core_bucket_names` literal_evals the page's
     own QUESTIONS dict, replays it through both `temp_bucket` and `scale_for`
     on a fabricated agent, and pins the shapes to
     `department: choice k=4 -> choice:3-5 @ 1.7601518630981445` and
     `refund_requested: noul k=2 -> noul:2 @ 1.983399510383606` -- the exact
     values this checkpoint publishes.
   - `test_page18_drops_the_unconditional_calibration_claim` is witnessed
     against an OLD_PAGE_18 literal of main's wording: the ban fires there, not
     here; the "over-confident" condition and the `temp_bucket(` lookup are
     required here and absent there. It also re-quotes the README's own line
     ("Both checkpoints are over-confident as shipped" and the "0.466 -> 0.081"
     ECE pair) so the page cannot misstate the calibration claim it points to.

Mutation harness `laya-bench/mutate_example18_gate.py`: 6/6 caught, tree
restored True. Suite: `python tests/test_confidence.py` -> 137 passed, 0
failed (up from 134 on main).
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.3.29.

examples/18_confidence_gating.py is the intro-level gating page, so it is the one where "the calibrated probability Laya puts on the answer it reports" does the most damage: it is the first thing a reader learns about the field they are about to threshold. The README's own Calibration section says mean ECE on laya is 0.466 as shipped, dropping to 0.081 only after refitting per bucket on held-out data.

Your framing of the distinction is the part I want to quote back: calibrated is a property a fit earns, not a property of the field. That is the sentence the whole batch turns on.

Reading back the temperature actually applied to each of the page's two questions, rather than deleting the word, keeps the page useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants