Skip to content

docs(examples): example 30 must call score confidence 1 - H/log(k), not the entropy - #974

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-example30-score-confidence-formula
Oct 7, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-example30-score-confidence-formula

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

examples/30_custom_schema_design.py teaches the four question types on the page — choice, score, noul — and the section "3. when score beats choice" ends with a definitional claim about score's confidence field:

print("   on a score is normalised entropy, so a wide-but-ordered distribution looks")
print("   unconfident even when the expected level is informative.")

That is the inverse of what the repo does everywhere else:

  • laya/common.py::confidence_from_probs returns 1 - H(p) / log(k) and its docstring says "Normalized Shannon entropy confidence: 1 - H(p) / log(k)".
  • The field's docstring in laya/agent.py says the same.
  • Example 18's own banner calls it "1 minus normalised entropy".
  • Example 40's Part C prints 1 - H/log(k).
  • README.md describes confidence as 1 - normalized entropy.

The sentence is also self-contradictory in place: if confidence were the entropy, a wide-but-ordered distribution would read as high confidence — but the very next clause says "looks unconfident". The conclusion the paragraph reaches is the correct one; the definition it uses to get there is wrong.

The page now:

  • Rewrites the sentence to 1 - H/log(k), one MINUS the normalised entropy — the correct definition, phrased in the same voice.
  • Adds a pure top-level helper entropy_confidence(probs) and, in place, recomputes the field from the reported probabilities for each of the two records so the reader sees the claim cross-checked. On the English checkpoint:
   crash  k=4  entropy confidence: reported 0.2314, recomputed 0.2313 (0.000080 apart)
   UI     k=4  entropy confidence: reported 0.2019, recomputed 0.2018 (0.000105 apart)
  • Notes that the residual between the two is the published-vs-internal rounding (probabilities come back at 4 dp while the agent computed the field from unrounded logits), and that each value is well below the corresponding answer_confidence — 0.464 / 0.481 respectively — because answer_confidence is max(p) on the same distribution while confidence is the wider, more conservative entropy scale.

Why

Example 30 is the page that tells a caller what the four question types are and what each answer field means. It's the intro to schema design. When its definition of confidence inverts from the code's own, every downstream reader learns the wrong quantity — the very confusion the field vocabulary at the top of test_confidence.py ("confidence carries two definitions, and only one of them is the calibrated quantity") exists to prevent.

The recomputation closes the loop the way examples 28 (this repo, already merged) does for its preset-page conclusions: instead of asserting a numeric property in prose and leaving the reader to trust it, the page derives the number from what it prints. That's what makes the claim checkable in place rather than a docstring.

How it was verified

# CI path (script mode; the file has a module-level sys.exit like every other example gate).
python tests/test_confidence.py
# → 138 passed, 0 failed. Baseline on upstream/main a4a8921 in the same venv: 134 passed, 0 failed.
#   All 4 new page-30 tests pass. The page-28 section is unchanged.

python examples/30_custom_schema_design.py
# → exit=0 against the real English checkpoint offline; prints the two-row
#   reported / recomputed table quoted above.

# gates from AGENTS.md:
ruff check laya/ examples/30_custom_schema_design.py tests/test_confidence.py \
  --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
# → All checks passed / COMPILE OK.

# Mutation harness (laya-bench/mutate_example30_gate.py) — seven edits to the example,
# each byte-restored, run with PYTHONDONTWRITEBYTECODE=1:
#   M1 prose restored to "is normalised entropy"           → test_page30_drops_the_inverted_definition
#   M2 helper returns H/log(k) instead of 1 - H/log(k)     → test_entropy_confidence_matches_core
#   M3 helper hardcodes k=4 instead of len(probs)          → test_entropy_confidence_matches_core
#   M4 helper renamed to hand_entropy                       → test_page30_helpers_are_live_and_pure
#   M5 helper call replaced by a literal                    → test_page30_helpers_are_live_and_pure
#   M6 print drops the "recomputed" field                   → test_page30_recomputes_in_place
# → 6/6 caught, tree restored: True.

The gate section in tests/test_confidence.py mirrors the merged page-28 idiom: entropy_confidence is AST-extracted from the example and exec'd with math injected, then driven against confidence_from_probs on seven fabricated distributions (including k=1 and k=0 edges). The definitional ban fires on a literal OLD_PAGE_30 block quoting main's wording and not on the fix; the positive rule (that the 1 - H/log(k) definition and the reported-vs-recomputed print are both present) fires on the fix and not on main's block. No weights are loaded.

Relationship to #972 and #973

All three PRs target the "example prose asserts something the code doesn't back" class, but the defects are independent:

The three page-N sections in tests/test_confidence.py are all self-contained: EXAMPLE_30, HELPERS_30, OLD_PAGE_30, _src30, _helpers30, _called30, PROBS_READ, HELPER_CALLED, PRINTS_REPORTED_AND_RECOMPUTED, INVERTED_SCORE_DEF, SCORE_DEFINES_1_MINUS_H, and the four test_page30_* functions all carry the 30 suffix. The only textual overlap between the three PRs is the same anchor — the page-N block goes after the page-28 runner and before the final print("\n%d passed, %d failed"). Whichever merges last needs a trivial rebase; the merge is a straight append in a different region, not a rewrite.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (a4a8921 release: 0.3.28)
  • Tests pass locally (script mode: 138 passed, 0 failed)
  • Docs or examples updated when the public API changed (no public API change; example prose and a gate)
  • ruff check --select=E9,F63,F7,F82,F401,F811 --line-length=120 clean
  • python -m compileall -q laya/ tests/ clean
  • No secrets in the diff (grep for api key / secret / password / token / bearer / private key / BEGIN OPENSSH / AKIA / ghp_ — one false positive on a tokens = ... variable name in the example, unrelated to this change)

Main's page asserts, in prose and with nothing to check it against, that
"`confidence` on a score is normalised entropy". That is the inverse of
`laya.common.confidence_from_probs`, examples 18/28/40's "1 minus normalised
entropy", and the field's docstring. The sentence was self-contradictory in
place: if confidence were the entropy itself, a wide distribution would read
as HIGH confidence, but the very next clause said "looks unconfident".

Rewrite the sentence to `1 - H/log(k), one MINUS the normalised entropy` and
make the claim checkable in place: `entropy_confidence()` recomputes the field
from the reported probabilities and both numbers are printed side by side for
each record. On the English checkpoint: crash k=4 reported 0.2314 recomputed
0.2313 (8.0e-5 apart), UI k=4 reported 0.2019 recomputed 0.2018 (1.05e-4
apart) -- the residual is the published-vs-internal rounding, and each value
is well below the matching `answer_confidence` (max(p) = 0.464 / 0.481).
The page-30 section mirrors the page-28 idiom: no weights are loaded, the
`entropy_confidence` helper is extracted from the example's own AST and exec'd
against `confidence_from_probs` on seven fabricated distributions (including
k=1 and k=0 edges). A ban regex fires on main's "`confidence` on a score is
normalised entropy" wording and not on the fix, and the positive rule (that
the corrected `1 - H/log(k)` definition and the reported/recomputed print are
both present) fires on the fix and not on main. Suite: 138 passed, 0 failed
(baseline 134).

Mutation harness `laya-bench/mutate_example30_gate.py`: six mutants -- prose
reverted to main, helper returns H/log(k), k hardcoded to 4, helper renamed,
helper call replaced with a literal, print drops "recomputed" -- 6/6 caught,
tree restored.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.3.29.

examples/30 presented the entropy itself where the reported confidence is 1 - H/log(k), so the page's number moved in the opposite direction from the field it claimed to explain. On a page whose purpose is explaining score confidence, an inverted formula is worse than no formula.

I checked the underlying contract independently on real weights rather than take the batch on trust:

triage  type=choice confidence=0.9286  1-H/log(k)=0.9288
sev     type=score  confidence=0.7651  1-H/log(k)=0.7652
urgent  type=noul   confidence=0.7823  1-H/log(k)=n/a

So 1 - H/log(k) is right for choice and score to four decimals, and noul has no option vector at all. Your correction matches the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants