Repository navigation
docs(examples): example 30 must call score confidence 1 - H/log(k), not the entropy - #974
Merged
NandhaKishorM merged 2 commits intoOct 7, 2026
Conversation
Main's page asserts, in prose and with nothing to check it against, that "`confidence` on a score is normalised entropy". That is the inverse of `laya.common.confidence_from_probs`, examples 18/28/40's "1 minus normalised entropy", and the field's docstring. The sentence was self-contradictory in place: if confidence were the entropy itself, a wide distribution would read as HIGH confidence, but the very next clause said "looks unconfident". Rewrite the sentence to `1 - H/log(k), one MINUS the normalised entropy` and make the claim checkable in place: `entropy_confidence()` recomputes the field from the reported probabilities and both numbers are printed side by side for each record. On the English checkpoint: crash k=4 reported 0.2314 recomputed 0.2313 (8.0e-5 apart), UI k=4 reported 0.2019 recomputed 0.2018 (1.05e-4 apart) -- the residual is the published-vs-internal rounding, and each value is well below the matching `answer_confidence` (max(p) = 0.464 / 0.481).
The page-30 section mirrors the page-28 idiom: no weights are loaded, the `entropy_confidence` helper is extracted from the example's own AST and exec'd against `confidence_from_probs` on seven fabricated distributions (including k=1 and k=0 edges). A ban regex fires on main's "`confidence` on a score is normalised entropy" wording and not on the fix, and the positive rule (that the corrected `1 - H/log(k)` definition and the reported/recomputed print are both present) fires on the fix and not on main. Suite: 138 passed, 0 failed (baseline 134). Mutation harness `laya-bench/mutate_example30_gate.py`: six mutants -- prose reverted to main, helper returns H/log(k), k hardcoded to 4, helper renamed, helper call replaced with a literal, print drops "recomputed" -- 6/6 caught, tree restored.
This was referenced Oct 6, 2026
Owner
|
Merged in 0.3.29.
I checked the underlying contract independently on real weights rather than take the batch on trust: So |
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/30_custom_schema_design.pyteaches the four question types on the page —choice,score,noul— and the section "3. when score beats choice" ends with a definitional claim aboutscore'sconfidencefield:That is the inverse of what the repo does everywhere else:
laya/common.py::confidence_from_probsreturns1 - H(p) / log(k)and its docstring says "Normalized Shannon entropy confidence: 1 - H(p) / log(k)".laya/agent.pysays the same.1 - H/log(k).README.mddescribesconfidenceas1 - normalized entropy.The sentence is also self-contradictory in place: if
confidencewere the entropy, a wide-but-ordered distribution would read as high confidence — but the very next clause says "looks unconfident". The conclusion the paragraph reaches is the correct one; the definition it uses to get there is wrong.The page now:
1 - H/log(k), one MINUS the normalised entropy— the correct definition, phrased in the same voice.entropy_confidence(probs)and, in place, recomputes the field from the reported probabilities for each of the two records so the reader sees the claim cross-checked. On the English checkpoint:probabilitiescome back at 4 dp while the agent computed the field from unrounded logits), and that each value is well below the correspondinganswer_confidence— 0.464 / 0.481 respectively — becauseanswer_confidenceismax(p)on the same distribution whileconfidenceis the wider, more conservative entropy scale.Why
Example 30 is the page that tells a caller what the four question types are and what each answer field means. It's the intro to schema design. When its definition of
confidenceinverts from the code's own, every downstream reader learns the wrong quantity — the very confusion the field vocabulary at the top oftest_confidence.py("confidence carries two definitions, and only one of them is the calibrated quantity") exists to prevent.The recomputation closes the loop the way examples 28 (this repo, already merged) does for its preset-page conclusions: instead of asserting a numeric property in prose and leaving the reader to trust it, the page derives the number from what it prints. That's what makes the claim checkable in place rather than a docstring.
How it was verified
The gate section in
tests/test_confidence.pymirrors the merged page-28 idiom:entropy_confidenceis AST-extracted from the example and exec'd withmathinjected, then driven againstconfidence_from_probson seven fabricated distributions (including k=1 and k=0 edges). The definitional ban fires on a literalOLD_PAGE_30block quoting main's wording and not on the fix; the positive rule (that the1 - H/log(k)definition and the reported-vs-recomputed print are both present) fires on the fix and not on main's block. No weights are loaded.Relationship to #972 and #973
All three PRs target the "example prose asserts something the code doesn't back" class, but the defects are independent:
answer_confidenceis a calibrated probability when the shipped temperatures put mean ECE at 0.466. Itsintentquestion hits a bucket that ships at T ≈ 1.0 (raw softmax).confidenceon a score, and gets it inverted. Nothing in docs(examples): example 40 must not call answer_confidence calibrated #972 or docs(examples): example 18 must not call the shipped answer_confidence calibrated #973 touches example 30.The three page-N sections in
tests/test_confidence.pyare all self-contained:EXAMPLE_30,HELPERS_30,OLD_PAGE_30,_src30,_helpers30,_called30,PROBS_READ,HELPER_CALLED,PRINTS_REPORTED_AND_RECOMPUTED,INVERTED_SCORE_DEF,SCORE_DEFINES_1_MINUS_H, and the fourtest_page30_*functions all carry the30suffix. The only textual overlap between the three PRs is the same anchor — the page-N block goes after the page-28 runner and before the finalprint("\n%d passed, %d failed"). Whichever merges last needs a trivial rebase; the merge is a straight append in a different region, not a rewrite.Checklist
main(a4a8921 release: 0.3.28)ruff check --select=E9,F63,F7,F82,F401,F811 --line-length=120cleanpython -m compileall -q laya/ tests/cleantokens = ...variable name in the example, unrelated to this change)