Repository navigation
docs(examples): example 03 must give confidence per question type, not one formula - #1004
Merged
NandhaKishorM merged 3 commits intoOct 7, 2026
Conversation
…not one formula The page that teaches the shape of an answer said "`confidence` is 1 minus normalised entropy -- a scale that moves with the option count on the same answer", but `Agent._decode_answers` picks the formula by type: `confidence_from_probs` for `choice` and `score`, `max(float(p[1]), 1.0 - float(p[1]))` for `noul`. The page calls all three types in one `predict()`, so the sentence is false for the `noul` row of its own table, where `confidence` and `answer_confidence` print the same number -- which a normalised entropy never does (0.9/0.1 reads 0.531 as entropy, 0.900 as max(p); 0.6/0.4 reads 0.029 against 0.600).
…as the agents build
Four arms, weight-free, over the page the new reader is told to come back to:
* the ban on the single-formula sentence, witnessed against the wording as it
ships on main, plus the seven positive rules that page must now satisfy (and
main's page satisfies none of them);
* `_decode_answers`'s per-type `confidence` expressions read out of laya/agent.py
by AST -- all three answer dicts, entropy for choice/score, max(p[1], 1-p[1])
for noul, and the noul branch must not have switched to entropy underneath the
page;
* the numbers the page cites are recomputed from laya.common.confidence_from_probs,
so a cited 0.029 or 0.531 that drifts fails there rather than silently;
* the ONNX builder is held to the same two expressions, because the page describes
both paths and is not free to be true of only one.
tests/test_confidence.py goes 134 -> 138 passed, 0 failed.
…both builders The page's last sentence -- a calibration payload remaps `answer_confidence` and leaves `confidence` alone -- is a claim about `_decode_answers`, not about prose. Read the `apply_binning_map` call in each builder and hold its input to `ans_raw`, the raw `answer_confidence`, so wrapping a `confidence` expression in the map fails a test instead of quietly making the page wrong again.
4 tasks done
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 7, 2026
… its own types Seven arms in `tests/test_confidence.py`, inserted above the example-28 section so the hunk lands at pre-image line 399 -- clear of the nine open PRs appending at 603 and of #1004's section at 589. No weights, no pydantic, no network. * a ban on "`confidence` is normalized entropy" and on the sample note that blamed the label count alone, each witnessed against the sentence it replaces; * `_attributions()`: a formula has to appear in a sentence naming the question types that agent code actually uses it for, with backticked type tokens only, so "the choice is explicit" cannot be read as a `choice` question; * the same map derived from `Agent._decode_answers` and `OnnxAgent._decode_answers` with `ast` (one-hop resolution for the ONNX builder's `conf_score`), so the pages are held to the builders and not to a transcription; * structured.md's sample recomputed: `answer_confidence` = max(p) and `confidence` = 1 - H(p)/log(k) over the probabilities printed on the next line, with the named option count equal to the label count and the field named as the type the compiler gives it; * the Q&A page's three sample answers recomputed per type, its noul row shown to be `max(p_true, 1 - p_true)` and not the 0.45 its prose cites as the entropy alternative, and its three-row table held to those sample blocks. Against main's pages 6 of the 7 arms fail; the suite goes 134 -> 141 on this branch.
Owner
|
Merged in 0.3.29. Example 03 is "reading the result", so giving one formula for Confirmed on real weights before merging:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
examples/03_reading_the_result.pyis the page the repo sends a reader to for the shape of ananswer — its own docstring says "This is the example to come back to when writing your own glue
code" — and it closed its field tour with:
Agent._decode_answersdoes not run one formula. It picks by question type(
laya/agent.py:1366-1402):choiceandscore→round(confidence_from_probs(p, k), 4), i.e.1 - H(p) / log(k);noul→round(max(float(p[1]), 1.0 - float(p[1])), 4), a probability of the reported side.The page calls all three types in one
predict(), so that sentence is false for thenoulrowof the very table printed six lines above it. The page's own example is the counter-proof: a
noulanswer reports the same number in
confidenceand inanswer_confidence, and a normalisedentropy of a two-option distribution is never
max(p).Measured with the repo's own functions (no weights loaded):
confidence_from_probs(p, 2)noul's rule /answer_confidence[0.9, 0.1][0.75, 0.25][0.6, 0.4]The page now states the two formulas with their types, calls the scales incomparable, keeps
answer_confidenceas the gate field, and adds the one scope note it was missing: a calibrationpayload carrying
binning_mapremapsanswer_confidenceand leavesconfidencealone(
laya/agent.py:1359-1362,laya/onnx_agent.py:686-691).Lines are 207 insertions / 4 deletions across
examples/03_reading_the_result.pyandtests/test_confidence.py. No behaviour changes: every formula, number and remap the gate assertsis what the code already produces.
Why
A reader gating on
confidenceat, say, 0.85 gets two different tests out of one threshold: for achoiceit asks whether the distribution is concentrated, and for anoulit asks whetherp(true)is past 0.85 — while anoulatp(true)=0.6reads aconfidenceof 0.600 on the page'sown table, and the same mass as a
choicewould read 0.029. The sentence taught the reader to treatthose as the same quantity.
This is the same defect class as #972, #973, #974 (examples 40, 18, 30) and as #980
(
DecisionResult's docstring), on the page a beginner reads first. #980 holdslaya/structured.py,not this file, so the two PRs share no line.
docs/structured.mdanddocs/questions-and-answers.mdstill carry the blanket sentence ("confidenceisnormalized entropy" with no type split). They are deliberately left out here so this PR stays one
page and one gate; they are the follow-up.
The gate
Five arms in
tests/test_confidence.py, weight-free, prefixpage-03/:ban is asserted to fire on the pre-fix wording as it ships on main — a ban with no witness is a
guess. The page must then name the entropy formula against
choice/score, write themax(p[1], 1 - p[1])form, attribute it tonoul, call the scales incomparable, keepanswer_confidenceasmax(p), name the gate field with its page, and state the remap scope;main's page satisfies none of those, so the rules could not have passed before.
_confidence_exprswalks_decode_answersby ASTand returns the source of each answer dict's
confidencevalue — all three types must appear,choice/scoremust go throughconfidence_from_probs,noulmust contain1.0 - float(p[1])and must not have switched to entropy underneath the page.laya.common.confidence_from_probsand matched back into the page's text, so a cited0.029or0.531that drifts by one digit fails there. The arm also asserts that anoul'sconfidenceequals
answer_confidence— the reason one sentence could never cover both.test_the_remap_reaches_only_answer_confidencereads theapply_binning_mapcall in both builders and holds its input to
ans_raw, the rawanswer_confidence. Wrappinga
confidenceexpression in the map now breaks a test instead of quietly making the page wrongagain.
_decode_answersinlaya/onnx_agent.pyisheld to the same two expressions (its pre-computed
conf_scoreis resolved to what it is boundto, so the arm compares formulas, not variable names).
Mutation harness
laya-bench/mutate_example03_confidence.py: 14 mutants, 14 killed, SURVIVORS[], with the M0 baseline green at 139 passed / 0 failed and
git statusempty after the run. Eachfailure is attributable to a named arm — reverting the page (M1) fails 3 arms, un-attributing the
noulformula (M2), dropping "not comparable" (M3), deleting the remap sentence (M4) and swappinganswer_confidence's definition (M7) each fail exactly 1; moving a cited digit by one (M50.029 → 0.028, M60.531 → 0.532) fails the earned-numbers arm; changing the agents fails M8 (torchnoul→ entropy, 2 arms), M9 (ONNX
noul→ rawp(true)), M10 (ONNXchoice/scoreoffconfidence_from_probs) and M13/M14 (the map put onconfidenceas well, one builder at a time).Three self-checks prove the gate itself is not decorative: M11 removes the whitespace collapsing from
_printed_textand the multi-line sentences stop matching (2 arms), M12 blinds the AST scan to allbut the
nouldict (2 arms).tests/test_confidence.pygoes 134 → 139 passed, 0 failed; the count was measured against apristine
a4a8921copy of the repo, and the same run on the pristine tree still prints 134 / 0.Neighbours, at the final commit:
tests/test_structured_docs.py116 checks / 0 failed (it holds theusage keys on this same page, untouched here),
tests/test_evals_api.py48 / 0,tests/test_agent_gate.py11 / 0,tests/test_hooks_api.py453 / 0.ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120→ All checks passed!;python -m compileall -q laya/ tests/ examples/→ clean.Deconfliction
Run before editing, hunk by hunk over the 70 open PRs:
examples/03_reading_the_result.py— the only other open PR with a hunk in it is perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687, atpre-image lines 14 and 24 (the banner). This PR's hunk is
@@ -94,10 +94,14 @@, so it coverspre-image 94-103 — 70 lines clear of perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687's second hunk.
tests/test_confidence.py— nine open PRs (docs(agent): predict_long gate comments must describe the None contract, not claim a report #996, docs(presets): questions-and-answers must count five, not three #981, docs(structured): DecisionResult's confidence is per-type, not uniformly entropy #980, docs(tl_kernels): the multiple-of-16 comes from tokens, not rows #979, docs(examples): example 26 must count triage_questions()'s three yes/no fields #978, docs(evals): the --min-confidence gate flags the answer, it does not overwrite it #976, docs(examples): example 30 must call score confidence 1 - H/log(k), not the entropy #974, docs(examples): example 18 must not call the shipped answer_confidence calibrated #973, docs(examples): example 40 must not call answer_confidence calibrated #972,all mine) append their confidence-doc gates at pre-image line 603, which is the tail of the
page-28 driver loop. This section is inserted above that loop as
@@ -589,6 +589,205 @@, coveringpre-image 589-594, so its context stops 6 lines before their context begins.
tests/test_structured_docs.pywas considered as the host and rejected: perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687 has hunks at its 513and 547, and its example-03 gate lives inside
main()in that same region.Checklist
main