You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
docs(common): say what confidence_from_probs returns below k=2 - #1164
laya/common.py::confidence_from_probs documents itself as 1 - H(p) / log(k) and, for k < 2,
returns 1.0 without saying so. The docstring now names that floor. It also corrects a claim about
what the floor does: writing this PR I first documented it as making a one-option field ungateable
by min_confidence, measured that, and it was wrong -- the abstention gate is not a reader of
this number. Tests: tests/test_confidence.py checked the formula at k=2, 3 and 4 and never pinned
the floor, so nothing distinguished it from the guard around it.
Two files, no behaviour line changes: laya/common.py +11/-0 (docstring only), tests/test_confidence.py +13/-1 (four checks, plus the module docstring's formula row now marked (k >= 2)).
Why
# laya/common.py:802-823, after this change's docstringdefconfidence_from_probs(p, k):
"""Normalized Shannon entropy confidence: 1 - H(p) / log(k). ..."""ifk<2:
return1.0# :819-820 the floor, undocumented until nowp=p[:k]
ent=-(p*np.log(np.clip(p, 1e-12, 1.0))).sum()
returnfloat(np.clip(1.0-ent/math.log(k), 0.0, 1.0)) # :823 log(1)==log(0)==0 divides here
log(1) and log(0) are zero, so below two options there is no entropy scale to divide by; the
function hands back the maximum score it can produce, whatever p holds. Measured on 682ac671:
p=[1.0] k=1 -> 1.0
p=[0.0] k=1 -> 1.0 <- the value is ignored entirely
p=[0.4] k=1 -> 1.0
p=[] k=0 -> 1.0
thresholds 0.0/0.5/1.0 all cleared at k=1: [True, True, True]
And a one-option choice is reachable from the documented surface, not only from a hand-built
question: laya.structured bounds the option count only above (MAX_OPTIONS = 32; there is no MIN_OPTIONS), so
The part that needed measuring, because the obvious inference is false. The temptation is to read
"a one-option question scores the maximum" as "so min_confidence cannot reject it". It is the wrong
mechanism: laya/confidence.py::_gate_confidence takes answer_confidence first and falls back to
this entropy field only when that one is absent --
gate reads answer_confidence first: 0.4 (from {"confidence": 1.0, "answer_confidence": 0.4})
gate falls back to confidence: 1.0 (from {"confidence": 1.0})
flag_low_confidence at 0.9 on a one-option answer with answer_confidence=0.4 -> [{'answers': {'mood': {'confidence': 1.0, 'answer_confidence': 0.4, 'low_confidence': True}}}]
-- so an answer can be gated at a one-option question, and the floor does not stop it. The real path
does pass, for a different reason. laya/agent.py:1345-1347 builds p as a softmax over the row, and
over one logit that is exactly [1.0], so answer_confidence and confidence are both 1.0:
(one-option choice on convaiinnovations/laya, CPU, offline, state "I would like to cancel my order, please.".) The docstring says exactly that: the reported confidence reads as perfect to a caller who thresholds it, the gate is not that caller, and the gate
passes a real one-option answer because the distribution is degenerate.
The four checks (tests/test_confidence.py, the block at @@ -108,6 +108,18 @@, after the k=2/3/4
formula loop) pin the floor's value, that it ignores p, the k=0 case, and that it sits at or above
every threshold the validator accepts (check_min_confidence allows [0.0, 1.0]).
How it was verified
python -B tests/test_confidence.py # 230 passed, 0 failed (226 at pristine 682ac671 -> exactly the 4 new checks)
python -B scripts/test_suites.py # rc=0, "all 87 suites passed"; 499 passed, 1 skipped in 15.82s
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 # rc=0, All checks passed!
ruff check tests/test_confidence.py --select=E9,F63,F7,F82,F401,F811 --line-length=120 # rc=0
python -m compileall -q laya/ tests/ # rc=0
zensical build --strict --clean # rc=0, "No issues found", 1.38s
The 226 -> 230 delta is the four new checks and not other movement: the same file run against a tree
extracted from 682ac671 gives 226 passed, 0 failed.
Mutation harness on the guard, each mutant reverted before the next (branch committed first; git status clean again after every run):
mutant
edit
result
floor boundary
k < 2 -> k < 1 at common.py:819
227 passed, 3 failed -- FAIL entropy/k=1 is the maximum, not 0/0: got nan, want 1.0, FAIL entropy/k=1 ignores the value: got nan, want 1.0, FAIL entropy/k=1 is at or above every threshold value the validator accepts
floor never fires
k < 2 -> k < 0
suite aborts at tests/test_confidence.py:119 (entropy/k=0 is the maximum) with ValueError: expected a positive input from math.log(0) at common.py:823, plus RuntimeWarning: invalid value encountered in scalar divide. Recorded as an abort, not as a pass count.
Both new-boundary mutants are killed by the added checks, so they are not restating the formula loop
above them.
Returning the maximum score the function can produce, for an input it cannot score, is exactly the floor that needs naming. A reader who filters on the reported confidence sees a one-option question pass every threshold and has no way to tell that from a concentrated distribution.
The part of this PR I want to single out is that you documented the floor as making a one-option field ungateable by min_confidence, measured it, found that wrong and said so in the body. The gate reads answer_confidence first and only falls back here, so the two are different questions, and a real one-option answer passes for a different reason: a softmax over one logit is [1.0], so both numbers are 1.0. Correcting your own claim inside the PR that documents it is more useful than a clean-looking body.
tests/test_confidence.py pinned the formula at k=2, 3 and 4 and nothing distinguished the floor from the guard around it, which is why this survived. Four checks and the module docstring's (k >= 2) marker close that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya/common.py::confidence_from_probsdocuments itself as1 - H(p) / log(k)and, fork < 2,returns
1.0without saying so. The docstring now names that floor. It also corrects a claim aboutwhat the floor does: writing this PR I first documented it as making a one-option field ungateable
by
min_confidence, measured that, and it was wrong -- the abstention gate is not a reader ofthis number. Tests:
tests/test_confidence.pychecked the formula at k=2, 3 and 4 and never pinnedthe floor, so nothing distinguished it from the guard around it.
Two files, no behaviour line changes:
laya/common.py+11/-0 (docstring only),tests/test_confidence.py+13/-1 (four checks, plus the module docstring's formula row now marked(k >= 2)).Why
log(1)andlog(0)are zero, so below two options there is no entropy scale to divide by; thefunction hands back the maximum score it can produce, whatever
pholds. Measured on682ac671:And a one-option
choiceis reachable from the documented surface, not only from a hand-builtquestion:
laya.structuredbounds the option count only above (MAX_OPTIONS = 32; there is noMIN_OPTIONS), soThe part that needed measuring, because the obvious inference is false. The temptation is to read
"a one-option question scores the maximum" as "so
min_confidencecannot reject it". It is the wrongmechanism:
laya/confidence.py::_gate_confidencetakesanswer_confidencefirst and falls back tothis entropy field only when that one is absent --
-- so an answer can be gated at a one-option question, and the floor does not stop it. The real path
does pass, for a different reason.
laya/agent.py:1345-1347buildspas a softmax over the row, andover one logit that is exactly
[1.0], soanswer_confidenceandconfidenceare both1.0:(one-option
choiceonconvaiinnovations/laya, CPU, offline, state"I would like to cancel my order, please.".) The docstring says exactly that: the reportedconfidencereads as perfect to a caller who thresholds it, the gate is not that caller, and the gatepasses a real one-option answer because the distribution is degenerate.
The four checks (
tests/test_confidence.py, the block at@@ -108,6 +108,18 @@, after the k=2/3/4formula loop) pin the floor's value, that it ignores
p, the k=0 case, and that it sits at or aboveevery threshold the validator accepts (
check_min_confidenceallows [0.0, 1.0]).How it was verified
The 226 -> 230 delta is the four new checks and not other movement: the same file run against a tree
extracted from
682ac671gives226 passed, 0 failed.Mutation harness on the guard, each mutant reverted before the next (branch committed first;
git statusclean again after every run):k < 2->k < 1atcommon.py:819227 passed, 3 failed--FAIL entropy/k=1 is the maximum, not 0/0: got nan, want 1.0,FAIL entropy/k=1 ignores the value: got nan, want 1.0,FAIL entropy/k=1 is at or above every threshold value the validator acceptsk < 2->k < 0tests/test_confidence.py:119(entropy/k=0 is the maximum) withValueError: expected a positive inputfrommath.log(0)atcommon.py:823, plusRuntimeWarning: invalid value encountered in scalar divide. Recorded as an abort, not as a pass count.Both new-boundary mutants are killed by the added checks, so they are not restating the formula loop
above them.
Deconfliction and scanning
56 open PRs as at 2026-10-12T08:26Z.
laya/common.pyis touched by three open PRs -- perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687 (@@ -97,6 +97,373 @@), fix: make choice criteria permutation-invariant#779 #783(
@@ -103,6 +103,22 @@) and feat(encoder): add opt-in residual bottleneck adapters #921 (@@ -575,7 +575,7 @@,@@ -605,6 +605,21 @@,@@ -707,13 +722,14 @@) -- found by listing every open PR's files and hunks, not by title. Thischange is
@@ -804,6 +804,17 @@: below feat(encoder): add opt-in residual bottleneck adapters #921's lowest hunk and above the other two, disjoint.tests/test_confidence.pyis touched by no open PR. My own fix(confidence): clear the stale low_confidence marker at min_confidence 0.0 #1119 and docs(tl_kernels): the caller that pads these kernels is laya.fast, not base.pad_batch #1148 did edit it, but bothmerged at 2026-10-12T03:40:51Z and are in
682ac671-- which is why this branch was rebased, and whyevery line number above was re-read on the rebased tree rather than carried over. This change appends
at
@@ -108,6 +108,18 @@and edits line 3 of the module docstring (thechoice / scoreformula row,now marked
(k >= 2)).682ac671(release 0.4.4),upstream/mainat the same commit. Two commits on top:099e2589(
docs(common):) and22091153(test(confidence):).Checklist
main(682ac671, release 0.4.4)