Skip to content

docs(common): say what confidence_from_probs returns below k=2 - #1164

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:docs-entropy-confidence-floor-044
Oct 12, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:docs-entropy-confidence-floor-044

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya/common.py::confidence_from_probs documents itself as 1 - H(p) / log(k) and, for k < 2,
returns 1.0 without saying so. The docstring now names that floor. It also corrects a claim about
what the floor does: writing this PR I first documented it as making a one-option field ungateable
by min_confidence, measured that, and it was wrong -- the abstention gate is not a reader of
this number. Tests: tests/test_confidence.py checked the formula at k=2, 3 and 4 and never pinned
the floor, so nothing distinguished it from the guard around it.

Two files, no behaviour line changes: laya/common.py +11/-0 (docstring only),
tests/test_confidence.py +13/-1 (four checks, plus the module docstring's formula row now marked
(k >= 2)).

Why

# laya/common.py:802-823, after this change's docstring
def confidence_from_probs(p, k):
    """Normalized Shannon entropy confidence: 1 - H(p) / log(k). ..."""
    if k < 2:
        return 1.0                      # :819-820  the floor, undocumented until now
    p = p[:k]
    ent = -(p * np.log(np.clip(p, 1e-12, 1.0))).sum()
    return float(np.clip(1.0 - ent / math.log(k), 0.0, 1.0))   # :823  log(1)==log(0)==0 divides here

log(1) and log(0) are zero, so below two options there is no entropy scale to divide by; the
function hands back the maximum score it can produce, whatever p holds. Measured on 682ac671:

  p=[1.0]    k=1 -> 1.0
  p=[0.0]    k=1 -> 1.0        <- the value is ignored entirely
  p=[0.4]    k=1 -> 1.0
  p=[]       k=0 -> 1.0
thresholds 0.0/0.5/1.0 all cleared at k=1: [True, True, True]

And a one-option choice is reachable from the documented surface, not only from a hand-built
question: laya.structured bounds the option count only above (MAX_OPTIONS = 32; there is no
MIN_OPTIONS), so

structured.questions_from_json_schema({"type": "object", "properties": {"mood": {"enum": ["solo"]}}})
  -> {'mood': {'type': 'choice', 'instructions': 'What is `mood`?', 'criteria': {'solo': None}}}

The part that needed measuring, because the obvious inference is false. The temptation is to read
"a one-option question scores the maximum" as "so min_confidence cannot reject it". It is the wrong
mechanism: laya/confidence.py::_gate_confidence takes answer_confidence first and falls back to
this entropy field only when that one is absent --

gate reads answer_confidence first: 0.4        (from {"confidence": 1.0, "answer_confidence": 0.4})
gate falls back to confidence:      1.0        (from {"confidence": 1.0})
flag_low_confidence at 0.9 on a one-option answer with answer_confidence=0.4 -> [{'answers': {'mood': {'confidence': 1.0, 'answer_confidence': 0.4, 'low_confidence': True}}}]

-- so an answer can be gated at a one-option question, and the floor does not stop it. The real path
does pass, for a different reason. laya/agent.py:1345-1347 builds p as a softmax over the row, and
over one logit that is exactly [1.0], so answer_confidence and confidence are both 1.0:

single-logit softmax: [1.0] -> max(p) = 1.0
min_confidence=0.0   -> confidence=1.0 answer_confidence=1.0 low_confidence=None abstention=passed
min_confidence=0.9   -> confidence=1.0 answer_confidence=1.0 low_confidence=None abstention=passed
min_confidence=1.0   -> confidence=1.0 answer_confidence=1.0 low_confidence=None abstention=passed

(one-option choice on convaiinnovations/laya, CPU, offline, state
"I would like to cancel my order, please.".) The docstring says exactly that: the reported
confidence reads as perfect to a caller who thresholds it, the gate is not that caller, and the gate
passes a real one-option answer because the distribution is degenerate.

The four checks (tests/test_confidence.py, the block at @@ -108,6 +108,18 @@, after the k=2/3/4
formula loop) pin the floor's value, that it ignores p, the k=0 case, and that it sits at or above
every threshold the validator accepts (check_min_confidence allows [0.0, 1.0]).

How it was verified

python -B tests/test_confidence.py   # 230 passed, 0 failed (226 at pristine 682ac671 -> exactly the 4 new checks)
python -B scripts/test_suites.py     # rc=0, "all 87 suites passed"; 499 passed, 1 skipped in 15.82s
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120    # rc=0, All checks passed!
ruff check tests/test_confidence.py --select=E9,F63,F7,F82,F401,F811 --line-length=120   # rc=0
python -m compileall -q laya/ tests/ # rc=0
zensical build --strict --clean      # rc=0, "No issues found", 1.38s

The 226 -> 230 delta is the four new checks and not other movement: the same file run against a tree
extracted from 682ac671 gives 226 passed, 0 failed.

Mutation harness on the guard, each mutant reverted before the next (branch committed first;
git status clean again after every run):

mutant edit result
floor boundary k < 2 -> k < 1 at common.py:819 227 passed, 3 failed -- FAIL entropy/k=1 is the maximum, not 0/0: got nan, want 1.0, FAIL entropy/k=1 ignores the value: got nan, want 1.0, FAIL entropy/k=1 is at or above every threshold value the validator accepts
floor never fires k < 2 -> k < 0 suite aborts at tests/test_confidence.py:119 (entropy/k=0 is the maximum) with ValueError: expected a positive input from math.log(0) at common.py:823, plus RuntimeWarning: invalid value encountered in scalar divide. Recorded as an abort, not as a pass count.

Both new-boundary mutants are killed by the added checks, so they are not restating the formula loop
above them.

Deconfliction and scanning

56 open PRs as at 2026-10-12T08:26Z.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (682ac671, release 0.4.4)
  • Tests pass locally
  • Docs or examples updated when the public API changed (docstring + tests; no API change)

@NandhaKishorM
NandhaKishorM merged commit 0379449 into NandhaKishorM:main Oct 12, 2026
44 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.5.

Confirmed, including that the value of p is ignored entirely:

p=[1.0]  k=1 -> 1.0
p=[0.0]  k=1 -> 1.0
p=[0.4]  k=1 -> 1.0
p=[]     k=0 -> 1.0

Returning the maximum score the function can produce, for an input it cannot score, is exactly the floor that needs naming. A reader who filters on the reported confidence sees a one-option question pass every threshold and has no way to tell that from a concentrated distribution.

The part of this PR I want to single out is that you documented the floor as making a one-option field ungateable by min_confidence, measured it, found that wrong and said so in the body. The gate reads answer_confidence first and only falls back here, so the two are different questions, and a real one-option answer passes for a different reason: a softmax over one logit is [1.0], so both numbers are 1.0. Correcting your own claim inside the PR that documents it is more useful than a clean-looking body.

tests/test_confidence.py pinned the formula at k=2, 3 and 4 and nothing distinguished the floor from the guard around it, which is why this survived. Four checks and the module docstring's (k >= 2) marker close that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants