Skip to content

fix(confidence): clear stale low_confidence flag on re-evaluation (#910) - #911

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Sarthak-Pandey:fix/confidence-stale-low-confidence
Oct 4, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Sarthak-Pandey:fix/confidence-stale-low-confidence

Conversation

@Sarthak-Pandey

@Sarthak-Pandey Sarthak-Pandey commented Oct 3, 2026 •

Copy link
Copy Markdown

flag_low_confidence only ever set low_confidence: True and never cleared it. When a result dict was re-evaluated with a more permissive threshold (e.g. reused across hook calls, cached, or passed through a pipeline stage that relaxes the gate), the stale flag remained, causing downstream callers and integrations to treat the answer as abstained even though it now satisfied the current threshold.

Fix: add an else branch that calls �.pop('low_confidence', None) when the confidence value clears or cannot be evaluated against the threshold, so each call produces an accurate, self-consistent state.

The 0.0 no-op path (early return) is unaffected: when min_confidence is 0.0 the function returns immediately and no answers are touched, preserving the existing guarantee that a 0.0 threshold is a no-op.

Reproducer (before fix):
answer = {'choice': 'refund', 'answer_confidence': 0.8}
result = {'answers': {'intent': answer}}
flag_low_confidence([result], 0.9) # sets low_confidence: True
flag_low_confidence([result], 0.5) # was: flag stays True
assert 'low_confidence' not in answer # now passes

tests/test_confidence.py: add three assertions under 'stale/' prefix:
- strict threshold sets the flag
- permissive threshold on the same dict clears the stale flag
- a clean answer gains no spurious key

What

Why

How it was verified

python tests/test_router.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes #910

…ndhaKishorM#910)

flag_low_confidence only ever set low_confidence: True and never
cleared it. When a result dict was re-evaluated with a more permissive
threshold (e.g. reused across hook calls, cached, or passed through a
pipeline stage that relaxes the gate), the stale flag remained, causing
downstream callers and integrations to treat the answer as abstained
even though it now satisfied the current threshold.

Fix:
- In flag_low_confidence: pop low_confidence when an answer's confidence
  meets or exceeds the threshold, or when confidence is unusable.
- In apply_confidence_gate: clear stale low_confidence when gating with
  min_confidence == 0.0, ensuring 0.0 correctly passes all usable answers.

Add regression tests in tests/test_confidence.py covering:
- flag_low_confidence clearing stale flag on re-evaluation with permissive threshold
- clean answers remaining unflagged
- apply_confidence_gate updating abstention state from abstained to passed
- apply_confidence_gate at 0.0 clearing stale flags
- unusable confidence clearing stale flags and marking unevaluated.
@Sarthak-Pandey
Sarthak-Pandey force-pushed the fix/confidence-stale-low-confidence branch from c4f1a41 to e948dbd Compare October 3, 2026 18:58
@NandhaKishorM
NandhaKishorM merged commit e8a7b62 into NandhaKishorM:main Oct 4, 2026
30 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.27, closing #910.

Two PRs arrived for this and I took yours because it is the focused one. @PerryLink's #925 makes the same fix but bundles it with the TensorFlow guard from #924, and AGENTS.md asks for one logical change per pull request for exactly this reason: the two fixes are unrelated, they land in different files, and either could need reverting without the other.

The fix itself is right, and the part I checked is that it clears in all three paths rather than just the obvious one: a previously-flagged answer that now clears the threshold, the else branch, and the min_confidence == 0.0 case where a scalar gate is a no-op. Only setting the flag and never clearing it is the bug; popping it in one place and not the others would have left the same bug for a narrower input.

The reasoning in the docstring is what makes this stay fixed. "Removed if a previously-flagged answer now clears it (e.g. when a result dict is reused or re-evaluated with a different threshold)" states the contract, so the next person to touch flag_low_confidence knows clearing is deliberate rather than incidental.

Verified: tests/test_confidence.py passes, the shared suite list reports all 79 suites passed, and a decision diff against 0.3.26 on the real checkpoints shows zero decision flips.

Thank you, and thanks for filing #910 with the reuse scenario spelled out.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Correction to my comment above, because I stated something about your code that is not true.

I wrote that the fix "clears in all three paths ... and the min_confidence == 0.0 case where a scalar gate is a no-op". That is wrong. flag_low_confidence starts with

is_map = isinstance(min_confidence, dict)
if not is_map and min_confidence == 0.0:
    return

so a 0.0 call returns before the loop and leaves a previously-set flag in place. Measured on the merged code:

flag_low_confidence(r, 0.9)  then  flag_low_confidence(r, 0.5)   ->  {}                          cleared
flag_low_confidence(r, 0.9)  then  flag_low_confidence(r, 0.0)   ->  {'low_confidence': True}    retained

I misread a reordered grep of your diff as placing the 0.0 pop inside the loop. The merge is unaffected and nothing needs changing in what shipped; I just do not want that claim standing on the record.

Your fix is still correct and still closes the case #910 leads with, which is re-evaluation under a more permissive non-zero threshold. Two residues remain, and I have written them up on #910 rather than reopening this:

  • flag_low_confidence(..., 0.0) keeps a stale flag, which is consistent with the README's "treats 0.0 as a no-op because nothing can fall below it" but inconsistent with apply_confidence_gate(..., 0.0), which does clear it.
  • apply_confidence_gate(..., None) keeps a stale flag too, for the same "no gate configured, touch nothing" reason.

Whether those are bugs or the documented no-op depends on whether "no gate" means "do not evaluate" or "evaluate and pass everything", and the two functions currently answer differently. That is a contract question rather than a defect in your patch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: apply_confidence_gate keeps stale low_confidence state across reused results

2 participants