Repository navigation
fix(confidence): clear stale low_confidence flag on re-evaluation (#910) - #911
Conversation
…ndhaKishorM#910) flag_low_confidence only ever set low_confidence: True and never cleared it. When a result dict was re-evaluated with a more permissive threshold (e.g. reused across hook calls, cached, or passed through a pipeline stage that relaxes the gate), the stale flag remained, causing downstream callers and integrations to treat the answer as abstained even though it now satisfied the current threshold. Fix: - In flag_low_confidence: pop low_confidence when an answer's confidence meets or exceeds the threshold, or when confidence is unusable. - In apply_confidence_gate: clear stale low_confidence when gating with min_confidence == 0.0, ensuring 0.0 correctly passes all usable answers. Add regression tests in tests/test_confidence.py covering: - flag_low_confidence clearing stale flag on re-evaluation with permissive threshold - clean answers remaining unflagged - apply_confidence_gate updating abstention state from abstained to passed - apply_confidence_gate at 0.0 clearing stale flags - unusable confidence clearing stale flags and marking unevaluated.
c4f1a41 to
e948dbd
Compare
|
Merged for 0.3.27, closing #910. Two PRs arrived for this and I took yours because it is the focused one. @PerryLink's #925 makes the same fix but bundles it with the TensorFlow guard from #924, and The fix itself is right, and the part I checked is that it clears in all three paths rather than just the obvious one: a previously-flagged answer that now clears the threshold, the The reasoning in the docstring is what makes this stay fixed. "Removed if a previously-flagged answer now clears it (e.g. when a result dict is reused or re-evaluated with a different threshold)" states the contract, so the next person to touch Verified: Thank you, and thanks for filing #910 with the reuse scenario spelled out. |
|
Correction to my comment above, because I stated something about your code that is not true. I wrote that the fix "clears in all three paths ... and the is_map = isinstance(min_confidence, dict)
if not is_map and min_confidence == 0.0:
returnso a I misread a reordered grep of your diff as placing the Your fix is still correct and still closes the case #910 leads with, which is re-evaluation under a more permissive non-zero threshold. Two residues remain, and I have written them up on #910 rather than reopening this:
Whether those are bugs or the documented no-op depends on whether "no gate" means "do not evaluate" or "evaluate and pass everything", and the two functions currently answer differently. That is a contract question rather than a defect in your patch. |
flag_low_confidence only ever set low_confidence: True and never cleared it. When a result dict was re-evaluated with a more permissive threshold (e.g. reused across hook calls, cached, or passed through a pipeline stage that relaxes the gate), the stale flag remained, causing downstream callers and integrations to treat the answer as abstained even though it now satisfied the current threshold.
Fix: add an else branch that calls �.pop('low_confidence', None) when the confidence value clears or cannot be evaluated against the threshold, so each call produces an accurate, self-consistent state.
The 0.0 no-op path (early return) is unaffected: when min_confidence is 0.0 the function returns immediately and no answers are touched, preserving the existing guarantee that a 0.0 threshold is a no-op.
Reproducer (before fix):
answer = {'choice': 'refund', 'answer_confidence': 0.8}
result = {'answers': {'intent': answer}}
flag_low_confidence([result], 0.9) # sets low_confidence: True
flag_low_confidence([result], 0.5) # was: flag stays True
assert 'low_confidence' not in answer # now passes
tests/test_confidence.py: add three assertions under 'stale/' prefix:
- strict threshold sets the flag
- permissive threshold on the same dict clears the stale flag
- a clean answer gains no spurious key
What
Why
How it was verified
Checklist
mainFixes #910