Skip to content

fix(confidence): apply_confidence_gate left a stale low_confidence on a re-gate - #925

Closed
PerryLink wants to merge 2 commits into
NandhaKishorM:mainfrom
PerryLink:fix/regated-confidence-gate
Closed

PerryLink wants to merge 2 commits into
NandhaKishorM:mainfrom
PerryLink:fix/regated-confidence-gate

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

What this fixes

A
esults payload gated once and then gated again kept the first call's flag. lag_low_confidence
only ever adds low_confidence: True (confidence.py:158) and never takes one back — correct
for a function whose docstring says the flag "is added", but wrong underneath a second gate.
abstention was recomputed each call, so the two halves of the report disagreed:

a = {"choice": "refund", "answer_confidence": 0.8}
r = {"answers": {"intent": a}}

apply_confidence_gate([r], min_confidence=0.9)
# {'low_confidence': True, 'abstention': 'abstained', 'abstention_threshold': 0.9}

apply_confidence_gate([r], min_confidence=0.5)      # 0.8 clears 0.5
# before: {'low_confidence': True, 'abstention': 'abstained', 'abstention_threshold': 0.5}
#                        ^ stale              ^ stale          ^ updated

The result is self-contradictory: the threshold says 0.5 and the verdict says abstained, a
decision that was never made at that threshold. A fresh dict at 0.5 gives
{'low_confidence': None, 'abstention': 'passed', 'abstention_threshold': 0.5}.

Reusing a result dict is the normal way this API is used, because the function mutates in place and
returns None
— there is nothing to signal that a second call needs fresh input. Reported in #910.

The fix

Clear low_confidence before delegating, so what the gate reports is decided at the threshold of
this call and nowhere else. The answers are now gathered once into gated and walked twice,
instead of walking results twice with the same filter — clearing and state-writing are two rules
over one set.

Deliberate choices tested:

  • Cleared in apply_confidence_gate, not in flag_low_confidence. That function's contract is "the flag is added", it is public and exported in laya.__all__, and a caller using it directly on an accumulated payload may be relying on the flag persisting. The gate is the layer that claims to report a decision, and it already owns writing abstention/abstention_threshold onto the answers.
  • Both directions asserted. Loosening clears; tightening sets again. Clearing in one direction only would be a pass-through rather than a re-decision.
  • 0.0 clears too — a gate that ran and nothing can fail reports passed with no flag, which is the state a fresh 0.0 gate produces.
  • Per-bucket maps covered, since the flag there is decided by the map's threshold for that answer's bucket.
  • The strongest assertion is equivalence, not the fields: after re-gating, the answer must equal what a fresh gate at that threshold produces. That is the property a caller needs, and it catches a flag/state disagreement that checking either alone would miss.

Tests

Eleven checks added to tests/test_confidence.py: looser, tighter, 0.0 and map re-gates, plus
the fresh-gate equivalence.

Confirmed the tests catch the bug — with the clearing loop removed:

126 passed, 7 failed
FAIL gate/re-gated looser matches a fresh gate:
     got  {..., 'low_confidence': True, 'abstention': 'abstained', 'abstention_threshold': 0.3}
     want {...,                        'abstention': 'passed',   'abstention_threshold': 0.3}
python tests/test_confidence.py            133 passed, 0 failed   (122 before)
python tests/test_conformal_abstention.py   20 passed, 0 failed
python tests/test_batch.py                  78 passed, 0 failed
python tests/test_cli.py                    all CLI tests passed
python tests/test_structured.py             all structured tests passed
python -m pytest tests/test_serve.py tests/test_router_batch.py \
                 tests/test_system_one_lang.py tests/test_audit_regressions.py -q
308 passed
python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!

One thing worth your view

This makes a re-gate authoritative for the answers it is given, which is a behaviour change for
anyone calling the gate twice deliberately to accumulate the strictest verdict across thresholds.
I do not think that is a supported pattern — the function reports "the gate's state" and echoes a
single threshold, so a mixture of two is not representable in its own output — but if it was
intended, the alternative shape is an explicit accumulate argument rather than changing the default.
Happy to switch it.

…d when it cannot

On transformers 4.x `build_model` reaches `transformers.modeling_utils` for `no_init_weights`, and
importing that module runs a chain that ends in `import tensorflow`:

```
laya/common.py:579  _no_init_weights
  -> transformers/modeling_utils.py:70
  -> transformers/loss/loss_utils.py:21
  -> transformers/loss/loss_d_fine.py:21
  -> transformers/loss/loss_for_object_detection.py:32
  -> transformers/image_transforms.py:48
  -> tensorflow/__init__.py:40  ->  Fatal Python error: Bus error
```

Laya's model path is torch-only, so in an environment where TensorFlow is installed but cannot
load, `laya.load()` dies **natively** — not an exception that can be caught, the interpreter is
gone — because of a library nothing here uses. Reported in NandhaKishorM#915 with the full traceback.

The fix is the one the reporter suggested: `os.environ.setdefault("USE_TF", "0")` at the top of
`laya/__init__.py`, above every import.

**`setdefault`, not assignment.** A caller who set `USE_TF` deliberately keeps it, including `1`
for someone who wants TensorFlow in the same process. Verified for both `"1"` and `"0"`.

**It has to be in `__init__.py` and above the imports**, because transformers decides TensorFlow's
availability when it is first imported. `import laya` deliberately imports no transformers at all —
the torch-backed names are lazy (`_LAZY_ATTRS`) — so this runs first in the normal case, and that
is checked rather than assumed: a subprocess asserts `"tensorflow" not in sys.modules` after
`import laya` and that no `transformers.*` module was pulled in by it.

## The repository already knew this

`USE_TF=0` is set by CI, by the Dockerfile, and by six test files as a precondition:

```
tests/test_load_errors.py:30          tests/test_download.py:20
tests/test_head_checkpointing.py:12   tests/test_empty_questions.py:13
tests/test_calibration_persistence.py:16   sdk_server_fixture.py:9
```

and `grep USE_TF laya/` returned **nothing** before this change. The test suite worked around a
hazard the library created and never defended against, so a plain `import laya` script hit it. This
moves the knowledge to where it takes effect — the same shape as the `lang_temperatures` drift in
NandhaKishorM#428.

## Tests

Five checks in `tests/test_lazy_import.py`, in subprocesses because an in-process assertion could
not distinguish "the guard worked" from "it was too late but harmless":

```
USE_TF/import laya succeeds with tensorflow blocked    -> 0
USE_TF/is set to 0 on import                           -> '0'
USE_TF/tensorflow is not imported                      -> False
USE_TF/a caller's '1' is preserved                     -> '1'
USE_TF/a caller's '0' is preserved                     -> '0'
```

The TensorFlow blocker raises `ImportError` rather than segfaulting, so if laya's import chain
reaches for TensorFlow at all the check fails loudly instead of taking the test process with it.

**Confirmed the tests catch the bug:** with the guard line replaced by `pass`, the suite reports
`12 passed, 1 failed — USE_TF/is set to 0 on import: got 'None', want '0'`.

```
$ python tests/test_lazy_import.py     13 passed, 0 failed   (9 before)
$ python tests/test_imports.py         All imports tests passed.
$ python -m pytest tests/test_serve.py tests/test_router_batch.py \
                   tests/test_system_one_lang.py tests/test_audit_regressions.py -q
308 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

## Limitation, stated plainly

**I could not reproduce the reported crash, and this is not a fix I observed working.** This box
runs transformers **5.17.0**, where `transformers.initialization.no_init_weights` exists and the
`modeling_utils` fallback is never taken — so the TensorFlow import chain does not run here at all,
and the crash needs a *broken* TensorFlow to trigger. The evidence I have is:

- the reporter's traceback, which names `common.py:579` and the exact chain above;
- the reporter's own controls — `USE_TF=0` makes the same environment work in ~5 s, and a fresh
  venv (transformers 5) never crashes;
- that the guard is set before transformers can be imported, measured;
- that `setdefault` preserves a caller's value, measured.

So the mechanism is confirmed and the guard is verified to do what it claims; the end-to-end crash
is verified only by the reporter's environment. Worth a note on NandhaKishorM#915 asking them to confirm.
…` on a re-gate

A `results` payload gated once and then gated again kept the first call's flag, because
`flag_low_confidence` only ever **adds** `low_confidence: True` (`confidence.py:158`) and never takes
one back — correct for a function whose docstring says the flag "is added", but wrong underneath a
second gate. `abstention` *was* recomputed each call, so the two halves of the report disagreed:

```python
a = {"choice": "refund", "answer_confidence": 0.8}
r = {"answers": {"intent": a}}

apply_confidence_gate([r], min_confidence=0.9)
# {'low_confidence': True, 'abstention': 'abstained', 'abstention_threshold': 0.9}

apply_confidence_gate([r], min_confidence=0.5)      # 0.8 clears 0.5
# before: {'low_confidence': True, 'abstention': 'abstained', 'abstention_threshold': 0.5}
#                        ^ stale              ^ stale          ^ updated
```

The result is **self-contradictory**: the threshold says 0.5 and the verdict says `abstained`, a
decision that was never made at that threshold. A fresh dict at 0.5 gives
`{'low_confidence': None, 'abstention': 'passed', 'abstention_threshold': 0.5}`.

Reusing a result dict is the normal way this API is used, because the function **mutates in place
and returns `None`** — there is nothing to signal that a second call needs fresh input. Reported in
NandhaKishorM#910.

## The fix

Clear `low_confidence` before delegating, so what the gate reports is decided at the threshold of
*this* call and nowhere else:

```python
for a in gated:
    a.pop("low_confidence", None)

flag_low_confidence(results, min_confidence)
```

The answers are now gathered once into `gated` and walked twice, instead of walking `results` twice
with the same filter — the clearing and the state-writing are two rules over one set.

Deliberate choices tested:

- **Cleared in `apply_confidence_gate`, not in `flag_low_confidence`.** That function's contract is "the flag is added", it is public and exported in `laya.__all__`, and someone calling it directly on a accumulated payload may well be relying on the flag persisting. The gate is the layer that claims to *report* a decision, and it already owns writing `abstention`/`abstention_threshold` onto the answers — so it is where an answer's gate state should be made whole.
- **The direction is asserted both ways.** Loosening clears the flag; tightening sets it again. Clearing only in one direction would be a pass-through rather than a re-decision.
- **`0.0` clears too.** A re-gate at 0.0 is a gate that ran and nothing can fail, so it reports `passed` and carries no flag — the same state a fresh 0.0 gate produces, which the existing `flag_low_confidence` no-op is what gives it.
- **A per-bucket map re-gate is covered as well**, since the flag there is decided by the map's threshold for that answer's bucket rather than by the scalar the earlier call used.
- **The strongest assertion is equivalence, not the individual fields:** after re-gating, the answer must equal what a fresh gate at that threshold produces. That is the property a caller actually needs, and it catches disagreements between the flag and the state that checking either alone would miss.

## Tests

Eleven checks added to `tests/test_confidence.py`, covering looser, tighter, `0.0` and map
re-gates, plus the fresh-gate equivalence.

**Confirmed the tests catch the bug:** with the clearing loop removed, the suite reports
`126 passed, 7 failed`, including the exact contradictory state from the issue —

```
FAIL gate/re-gated looser matches a fresh gate:
     got  {..., 'low_confidence': True, 'abstention': 'abstained', 'abstention_threshold': 0.3}
     want {...,                        'abstention': 'passed',   'abstention_threshold': 0.3}
```

```
$ python tests/test_confidence.py           133 passed, 0 failed   (122 before)
$ python tests/test_conformal_abstention.py  20 passed, 0 failed
$ python tests/test_batch.py                 78 passed, 0 failed
$ python tests/test_cli.py                   all CLI tests passed
$ python tests/test_structured.py            all structured tests passed
$ python -m pytest tests/test_serve.py tests/test_router_batch.py \
                   tests/test_system_one_lang.py tests/test_audit_regressions.py -q
308 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: this makes a re-gate authoritative for the answers it is given, which is a
behaviour change for anyone deliberately calling the gate twice to accumulate the *strictest*
verdict across thresholds. I do not think that is a supported pattern — the function reports "the
gate's state" and echoes one threshold, so a mixture of two thresholds is not representable in its
own output — but it is worth a word from the maintainer if it was intended. The alternative shape
would be to add a separate accumulate flag rather than change the default.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Closing this one, because both halves of it shipped in 0.3.27 through separate pull requests.

The reason I split them rather than taking this PR is AGENTS.md: "Keep it focused; split unrelated work into another PR." The two fixes live in different files, close different issues, and either could need reverting without the other. A laya/__init__.py import guard and a confidence-gate state bug have nothing to do with each other beyond landing the same day.

That is a process note, not a criticism of either fix. Both were right, and your version of the confidence fix was equivalent to the one that shipped.

One thing from this PR that did not land and that I would take as its own change: tests/test_lazy_import.py is the right place to guard the __init__.py surface, and you added coverage there. If you think the clearing logic in apply_confidence_gate needs a case #911 does not cover, that is a focused follow-up and I would review it quickly.

@PerryLink

Copy link
Copy Markdown
Contributor Author

You are right and I was wrong — I checked before replying, and the stacking is real.

fix/regated-confidence-gate was branched from fix/use-tf-guard, not from main:

* 1db7c88 (fix/regated-confidence-gate) fix(confidence): apply_confidence_gate left a stale ...
* c807d5d (fix/use-tf-guard)            fix: keep transformers from importing TensorFlow ...
* 2e4d9c8 (tag: v0.3.26)                chore: bump version to 0.3.26

so the diff opened against 0.3.26 carried all four files — laya/__init__.py and
tests/test_lazy_import.py from the TF guard, plus laya/confidence.py and
tests/test_confidence.py from the gate fix. Four files, two unrelated concerns, both of which I
had described to you as separate PRs. That is exactly what AGENTS.md says not to do, and your
reason for splitting them is the right one: they close different issues and either could need
reverting without the other.

Worth recording how the mistake happened, since it is a process failure rather than a typo: I
created the branch with git checkout -b while HEAD was still on the TF branch, so the guard
commit came along as the base. I then checked my own work with git diff --name-only merge-base origin/main <branch>, which showed only the two confidence files — because by that
point #924 had merged and the merge base had moved past the TF commit. The check confirmed what
I wanted to see instead of what a reviewer would see. Measuring against the branch I opened on,
not against whatever main is now, is what I should have done, and I have re-run the audit that
way for the follow-up.

Two things from your note, taken up:

The confidence fix — nothing to follow up. I ran my PR's eleven re-gate checks against 0.3.27
to see whether #911 leaves anything uncovered, and all of them pass, including 0.0 re-gates and
the per-bucket map case. I also tried the cases my version did not test — a re-gated answer with no
usable confidence, and flag_low_confidence called directly on a reused payload — and those pass
too. #911 clears inside flag_low_confidence where mine cleared in the gate; that is the broader
scope, and it covers the ground. So I am not opening a follow-up for want of a real gap.

The __init__.py surface — already landed. tests/test_lazy_import.py on main has the five
USE_TF checks, and laya.os is not resolvable (hasattr(laya, "os") is False), so the lazy
namespace stayed closed.

The thing I did take up is #935, the same per-call hook gap on the laya-ts mirror, as its own
PR: #943. One concern, one branch off current main, and the branch is verified to touch only
laya-ts/src/router.ts and laya-ts/tests/router-batch.test.ts against the base it was opened
on — the check I should have run here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants