Skip to content

feat(evals): make laya-evals run --min-confidence reach the abstention gate - #777

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-min-confidence
Oct 1, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-min-confidence

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya-evals run --min-confidence T now forwards core's opt-in abstention threshold (#361) to
every call the harness makes, so a precision@coverage sweep is a series of laya-evals run
commands rather than a hand-wrapped Router. A run whose entry point cannot accept the gate is
refused with a named EvalError before any checkpoint loads, and the report's
config.timing records both the ask and whether the threshold actually reached a call.

Why

Router.predict, Router.predict_batch and ONNXAgent.predict all gate on
answer_confidence when a caller passes min_confidence (#361). Every other wrapper in this
repo -- the CLI (#91), laya-serve (#90/#92/#95), the MCP server (#92/#93/#94) -- was widened
to forward that gate; the evals harness was the last surface still able to parse it and drop
it. That left a scoring control the shipped checkpoints document (docs/calibration.md) and a
reviewer cannot reproduce with a single command, and any precision@coverage figure in
research/ was unreachable from laya-evals run without writing a custom runner.

Unlike sort_by_length, this changes what scores, so the harness cannot treat it as an
optimisation. A run that asked for T=0.7 and silently landed on a gate-blind runner would
publish a report claiming the policy was applied when it never was -- the class of artifact
this module's identity work (dataset_sha256, questions_sha256, comparable_to) exists to
prevent. So the harness refuses at the guard, not after.

How it works

evaluate(..., min_confidence=T) calls laya.confidence.check_min_confidence (the same
validator the Router uses), so the accepted range -- [0.0, 1.0], finite, not a bool -- cannot
drift from what the gate itself enforces. It then uses _takes_min_confidence, the sibling of
_takes_sort_by_length, to prove every entry point this run can reach accepts the kwarg:
predict_batch for a chunk of >1, predict for a chunk of one. A run that cannot honour the
ask raises with the target entry point named.

config.timing gets min_confidence (the ask) and min_confidence_sent (whether any call this
run made carried it), keeping the two claims apart exactly the way sort_by_length and
sort_by_length_sent already do.

The batch-shape min_confidence rides in the same place sort_by_length does, so a
pass-through wrapper (predict_batch(*args, **kwargs)) forwards both. sort_by_length_sent
was reading bool(shape) -- which becomes true on a run that asked only for the gate -- and
now keys on shape["sort_by_length"] alone. The two controls are independent, and
test_min_confidence_and_sort_by_length_are_independent_controls pins it both ways.

The CLI adds run --min-confidence T and forwards it through evals.evaluate. Out-of-range is
an EvalError, which main already turns into exit 2 with a printed message, so a mistyped
--min-confidence 1.5 is a usage error not a traceback, and never loads a checkpoint.

How it was verified

$ python -m pytest tests/test_evals.py -q
96 passed in 0.84s

$ python tests/test_evals_api.py
35 passed, 0 failed
all eval API checks passed

$ python tests/test_evals_onnx.py
27 passed, 0 failed

$ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!

$ python -m compileall -q laya/ tests/

Mutation check

Every mutant is a single, plausible break to a real implementation choice. The clean tree is
re-verified before and after, and __pycache__ is cleared between each mutant so a stale
.pyc cannot mask a survivor.

mutant what it breaks named test that dies
M1 drop the pre-flight EvalError guard test_a_runner_without_the_gate_is_refused_not_silently_scored
M2 drop the batch predict_batch forward test_min_confidence_reaches_a_requests_shaped_runner_when_asked
M3 drop the single-predict forward test_min_confidence_reaches_a_states_shaped_runner_too
M4 use if min_confidence: (drops 0.0) test_min_confidence_zero_is_still_a_threshold
M5 drop the min_confidence / _sent timing keys test_min_confidence_reaches_a_requests_shaped_runner_when_asked
M6 always forward from RouterRunner.predict (None becomes an override) test_cli_run_records_the_run_identity (TypeError)
M7 CLI never forwards the threshold test_cli_min_confidence_forwards_through_evaluate
M8 conflate sort_by_length_sent: bool(shape) with the new shape key test_min_confidence_and_sort_by_length_are_independent_controls

all mutants killed by a NAMED test failure, tree restored clean.

What this changes about existing behaviour

  • evaluate()'s parameter list widened with min_confidence at the tail. Same rule as
    sort_by_length: the knob sits after batch_size and never before, so a caller that passed
    on_error or config positionally still means the same thing. tests/test_evals_api.py
    pins the exact order.
  • RouterRunner.predict / predict_batch and OnnxRunner.predict / predict_batch accept
    the new kwarg. All default to None, and the conditional forward keeps a None -- "inherit"
    in core's terms -- from reaching the underlying call: an unconditional forward would let a
    run that did not ask override the deployment's Router(min_confidence=...).
  • sort_by_length_sent reads bool(shape.get("sort_by_length")) rather than bool(shape), so
    a run that asked for --min-confidence and nothing else keeps reporting the grouping as
    unsent. test_grouping_without_a_batch_to_reorder_is_recorded_as_not_sent and the new
    independence test both cover this.
  • FakeRouter in test_evals.py grows a min_confidence=None parameter on its predict, so
    a --min-confidence CLI test proves the flag reached the Router without needing weights.

Design notes

  • Why refuse rather than degrade: sort_by_length is an optimisation -- dropping it makes
    the run slower, not wrong -- so a pre-feat(agent): group batch inputs by encoded length #294 runner is scored unsorted. An abstention threshold
    changes which answers count, so the same "quietly ignore the ask" policy would turn the
    report into a claim it cannot support. Both branches are in evaluate's docstring.
  • Why the check is per-entry-point, not "does the runner have the gate": a chunk of one
    still goes through predict, so a batched run needs both to accept the kwarg. The refusal
    message names which entry point failed, so an operator sees whether their wrapper forwards
    the gate on the batch path or on the single path.
  • Why core's validator is called from evaluate, not the CLI: the CLI's argparse type
    is float, so it will happily parse 1.5 and hand a bad number down. Calling
    check_min_confidence at the top of evaluate -- before the guard, before any runner call,
    before any checkpoint load -- makes the harness the enforcement point. The CLI just turns
    EvalError into exit 2 the way it already does for every other usage error.
  • Why min_confidence=0.0 has its own test: it is the control arm for a
    precision@coverage sweep -- "score every answer, no gate" -- and if min_confidence: would
    make the sweep's floor indistinguishable from "the caller did not ask". check_min_confidence
    accepts 0.0 explicitly.

Not in this PR

  • Reporting precision@coverage as a metric. The gate flags low-confidence answers with
    low_confidence: True; a metric that reads that flag is a separate evaluator and a separate
    design decision about the denominator (all rows, or only rows whose answer reported a usable
    confidence -- answer_confidence_value already returns None for bool and NaN). The
    wiring is here; the metric is not.
  • A --min-confidence sweep subcommand. Given the refusal, a sweep is a shell loop over
    laya-evals run --min-confidence T; a first-class sweep would be a second naming surface for
    the same thing.
  • Threading min_confidence into laya.compare or the report identity. Two runs at different
    thresholds are already non-comparable via dataset_sha256 alone -- same data, different
    policy -- so the identity block should not silently pretend otherwise. A follow-up can name
    the threshold in comparable_to's refusal if a reviewer wants that specifically.

Checklist

  • Focused on one change (split unrelated work into separate PRs)
  • Rebased on the latest main (upstream/main @ 6d942c9)
  • Tests pass locally (96 pytest / 35 API / 27 ONNX)
  • Ruff passes with the CI selection
  • Byte-compile succeeds
  • Mutation-check kills every mutant with a named test failure
  • Docs updated when the public API changed (docs/evals.md: new "The abstention gate at a threshold" section; test_docs_and_the_cli_name_the_same_flags extended)
  • No new hard dependency; imports stay torch-free (verified by test_evals_api.py)

…n gate

Router and ONNXAgent gate answers on `answer_confidence` when a caller passes
`min_confidence` (NandhaKishorM#361). Every other wrapper in the repo -- the CLI, the demo
server, MCP -- now forwards that gate; the evals harness was the last surface
still able to parse it and drop it. A `precision@coverage` sweep had to wrap
`Router` by hand to produce one number.

`evaluate(min_confidence=T)` validates through core's `check_min_confidence`,
so the accepted range cannot drift from what the gate itself enforces, then
forwards T to whichever entry points this run can actually reach:
`predict_batch` for a chunk, `predict` for a chunk of one. Unlike
`sort_by_length`, a run whose entry point predates the gate is refused with a
named `EvalError` rather than silently scored without it -- a report that said
"policy at T=0.7" over a run that never applied a threshold is the exact class
of lie the artifact exists to prevent. `config.timing` keeps the two claims
apart: `min_confidence` records the ask, `min_confidence_sent` says whether
any call this run made actually carried it.

The CLI adds `run --min-confidence T` with the same validator, and a
mistyped threshold fails as a usage error (exit 2) before any checkpoint
loads. Eight mutants -- drop the refusal, drop either forwarding path, treat
0.0 as unset, drop the report fields, always-forward from the CLI, or
conflate `sort_by_length_sent` with the new shape key -- each die on a named
test.
@NandhaKishorM
NandhaKishorM merged commit 3dbefb3 into NandhaKishorM:main Oct 1, 2026
24 of 25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.23.

Verified on the integrated branch, not on the PR in isolation: every suite in the ci.yml test lane plus the pytest lane green, compileall clean, and a decision diff against main on the real english and multilingual checkpoints (72 answers over choice, score and noul, short and long states, budgeted and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants