Repository navigation
feat(evals): make laya-evals run --min-confidence reach the abstention gate - #777
Merged
NandhaKishorM merged 1 commit intoOct 1, 2026
Merged
Conversation
…n gate Router and ONNXAgent gate answers on `answer_confidence` when a caller passes `min_confidence` (NandhaKishorM#361). Every other wrapper in the repo -- the CLI, the demo server, MCP -- now forwards that gate; the evals harness was the last surface still able to parse it and drop it. A `precision@coverage` sweep had to wrap `Router` by hand to produce one number. `evaluate(min_confidence=T)` validates through core's `check_min_confidence`, so the accepted range cannot drift from what the gate itself enforces, then forwards T to whichever entry points this run can actually reach: `predict_batch` for a chunk, `predict` for a chunk of one. Unlike `sort_by_length`, a run whose entry point predates the gate is refused with a named `EvalError` rather than silently scored without it -- a report that said "policy at T=0.7" over a run that never applied a threshold is the exact class of lie the artifact exists to prevent. `config.timing` keeps the two claims apart: `min_confidence` records the ask, `min_confidence_sent` says whether any call this run made actually carried it. The CLI adds `run --min-confidence T` with the same validator, and a mistyped threshold fails as a usage error (exit 2) before any checkpoint loads. Eight mutants -- drop the refusal, drop either forwarding path, treat 0.0 as unset, drop the report fields, always-forward from the CLI, or conflate `sort_by_length_sent` with the new shape key -- each die on a named test.
4 tasks done
Owner
|
Merged for 0.3.23. Verified on the integrated branch, not on the PR in isolation: every suite in the Thank you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya-evals run --min-confidence Tnow forwards core's opt-in abstention threshold (#361) toevery call the harness makes, so a
precision@coveragesweep is a series oflaya-evals runcommands rather than a hand-wrapped
Router. A run whose entry point cannot accept the gate isrefused with a named
EvalErrorbefore any checkpoint loads, and the report'sconfig.timingrecords both the ask and whether the threshold actually reached a call.Why
Router.predict,Router.predict_batchandONNXAgent.predictall gate onanswer_confidencewhen a caller passesmin_confidence(#361). Every other wrapper in thisrepo -- the CLI (#91),
laya-serve(#90/#92/#95), the MCP server (#92/#93/#94) -- was widenedto forward that gate; the evals harness was the last surface still able to parse it and drop
it. That left a scoring control the shipped checkpoints document (
docs/calibration.md) and areviewer cannot reproduce with a single command, and any
precision@coveragefigure inresearch/was unreachable fromlaya-evals runwithout writing a custom runner.Unlike
sort_by_length, this changes what scores, so the harness cannot treat it as anoptimisation. A run that asked for
T=0.7and silently landed on a gate-blind runner wouldpublish a report claiming the policy was applied when it never was -- the class of artifact
this module's identity work (
dataset_sha256,questions_sha256,comparable_to) exists toprevent. So the harness refuses at the guard, not after.
How it works
evaluate(..., min_confidence=T)callslaya.confidence.check_min_confidence(the samevalidator the Router uses), so the accepted range --
[0.0, 1.0], finite, not a bool -- cannotdrift from what the gate itself enforces. It then uses
_takes_min_confidence, the sibling of_takes_sort_by_length, to prove every entry point this run can reach accepts the kwarg:predict_batchfor a chunk of >1,predictfor a chunk of one. A run that cannot honour theask raises with the target entry point named.
config.timinggetsmin_confidence(the ask) andmin_confidence_sent(whether any call thisrun made carried it), keeping the two claims apart exactly the way
sort_by_lengthandsort_by_length_sentalready do.The batch-shape
min_confidencerides in the same placesort_by_lengthdoes, so apass-through wrapper (
predict_batch(*args, **kwargs)) forwards both.sort_by_length_sentwas reading
bool(shape)-- which becomes true on a run that asked only for the gate -- andnow keys on
shape["sort_by_length"]alone. The two controls are independent, andtest_min_confidence_and_sort_by_length_are_independent_controlspins it both ways.The CLI adds
run --min-confidence Tand forwards it throughevals.evaluate. Out-of-range isan
EvalError, whichmainalready turns into exit 2 with a printed message, so a mistyped--min-confidence 1.5is a usage error not a traceback, and never loads a checkpoint.How it was verified
Mutation check
Every mutant is a single, plausible break to a real implementation choice. The clean tree is
re-verified before and after, and
__pycache__is cleared between each mutant so a stale.pyccannot mask a survivor.EvalErrorguardtest_a_runner_without_the_gate_is_refused_not_silently_scoredpredict_batchforwardtest_min_confidence_reaches_a_requests_shaped_runner_when_askedpredictforwardtest_min_confidence_reaches_a_states_shaped_runner_tooif min_confidence:(drops 0.0)test_min_confidence_zero_is_still_a_thresholdmin_confidence/_senttiming keystest_min_confidence_reaches_a_requests_shaped_runner_when_askedRouterRunner.predict(Nonebecomes an override)test_cli_run_records_the_run_identity(TypeError)test_cli_min_confidence_forwards_through_evaluatesort_by_length_sent: bool(shape)with the new shape keytest_min_confidence_and_sort_by_length_are_independent_controlsall mutants killed by a NAMED test failure, tree restored clean.What this changes about existing behaviour
evaluate()'s parameter list widened withmin_confidenceat the tail. Same rule assort_by_length: the knob sits afterbatch_sizeand never before, so a caller that passedon_errororconfigpositionally still means the same thing.tests/test_evals_api.pypins the exact order.
RouterRunner.predict/predict_batchandOnnxRunner.predict/predict_batchacceptthe new kwarg. All default to
None, and the conditional forward keeps aNone-- "inherit"in core's terms -- from reaching the underlying call: an unconditional forward would let a
run that did not ask override the deployment's
Router(min_confidence=...).sort_by_length_sentreadsbool(shape.get("sort_by_length"))rather thanbool(shape), soa run that asked for
--min-confidenceand nothing else keeps reporting the grouping asunsent.
test_grouping_without_a_batch_to_reorder_is_recorded_as_not_sentand the newindependence test both cover this.
FakeRouterintest_evals.pygrows amin_confidence=Noneparameter on itspredict, soa
--min-confidenceCLI test proves the flag reached the Router without needing weights.Design notes
sort_by_lengthis an optimisation -- dropping it makesthe run slower, not wrong -- so a pre-feat(agent): group batch inputs by encoded length #294 runner is scored unsorted. An abstention threshold
changes which answers count, so the same "quietly ignore the ask" policy would turn the
report into a claim it cannot support. Both branches are in
evaluate's docstring.still goes through
predict, so a batched run needs both to accept the kwarg. The refusalmessage names which entry point failed, so an operator sees whether their wrapper forwards
the gate on the batch path or on the single path.
evaluate, not the CLI: the CLI's argparsetypeis
float, so it will happily parse1.5and hand a bad number down. Callingcheck_min_confidenceat the top ofevaluate-- before the guard, before any runner call,before any checkpoint load -- makes the harness the enforcement point. The CLI just turns
EvalErrorinto exit 2 the way it already does for every other usage error.min_confidence=0.0has its own test: it is the control arm for aprecision@coveragesweep -- "score every answer, no gate" -- andif min_confidence:wouldmake the sweep's floor indistinguishable from "the caller did not ask".
check_min_confidenceaccepts 0.0 explicitly.
Not in this PR
precision@coverageas a metric. The gate flags low-confidence answers withlow_confidence: True; a metric that reads that flag is a separate evaluator and a separatedesign decision about the denominator (all rows, or only rows whose answer reported a usable
confidence --
answer_confidence_valuealready returnsNoneforboolandNaN). Thewiring is here; the metric is not.
--min-confidencesweep subcommand. Given the refusal, a sweep is a shell loop overlaya-evals run --min-confidence T; a first-class sweep would be a second naming surface forthe same thing.
min_confidenceintolaya.compareor the report identity. Two runs at differentthresholds are already non-comparable via
dataset_sha256alone -- same data, differentpolicy -- so the identity block should not silently pretend otherwise. A follow-up can name
the threshold in
comparable_to's refusal if a reviewer wants that specifically.Checklist
main(upstream/main @ 6d942c9)docs/evals.md: new "The abstention gate at a threshold" section;test_docs_and_the_cli_name_the_same_flagsextended)test_evals_api.py)