Repository navigation
feat(evals): selective-classification metrics (brier, aurc, selective accuracy) - #854
Merged
NandhaKishorM merged 1 commit intoOct 2, 2026
Conversation
… accuracy) The harness reports ECE (a calibration number) but nothing about how well the confidence *ranks* right from wrong, or what a coverage/risk trade buys -- the yardstick the abstention gate (NandhaKishorM#361/NandhaKishorM#456) and the per-bucket thresholds (NandhaKishorM#394) are built to move. Add three, computed from the same (confidence, correct) pairs ECE uses, pure NumPy / torch-free: - `brier`: Brier score of confidence as P(correct), mean((conf - correct)**2). - `aurc`: area under the risk-coverage curve (mean selective risk over coverage); rewards a confidence that orders answers, which ECE does not. - `selective_accuracy@50` / `@80`: accuracy over the most-confident 50% / 80% -- what abstaining on the least-confident tail buys. They land in `EvalReport.overall` and every slice alongside `ece`, so `compare` and `assert_regression` gate on them with no extra plumbing. `is_confidence_metric` marks the dataset-level (confidence,correct) metrics so `_eval_policy._count` counts them the way it already counts `ece`, instead of special-casing one name. tests/test_evals.py: metric math on known inputs (perfect vs scrambled ranking) and that all four reach a report's overall. tests/test_evals_api.py: the new public names are part of the torch-free API surface. docs/evals.md: metric table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
|
Merged for 0.3.24. Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, Thank you. |
1 task done
8 tasks done
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya-evalsreports ECE but nothing about how well a confidence ranks right from wrong, or what a coverage/risk trade buys — which is the yardstick the abstention gate (#361/#456) and the per-bucket thresholds (#394) exist to move. This adds three selective-classification metrics, computed from the same(confidence, correct)pairs ECE already uses, pure NumPy / torch-free:brier— Brier score of confidence as P(correct),mean((confidence - correct)**2). Lower is better.aurc— area under the risk–coverage curve (mean selective risk over every coverage). Lower is better; it rewards a confidence that orders answers, which ECE (calibration) does not.selective_accuracy@50/@80— accuracy over the most-confident 50% / 80% of answers: what abstaining on the least-confident tail actually buys.Integration (no new plumbing)
They are produced in
_aggregatenext toece, so they appear inEvalReport.overalland every slice, andcompare/assert_regressiongate on them with no change (both readoverallgenerically).is_confidence_metric(name)marks the dataset-level(confidence, correct)metrics so_eval_policy._countcounts them the way it already countsece, instead of special-casing a single name — so--min-/--max-policy rules andmin_countwork for the new metrics too.Why it matters
ECE says whether
0.8means 80%; it says nothing about whether the model's most confident answers are its most correct ones. AURC and selective accuracy measure exactly that — the property an abstention gate relies on. With #394's per-bucket thresholds now landing, this is the measurement layer that shows whether a gate's coverage/risk trade is actually improving.Tests / docs
tests/test_evals.py: metric math on known inputs (a perfect confidence ranking vs a scrambled one), and that all four reach a report'soverall(113 passed).tests/test_evals_api.py: the new public names are part of the torch-free API surface (44 passed).docs/evals.md: metric table rows.