Skip to content

feat(evals): selective-classification metrics (brier, aurc, selective accuracy) - #854

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
AlKor13:feat/selective-eval-metrics
Oct 2, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
AlKor13:feat/selective-eval-metrics

Conversation

@AlKor13

@AlKor13 AlKor13 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

What

laya-evals reports ECE but nothing about how well a confidence ranks right from wrong, or what a coverage/risk trade buys — which is the yardstick the abstention gate (#361/#456) and the per-bucket thresholds (#394) exist to move. This adds three selective-classification metrics, computed from the same (confidence, correct) pairs ECE already uses, pure NumPy / torch-free:

  • brier — Brier score of confidence as P(correct), mean((confidence - correct)**2). Lower is better.
  • aurc — area under the risk–coverage curve (mean selective risk over every coverage). Lower is better; it rewards a confidence that orders answers, which ECE (calibration) does not.
  • selective_accuracy@50 / @80 — accuracy over the most-confident 50% / 80% of answers: what abstaining on the least-confident tail actually buys.

Integration (no new plumbing)

They are produced in _aggregate next to ece, so they appear in EvalReport.overall and every slice, and compare / assert_regression gate on them with no change (both read overall generically). is_confidence_metric(name) marks the dataset-level (confidence, correct) metrics so _eval_policy._count counts them the way it already counts ece, instead of special-casing a single name — so --min-/--max- policy rules and min_count work for the new metrics too.

Why it matters

ECE says whether 0.8 means 80%; it says nothing about whether the model's most confident answers are its most correct ones. AURC and selective accuracy measure exactly that — the property an abstention gate relies on. With #394's per-bucket thresholds now landing, this is the measurement layer that shows whether a gate's coverage/risk trade is actually improving.

Tests / docs

  • tests/test_evals.py: metric math on known inputs (a perfect confidence ranking vs a scrambled one), and that all four reach a report's overall (113 passed).
  • tests/test_evals_api.py: the new public names are part of the torch-free API surface (44 passed).
  • docs/evals.md: metric table rows.

… accuracy)

The harness reports ECE (a calibration number) but nothing about how well the
confidence *ranks* right from wrong, or what a coverage/risk trade buys -- the
yardstick the abstention gate (NandhaKishorM#361/NandhaKishorM#456) and the per-bucket thresholds (NandhaKishorM#394)
are built to move. Add three, computed from the same (confidence, correct) pairs
ECE uses, pure NumPy / torch-free:

- `brier`: Brier score of confidence as P(correct), mean((conf - correct)**2).
- `aurc`: area under the risk-coverage curve (mean selective risk over coverage);
  rewards a confidence that orders answers, which ECE does not.
- `selective_accuracy@50` / `@80`: accuracy over the most-confident 50% / 80% --
  what abstaining on the least-confident tail buys.

They land in `EvalReport.overall` and every slice alongside `ece`, so `compare`
and `assert_regression` gate on them with no extra plumbing. `is_confidence_metric`
marks the dataset-level (confidence,correct) metrics so `_eval_policy._count`
counts them the way it already counts `ece`, instead of special-casing one name.

tests/test_evals.py: metric math on known inputs (perfect vs scrambled ranking)
and that all four reach a report's overall. tests/test_evals_api.py: the new
public names are part of the torch-free API surface. docs/evals.md: metric table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@NandhaKishorM
NandhaKishorM merged commit 2fa3571 into NandhaKishorM:main Oct 2, 2026
25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.24.

Verified on the integrated branch rather than on the PR alone: the full shared test list plus the pytest lane green, compileall clean, both TypeScript packages green (laya-ts 597 tests, the SDK 26), and a decision diff against 0.3.23 on the real english and multilingual checkpoints (74 answers over choice, score and noul; short and long states; budgeted calls and predict_long) showing zero decision flips and a max probability delta of 0.000000.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants