Skip to content

fix(evals): fail the baseline comparison when a metric is missing - #510

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
can-gaa-hou:fix/evals-compare-missing-metric
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
can-gaa-hou:fix/evals-compare-missing-metric

Conversation

@can-gaa-hou

Copy link
Copy Markdown
Contributor

What

EvalReport.compare now fails when a baseline metric is missing from the report, instead of skipping it. The missing metric is marked "missing": True in the deltas. The CLI and check_regression.py print "missing from the report" for it. Latency metrics are still skipped unless a tolerance names them.

Why

If every example errors (--on-error skip), the report has no metrics, so compare has nothing to check and passes. The scheduled evals.yml gate has the same problem: when laya_eval.py fails, it writes {"error": ...} and exits 0, and check_regression.py then passes. A completely failed evaluation shows green.

How it was verified

python -m pytest tests/test_evals.py -q   # 20 passed (2 new tests, both fail on main)
python tests/test_evals_api.py            # 12 passed
python tests/test_cli.py                  # 38 passed
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally

EvalReport.compare skipped any baseline metric absent from the fresh
report. A run where every example errors under on_error="skip" has an
empty `overall`, so compare returned (True, {}) and assert_regression,
`laya-evals compare` and `laya-evals run --baseline` all passed the gate
with nothing compared.

A missing baseline metric now fails the comparison and is reported in
the deltas with `missing: True`. Latency metrics stay informational
unless a tolerance names them. The CLI and research/evals/check_regression.py
print "missing from the report" for such metrics instead of nan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aashish254

Copy link
Copy Markdown
Contributor

Worth knowing where I hit the empty-overall case you are guarding: laya.evals.evaluate gates batching on hasattr(runner, "predict_batch") and then calls it as predict_batch(states, questions, model=...), which is not the shape Router.predict_batch(requests, ...) has — so with on_error="skip" every chunk of more than one row errors and the report comes back with zero cases. On the english checkpoint, 8 rows, batch_size=4: cases=0, errored=8, overall empty, and compare against a good 5-metric baseline returned ok with 0 metrics compared.

I put up #584 to drive the request-dict form so that particular cause goes away. It does not overlap your diff (evaluate versus compare/assert_regression), and the check here is still what catches every other way a run comes back empty, so nothing about this changes what this PR should do.

@aashish254

Copy link
Copy Markdown
Contributor

One interaction worth knowing about from the evals side: #593 adds laya-evals run --score-within TOL, which puts the tolerances a run was asked for into report.config["score_within"]. With your missing-metric rule in, a baseline report that has a score_within_0.25 column and a CI run that forgot the flag stops being a silent skip and fails the comparison, which is the behaviour I'd want but did not implement here. So if you need a way to tell which columns a baseline expects, that config key is now in the artifact rather than only in the metrics block.

@NandhaKishorM
NandhaKishorM merged commit 90ca791 into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @can-gaa-hou, merged. An empty overall passing the gate was a real hole. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants