Repository navigation
fix(evals): fail the baseline comparison when a metric is missing - #510
NandhaKishorM merged 1 commit into
Conversation
EvalReport.compare skipped any baseline metric absent from the fresh
report. A run where every example errors under on_error="skip" has an
empty `overall`, so compare returned (True, {}) and assert_regression,
`laya-evals compare` and `laya-evals run --baseline` all passed the gate
with nothing compared.
A missing baseline metric now fails the comparison and is reported in
the deltas with `missing: True`. Latency metrics stay informational
unless a tolerance names them. The CLI and research/evals/check_regression.py
print "missing from the report" for such metrics instead of nan.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Worth knowing where I hit the empty- I put up #584 to drive the request-dict form so that particular cause goes away. It does not overlap your diff ( |
|
One interaction worth knowing about from the evals side: #593 adds |
|
Thanks @can-gaa-hou, merged. An empty |
What
EvalReport.comparenow fails when a baseline metric is missing from the report, instead of skipping it. The missing metric is marked"missing": Truein the deltas. The CLI andcheck_regression.pyprint "missing from the report" for it. Latency metrics are still skipped unless a tolerance names them.Why
If every example errors (
--on-error skip), the report has no metrics, socomparehas nothing to check and passes. The scheduledevals.ymlgate has the same problem: whenlaya_eval.pyfails, it writes{"error": ...}and exits 0, andcheck_regression.pythen passes. A completely failed evaluation shows green.How it was verified
Checklist