Repository navigation
fix(evals): report the checkpoint that answered, not the payload's family tag - #571
Conversation
evaluate() recorded `example.model or result.get("model")`, but a Router leaves no checkpoint
name at the top level of its result -- its choice is under `routing` -- and `result["model"]`
is the payload's family tag `laya-rl-agent` on every Laya runner. A mixed-language run through
a Router therefore sliced into one bucket that repeated the overall numbers. Read `routing`
first, fall back to the old key, and keep a row's own pinned checkpoint ahead of both.
|
Thanks for the PR. Fixing the A few points:
Otherwise a clean, well-documented fix. |
|
Thanks @aashish254, merged. It ships in 0.3.21. |
What
evaluate()labels each case withexample.model or result.get("model"). ARouternever writes a top-levelmodelkey — the checkpoint it picked goes intoresult["routing"]["model"]— andresult["model"]on any Laya payload is the family tag"laya-rl-agent", the same string for all three checkpoints. So a routed run'sby modelslice is one bucket holding a copy of the overall numbers.The checkpoint that answered is already in the result, one level down. This reads it.
Measured, on the real checkpoint (CPU, 8 MASSIVE rows: 4
en, 4de, one 6-option choice question)report.slices["model"]before:laya-rl-agentafter:
englishmultilingualRouting split the four German rows 2/2 between the two checkpoints, which is the difference the slice exists to show. Nothing else moved: the two reports are identical field-for-field apart from the label and the latency numbers (89 vs 109 ms p50, run-to-run noise on the same eight rows) — per-case
expected,answer,correctandconfidenceall compare equal.The only way to get a per-checkpoint breakdown today is to set
modelon every row, and that pins the checkpoint, so routing no longer happens and the run measures something other than what a mixed-language dataset is there to measure.Change
_answered_model(result)returnsrouting["model"]whenroutingis a dict carrying a non-empty string, and otherwise falls back to the top-levelmodelit used before. A row that forces a checkpoint keeps its own label (example.model or _answered_model(result)), so a pin still outranks a route, and a runner that never routes (a bareAgent, anONNXAgent) is labelled exactly as it was.Tests
tests/test_evals.py: 18 → 20. The new checks cover the routed checkpoint on both the single-call and thebatch_size=8path, a caller-pinned row that the stub routes somewhere else, and sevenroutingshapes (absent, empty,None,"", a number, a bare string, a list) that must fall back rather than crash. Without the emptiness check aroutingwith no usable name yieldsNone, and_group_casesskipsNonevalues, so themodeldimension silently disappears from the report instead of falling back — one of the mutations below reproduces exactly that.Four mutations, each breaking a named check:
(result or {}).get("model")test_model_slice_follows_the_routed_checkpointtest_model_slice_falls_back_when_a_runner_reports_no_routeisinstance(routing, dict)checkGates:
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120andpython -m compileall -q laya/ tests/clean;tests/test_evals.py20 passed,tests/test_evals_api.py12 passed,tests/test_router.py546 passed.docs/evals.mdgains a line in the dataset-format table and one in the Slices paragraph;zensical build --strict --cleanreports no issues.Overlap
compare/assert_regressioninlaya/evals.pyand further-down tests intests/test_evals.py— different functions, no shared lines, and it never reads themodelslice value.laya/evals_cli.pyanddocs/evals.mdabove this table. The CLI prints whateverreport.slicesholds, so it picks this up unchanged; the two doc edits share no lines, so whichever merges second rebases without a content conflict.Probe and both reports:
probe_evals_model_slice.py,evals_before.log/evals_after.log,evals_before.json/evals_after.json(kept locally, not committed).