Skip to content

fix(evals): report the checkpoint that answered, not the payload's family tag - #571

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-model-slice
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-model-slice

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

evaluate() labels each case with example.model or result.get("model"). A Router never writes a top-level model key — the checkpoint it picked goes into result["routing"]["model"] — and result["model"] on any Laya payload is the family tag "laya-rl-agent", the same string for all three checkpoints. So a routed run's by model slice is one bucket holding a copy of the overall numbers.

The checkpoint that answered is already in the result, one level down. This reads it.

Measured, on the real checkpoint (CPU, 8 MASSIVE rows: 4 en, 4 de, one 6-option choice question)

report.slices["model"] before:

model choice_accuracy mean_confidence ece rows
laya-rl-agent 0.8750 0.9306 0.0705 8

after:

model choice_accuracy mean_confidence ece rows
english 0.8333 0.9076 0.0938 6
multilingual 1.0000 0.9995 0.0005 2

Routing split the four German rows 2/2 between the two checkpoints, which is the difference the slice exists to show. Nothing else moved: the two reports are identical field-for-field apart from the label and the latency numbers (89 vs 109 ms p50, run-to-run noise on the same eight rows) — per-case expected, answer, correct and confidence all compare equal.

The only way to get a per-checkpoint breakdown today is to set model on every row, and that pins the checkpoint, so routing no longer happens and the run measures something other than what a mixed-language dataset is there to measure.

Change

_answered_model(result) returns routing["model"] when routing is a dict carrying a non-empty string, and otherwise falls back to the top-level model it used before. A row that forces a checkpoint keeps its own label (example.model or _answered_model(result)), so a pin still outranks a route, and a runner that never routes (a bare Agent, an ONNXAgent) is labelled exactly as it was.

Tests

tests/test_evals.py: 18 → 20. The new checks cover the routed checkpoint on both the single-call and the batch_size=8 path, a caller-pinned row that the stub routes somewhere else, and seven routing shapes (absent, empty, None, "", a number, a bare string, a list) that must fall back rather than crash. Without the emptiness check a routing with no usable name yields None, and _group_cases skips None values, so the model dimension silently disappears from the report instead of falling back — one of the mutations below reproduces exactly that.

Four mutations, each breaking a named check:

mutation broken check
call site back to (result or {}).get("model") test_model_slice_follows_the_routed_checkpoint
route outranks the caller's pin same test, third row
drop the non-empty-string check test_model_slice_falls_back_when_a_runner_reports_no_route
drop the isinstance(routing, dict) check same test

Gates: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 and python -m compileall -q laya/ tests/ clean; tests/test_evals.py 20 passed, tests/test_evals_api.py 12 passed, tests/test_router.py 546 passed. docs/evals.md gains a line in the dataset-format table and one in the Slices paragraph; zensical build --strict --clean reports no issues.

Overlap

Probe and both reports: probe_evals_model_slice.py, evals_before.log / evals_after.log, evals_before.json / evals_after.json (kept locally, not committed).

evaluate() recorded `example.model or result.get("model")`, but a Router leaves no checkpoint
name at the top level of its result -- its choice is under `routing` -- and `result["model"]`
is the payload's family tag `laya-rl-agent` on every Laya runner. A mixed-language run through
a Router therefore sliced into one bucket that repeated the overall numbers. Read `routing`
first, fall back to the old key, and keep a row's own pinned checkpoint ahead of both.
@rahul05ranjan

Copy link
Copy Markdown
Contributor

Thanks for the PR. Fixing the by model slice to report the checkpoint that actually answered (via routing["model"]) rather than the payload's family tag is a real correctness fix — the old code would have lumped every checkpoint under laya-rl-agent.

A few points:

  1. _answered_model fallback: when routing is absent or routing["model"] is empty, it falls back to result.get("model") — which is the family tag. For a non-routing runner that's correct (the runner's model is the checkpoint), but for a routing runner whose routing block is somehow missing, this silently reports the family tag again. Worth distinguishing "runner doesn't route" from "routing block missing" if possible.

  2. example.model or _answered_model(result): the or means an empty-string example.model would fall through to _answered_model. Empty string is unlikely to be a valid model name, but the or (vs is not None) is a subtle truthiness trap. Consider example.model if example.model is not None else _answered_model(result).

  3. Doc consistency: the docs/evals.md change says "the model slice holds the checkpoint that answered each row" — good, matches the code. The _answered_model docstring is thorough and explains the routing vs top-level distinction well.

Otherwise a clean, well-documented fix.

@NandhaKishorM
NandhaKishorM merged commit 4ae949b into NandhaKishorM:main Sep 27, 2026
22 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants