Skip to content

feat(evals): pin the checkpoint commit a run scores, and record it in the report - #588

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-revision
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-evals-revision

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

docs/evals.md opens its reproducibility claim with "The harness is deterministic for a fixed
checkpoint revision", and then says what the report records: "the dataset, model and device".
A revision was never one of the things laya-evals run could fix or report. Router already
takes revision / revisions, and Router.loaded_revisions already answers "which commit is
resident" — /health publishes it (laya/serve.py:312) — but the evals CLI dropped both ends.

So a baseline JSON could not tell you whether a drift came from the weights or the code, which is
the failure mode #208 describes for committed numbers that no longer reproduce.

Change

  • laya/evals_cli.py: --revision on run, in the two forms Router takes — a bare SHA applies
    to every checkpoint the run loads, NAME=SHA pins one. Repeatable. The per-name form is the one
    an auto-routing run wants: the three checkpoints are three repositories, so no single commit
    exists in all of them.
  • After the run, report.config gains revisions: Router.loaded_revisions, the commit each
    checkpoint that answered came from. It can only be read after the run because preload=False
    means nothing is resident before the first row, and evaluate() copies the config it is given,
    so the write goes on the report.
  • A checkpoint name in a pin is checked by Router.__init__, not by a copy of the model list here,
    so aliases (en=…) work and a typo exits 1 with core's own message instead of a traceback.
  • docs/evals.md: the paragraph now states what the report carries and how to pin, and points at
    PINNED_REVISIONS.

Measured

laya-bench/probe_evals_revision_pin.py drives the real CLI as a subprocess with
PYTHONPATH=<tree>, against research/evals/fixture.jsonl on cpu, and reads the reviewed pin from
that same tree's laya/revisions.py. Before = 4066d5d (laya-bench/evals_revision_before_v2.log),
after = this branch (…_after_v2.log).

call before after
run fixture.jsonl --model english exit=0, config {"dataset", "device", "model"} exit=0, config gains "revisions": {"english": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"}
… --revision 55cf4c4e… exit=2 — unrecognized arguments: --revision 55cf4c4e… exit=0
… --revision english=55cf4c4e… + --json exit=2, (no report) exit=0, recorded commit == reviewed pin : True
auto-routed, --revision english=… --revision ml=… exit=2 exit=0, records {"english": "55cf4c4e…"} — only the checkpoint that answered
--revision englishg=… exit=2 (unknown flag) exit=1 — laya-evals: unknown model 'englishg'; choose one of ['english', 'multilingual', 'typed-decisions'] (or an alias: …
--revision english=0000… (well-formed, absent) exit=2 exit=1, the Hub refuses it

Two things this does not claim. Pinned and unpinned quality numbers are the same run
(pinned vs unpinned overall (no latency) : identical) because offline the default branch resolves
to the cached snapshot — that row shows the pin is plumbed to the loading path, not that pinning
changes a score; deciding that needs a commit that differs from the default branch. And the
unpinned config line above is not a guess: an unpinned run records the commit that answered, which
is the half that closes the documented gap.

Tests

tests/test_evals.py: 18 passed on 4066d5d, 34 here (both measured on the tree named in the log
line). The new 16 cover both
pin forms, the recorded commit for pinned and unpinned runs, the typo and alias cases, and
_parse_revisions' rejects (half a pair, two different bare commits). The Router stand-in
validates names with core's own normalise_name, so the typo test fails for the real reason and a
change to the alias table would break it rather than pass silently.

Gate check (laya-bench/mutate_evals_revision_gate.py, every variant restored — restored: True):

  0  as shipped                  exit=0  34 passed  failures: none
  m1 never record the commit     exit=1  32  forwards_the_pin_and_records… , records_an_unpinned_run_too
  m2 drop the NAME=SHA dict      exit=1  31  forwards_the_pin… , rejects_a_typo… , accepts_a_checkpoint_alias…
  m3 let core's ValueError escape exit=1 33  rejects_a_typo_in_a_pinned_checkpoint_name
  m4 guard names by exact match  exit=1  32  rejects_a_typo… , accepts_a_checkpoint_alias…
  m5 record the request, not the load exit=1 33 records_an_unpinned_run_too
  m6 allow two bare commits      exit=1  33  parse_revisions_rejects_two_different_bare_commits
  m7 allow a half pair           exit=1  32  parse_revisions_rejects_half_a_pair (both params)

m4 is the one worth reading: a local exact-match guard on DEFAULT_MODELS looks reasonable and
turns en=<SHA> into a caller error, so the alias test is the check that keeps the validation in
core. m5 is the check that the reported commit is what loaded, not what was asked for.

Other gates, all on this tree: ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
→ All checks passed!; python -m compileall -q laya/ tests/; tests/test_evals_api.py → 12
passed; tests/test_cli.py → 38 passed; tests/test_packaging.py → 43 passed; zensical build --strict --clean → No issues found, and --revision / revisions both appear in the built
site/evals/index.html.

Overlap

laya/evals_cli.py has two other open PRs on it. #510 (@can-gaa-hou) changes _check_thresholds
and compare, a different part of the file. My own #506 (--onnx) rewrites the same Router
construction line in _cmd_run and the same docs/evals.md paragraph, so whichever of those two
lands second needs a one-line rebase; I would rather rebase mine than renumber this.

tests/test_evals.py is the real collision point across my stack: #584 and #571 (both mine) also
append tests at its tail, and #510 edits it too. Those are textual conflicts at the end of the file,
not competing designs, and I will resolve them in series.

#528 (@keemsisi) documents revision pinning for Router in docs/security.md; this is the evals-CLI
end of the same feature, with no shared file. Related to #571, which makes the report name the
checkpoint that answered — this makes it name the commit.

docs/evals.md promises a report is reproducible "for a fixed checkpoint
revision", but `laya-evals run` had no way to fix one and its artifact
named only the dataset, model and device -- so a baseline drift could not
be attributed to the weights or to the code.

`run` now takes `--revision <SHA>` for every checkpoint it loads and
`--revision NAME=SHA` for one (repeatable, since the three checkpoints are
three repositories), and records `Router.loaded_revisions` in the report's
config block after the run, so unpinned runs say which commit answered too.
A mistyped checkpoint name reaches `Router`'s own validation, including its
alias table, and is reported as a usage error instead of a traceback.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254, merged. Combined with #506's --onnx: there only a bare --revision SHA applies, and NAME=SHA is a usage error. It ships in 0.3.21.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants