Repository navigation
feat(evals): pin the checkpoint commit a run scores, and record it in the report - #588
Merged
Merged
Conversation
docs/evals.md promises a report is reproducible "for a fixed checkpoint revision", but `laya-evals run` had no way to fix one and its artifact named only the dataset, model and device -- so a baseline drift could not be attributed to the weights or to the code. `run` now takes `--revision <SHA>` for every checkpoint it loads and `--revision NAME=SHA` for one (repeatable, since the three checkpoints are three repositories), and records `Router.loaded_revisions` in the report's config block after the run, so unpinned runs say which commit answered too. A mistyped checkpoint name reaches `Router`'s own validation, including its alias table, and is reported as a usage error instead of a traceback.
This was referenced Sep 26, 2026
Owner
|
Thanks @aashish254, merged. Combined with #506's |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
docs/evals.mdopens its reproducibility claim with "The harness is deterministic for a fixedcheckpoint revision", and then says what the report records: "the dataset, model and device".
A revision was never one of the things
laya-evals runcould fix or report.Routeralreadytakes
revision/revisions, andRouter.loaded_revisionsalready answers "which commit isresident" —
/healthpublishes it (laya/serve.py:312) — but the evals CLI dropped both ends.So a baseline JSON could not tell you whether a drift came from the weights or the code, which is
the failure mode #208 describes for committed numbers that no longer reproduce.
Change
laya/evals_cli.py:--revisiononrun, in the two formsRoutertakes — a bare SHA appliesto every checkpoint the run loads,
NAME=SHApins one. Repeatable. The per-name form is the onean auto-routing run wants: the three checkpoints are three repositories, so no single commit
exists in all of them.
report.configgainsrevisions:Router.loaded_revisions, the commit eachcheckpoint that answered came from. It can only be read after the run because
preload=Falsemeans nothing is resident before the first row, and
evaluate()copies theconfigit is given,so the write goes on the report.
Router.__init__, not by a copy of the model list here,so aliases (
en=…) work and a typo exits1with core's own message instead of a traceback.docs/evals.md: the paragraph now states what the report carries and how to pin, and points atPINNED_REVISIONS.Measured
laya-bench/probe_evals_revision_pin.pydrives the real CLI as a subprocess withPYTHONPATH=<tree>, againstresearch/evals/fixture.jsonlon cpu, and reads the reviewed pin fromthat same tree's
laya/revisions.py. Before =4066d5d(laya-bench/evals_revision_before_v2.log),after = this branch (
…_after_v2.log).run fixture.jsonl --model englishexit=0, config{"dataset", "device", "model"}exit=0, config gains"revisions": {"english": "55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851"}… --revision 55cf4c4e…exit=2—unrecognized arguments: --revision 55cf4c4e…exit=0… --revision english=55cf4c4e…+--jsonexit=2,(no report)exit=0,recorded commit == reviewed pin : True--revision english=… --revision ml=…exit=2exit=0, records{"english": "55cf4c4e…"}— only the checkpoint that answered--revision englishg=…exit=2(unknown flag)exit=1—laya-evals: unknown model 'englishg'; choose one of ['english', 'multilingual', 'typed-decisions'] (or an alias: …--revision english=0000…(well-formed, absent)exit=2exit=1, the Hub refuses itTwo things this does not claim. Pinned and unpinned quality numbers are the same run
(
pinned vs unpinned overall (no latency) : identical) because offline the default branch resolvesto the cached snapshot — that row shows the pin is plumbed to the loading path, not that pinning
changes a score; deciding that needs a commit that differs from the default branch. And the
unpinned config line above is not a guess: an unpinned run records the commit that answered, which
is the half that closes the documented gap.
Tests
tests/test_evals.py: 18 passed on4066d5d, 34 here (both measured on the tree named in the logline). The new 16 cover both
pin forms, the recorded commit for pinned and unpinned runs, the typo and alias cases, and
_parse_revisions' rejects (half a pair, two different bare commits). TheRouterstand-invalidates names with core's own
normalise_name, so the typo test fails for the real reason and achange to the alias table would break it rather than pass silently.
Gate check (
laya-bench/mutate_evals_revision_gate.py, every variant restored —restored: True):m4 is the one worth reading: a local exact-match guard on
DEFAULT_MODELSlooks reasonable andturns
en=<SHA>into a caller error, so the alias test is the check that keeps the validation incore. m5 is the check that the reported commit is what loaded, not what was asked for.
Other gates, all on this tree:
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120→
All checks passed!;python -m compileall -q laya/ tests/;tests/test_evals_api.py→ 12passed;
tests/test_cli.py→ 38 passed;tests/test_packaging.py→ 43 passed;zensical build --strict --clean→No issues found, and--revision/revisionsboth appear in the builtsite/evals/index.html.Overlap
laya/evals_cli.pyhas two other open PRs on it. #510 (@can-gaa-hou) changes_check_thresholdsand
compare, a different part of the file. My own #506 (--onnx) rewrites the sameRouterconstruction line in
_cmd_runand the samedocs/evals.mdparagraph, so whichever of those twolands second needs a one-line rebase; I would rather rebase mine than renumber this.
tests/test_evals.pyis the real collision point across my stack: #584 and #571 (both mine) alsoappend tests at its tail, and #510 edits it too. Those are textual conflicts at the end of the file,
not competing designs, and I will resolve them in series.
#528 (@keemsisi) documents revision pinning for
Routerindocs/security.md; this is the evals-CLIend of the same feature, with no shared file. Related to #571, which makes the report name the
checkpoint that answered — this makes it name the commit.