Repository navigation
test(benchmarks): check the published headline cells against the T4 run - #1096
Conversation
The English-vs-the-rest figures are the project's most-read numbers and the only benchmark tables that named neither an artifact nor a protocol. NandhaKishorM#776 re-ran them with an independent harness; the transcription still had no gate. Every printed cell in all three tables is now recomputed from the run's per-language suites and compared, and the artifact's own english_vs_rest summary is compared against the same recompute so an aggregation change cannot pass silently.
|
Merged in 0.4.3. Checking the three most-read benchmark tables against the artifact that produced them is the right place to spend 100 lines of test, because these are the cells someone quotes when deciding whether to adopt the project, and they were published bare. Recomputing the accuracy from the per-language suites rather than reading the artifact's own summary is what makes the check real, and then checking that summary against the recomputation too closes the loop. Using the file's existing half-up Naming the artifact and the protocol in BENCHMARKS.md and README, seed 13, 20 options per choice question, first 300 test rows per language, is the half of this I would have asked for if it were missing. A number without its protocol cannot be reproduced or disputed, and this project's position is that its benchmarks are reproducible. Verified on the merged tree: |
What
tests/test_doc_tables.pynow checks the project's most-read benchmark figures — the "English vs the rest" accuracy cells published in three places (README's "Why Route: The Evidence", README's 51-language section, and BENCHMARKS.md's "English vs the rest") — against the run that produced them,research/results/t4_colab_benchmark.json. Checks go from 114 to 181. BENCHMARKS.md and README.md now also name the artifact and the protocol (seed 13, 20 options per choice question, first 300 test rows per language) instead of printing the numbers bare.For each of the 14 figure rows × 2 checkpoint columns (28 printed cells) the cell is compared to the accuracy recomputed from the artifact's per-language suites, formatted with the file's existing
rounded()(half-up, the convention the table already uses). The artifact's own*_english_vs_restsummary is checked against the same recompute, so the two must agree before any printed cell is judged.It also pins the things the prose leans on but never stated: every one of the 43 language suites carries exactly
per_langquestions (which is what makes "average over languages" and "pool the questions" the same number), the suite and question counts in the summary equal the suites on disk, README's17,416 questionsequals the sum over the artifact's 49 suites for both checkpoints, and "13 other languages" / "14 other languages" equal the non-enlanguage counts derived from the suite names.Why
#776 independently re-ran the four README figures and reported all of them within 0.0003 — which is this table's own rounding to three places. Good news, and it is the second opinion on the run. It is not a gate: nothing here noticed if a future refresh of the artifact, or a hand-edit of the transcription, moved a cell. These were the only published benchmark tables in the repo that named neither an artifact nor a protocol, which is exactly the shape #298 was filed about — "the run is not stale, the transcription is". Re-measuring cannot catch that; comparing the printed cell to the committed JSON can, and it needs no weights, no GPU and no network, so it runs in the model-free lane.
How it was verified
Ruff on this file goes from 2 findings to 1: the section puts the file's previously unused
import reto work. The remaining one (Optional) predates this change and CI lintslaya/only, so it is left alone rather than drive-bys fixed.The gate was then mutation-checked — 10 mutants, one at a time, file reset between each, baseline re-run after:
0.783→0.7840.657→0.6560.439→0.44017,416 questions→17,44013 other languages→120.8600)meta.seed13 → 14Zero survivors.
cells_checkedis asserted to equal 28, so an empty or partial table parse cannot pass vacuously.Not asserted, and stated as such in the code: README's
Router(Routed) column. No routed run exists in the artifact, so a check on it would be arithmetic on the other two columns rather than evidence. Adding it to the file'sunbackedlist would edit the same lines as #838's in-review hunk, so it is recorded as a comment here and will follow #838 rather than stack on it.Ref #776, #298.
Checklist
mainRef #776