Skip to content

test(benchmarks): check the published headline cells against the T4 run - #1096

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:test-headline-figure-provenance
Oct 11, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:test-headline-figure-provenance

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

tests/test_doc_tables.py now checks the project's most-read benchmark figures — the "English vs the rest" accuracy cells published in three places (README's "Why Route: The Evidence", README's 51-language section, and BENCHMARKS.md's "English vs the rest") — against the run that produced them, research/results/t4_colab_benchmark.json. Checks go from 114 to 181. BENCHMARKS.md and README.md now also name the artifact and the protocol (seed 13, 20 options per choice question, first 300 test rows per language) instead of printing the numbers bare.

For each of the 14 figure rows × 2 checkpoint columns (28 printed cells) the cell is compared to the accuracy recomputed from the artifact's per-language suites, formatted with the file's existing rounded() (half-up, the convention the table already uses). The artifact's own *_english_vs_rest summary is checked against the same recompute, so the two must agree before any printed cell is judged.

It also pins the things the prose leans on but never stated: every one of the 43 language suites carries exactly per_lang questions (which is what makes "average over languages" and "pool the questions" the same number), the suite and question counts in the summary equal the suites on disk, README's 17,416 questions equals the sum over the artifact's 49 suites for both checkpoints, and "13 other languages" / "14 other languages" equal the non-en language counts derived from the suite names.

Why

#776 independently re-ran the four README figures and reported all of them within 0.0003 — which is this table's own rounding to three places. Good news, and it is the second opinion on the run. It is not a gate: nothing here noticed if a future refresh of the artifact, or a hand-edit of the transcription, moved a cell. These were the only published benchmark tables in the repo that named neither an artifact nor a protocol, which is exactly the shape #298 was filed about — "the run is not stale, the transcription is". Re-measuring cannot catch that; comparing the printed cell to the committed JSON can, and it needs no weights, no GPU and no network, so it runs in the model-free lane.

How it was verified

python -B tests/test_doc_tables.py                      # 181 passed, 0 failed
python -B <upstream/main copy of the same file>         # 114 passed, 0 failed on the same tree
ruff check tests/test_doc_tables.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/                    # exit 0

Ruff on this file goes from 2 findings to 1: the section puts the file's previously unused import re to work. The remaining one (Optional) predates this change and CI lints laya/ only, so it is left alone rather than drive-bys fixed.

The gate was then mutation-checked — 10 mutants, one at a time, file reset between each, baseline re-run after:

mutant result
README "Why Route" cell 0.783 → 0.784 FAIL 1
README 51-language table cell 0.657 → 0.656 FAIL 1
BENCHMARKS cell 0.439 → 0.440 FAIL 1
a README figure row deleted FAIL 2 (row count + cell count)
17,416 questions → 17,440 FAIL 1
13 other languages → 12 FAIL 1
a cell printed unrounded (0.8600) FAIL 1
the two checkpoint columns swapped FAIL 2
artifact summary perturbed by 1e-4 FAIL 1 (cross-check is live, not tautological)
artifact meta.seed 13 → 14 FAIL 1

Zero survivors. cells_checked is asserted to equal 28, so an empty or partial table parse cannot pass vacuously.

Not asserted, and stated as such in the code: README's Router (Routed) column. No routed run exists in the artifact, so a check on it would be arithmetic on the other two columns rather than evidence. Adding it to the file's unbacked list would edit the same lines as #838's in-review hunk, so it is recorded as a comment here and will follow #838 rather than stack on it.

Ref #776, #298.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed (no public API change; the doc edits are the artifact and protocol provenance for the tables the test now pins)

Ref #776

The English-vs-the-rest figures are the project's most-read numbers and the only
benchmark tables that named neither an artifact nor a protocol. NandhaKishorM#776 re-ran them
with an independent harness; the transcription still had no gate. Every printed
cell in all three tables is now recomputed from the run's per-language suites and
compared, and the artifact's own english_vs_rest summary is compared against the
same recompute so an aggregation change cannot pass silently.
@NandhaKishorM
NandhaKishorM merged commit c92499e into NandhaKishorM:main Oct 11, 2026
36 of 38 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.3. Checking the three most-read benchmark tables against the artifact that produced them is the right place to spend 100 lines of test, because these are the cells someone quotes when deciding whether to adopt the project, and they were published bare.

Recomputing the accuracy from the per-language suites rather than reading the artifact's own summary is what makes the check real, and then checking that summary against the recomputation too closes the loop. Using the file's existing half-up rounded() matters as well: a check that rounded differently from the table would have failed on correct cells.

Naming the artifact and the protocol in BENCHMARKS.md and README, seed 13, 20 options per choice question, first 300 test rows per language, is the half of this I would have asked for if it were missing. A number without its protocol cannot be reproduced or disputed, and this project's position is that its benchmarks are reproducible.

Verified on the merged tree: tests/test_doc_tables.py passes at 181 checks, up from 114, across 28 printed cells. All 87 suites pass. The not-checkable list the suite prints is unchanged, so nothing that was previously asserted became unasserted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants