Repository navigation
research: refresh the 51-language multilingual columns from the re-run (#208) - #389
Conversation
56ef2a9 to
476fa69
Compare
|
Thank you for chasing this down so thoroughly, and credit to @aashish254 for the cross-device runs. Keeping the committed rows beside the refreshed ones, and checking every cell in test_doc_tables, is exactly right. A few fixes before merge. The English column in the new file is identical to cpu_51_language_sweep_clamped.json, including the seconds, so please record per model where each column came from (English from the #222 run at 0.3.7) or re-run it on this machine. Please also update research/README.md, which still says 45 of 51 and 0.366, mark the Khmer 95.2% figure as raw temperature (it is 70.5% as served), and point the note about the other re-runs at #208, since only one of them is committed. |
476fa69 to
e053986
Compare
|
All four fixed in e053986, rebased onto current 1. Column provenance. You were right, and it was worse than a missing label — "english": {"source": "cpu_51_language_sweep_clamped.json (PR #222, field `clamped_rerun`)",
"laya_version": "0.3.7", "max_len": 512, "freshly_run_for_this_file": false},
"multilingual": {"source": "a fresh run on the machine described in `meta`",
"laya_version": "0.3.7", "max_len": 1024, "freshly_run_for_this_file": true}and the English claim is now checked rather than asserted: the build compares it against #222's 2. 3. Khmer — now labelled: 4. The other re-runs — the |
e053986 to
5bb8ab7
Compare
|
Both halves are now one run at one version, which removes the caveat you asked me to record rather than documenting it.
On the provenance point. You were right that the English column was #222's, and right that it was worse than a missing label — Asserted per language over accuracy, ECE, macro-F1 and mean confidence, inside the build script, and it fails if anything moves. Worth noting for #208 itself: the multilingual column at 0.3.20 is 0.4008, the same as 0.3.7 and 0.2.0. The laya version is now ruled out on the multilingual half as well as the English one, so the committed row is not explained by a version difference in either direction. Your other three points, all in
|
|
Thank you, re-running both halves at 0.3.20 and recording per-model provenance is exactly what was needed, and the cell-by-cell check in test_doc_tables is great. A few stale figures are left in files this PR touches. README.md line 354 still says 45 / 51 in the Router table. README.md lines 358 and 864 and BENCHMARKS.md line 129 still quote Khmer at 0.952 / 95.2% without saying it is the raw temperature. Please label those raw with 0.705 as served, or switch them to the served numbers. On BENCHMARKS.md line 129, the 'never drops below 0.885' figure is also raw (the served minimum is 0.621). Once those match the table, this is ready. |
NandhaKishorM#208) Implements the refresh the maintainer asked for in [NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and the table from this re-run, with the environment recorded next to it, would be very welcome."* **Both halves are one run at one version** — a fresh 0.3.20 run on this machine — and the per-model `provenance` block records where each came from, as the review asked. ## Why the multilingual columns are a re-run The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records `laya-multilingual` at **0.3661** macro accuracy. It does not reproduce on any setup tried: | run | macro accuracy | per-language accuracy matching the committed file | |---|---|---| | committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — | | laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.20, x86 CPU (this file) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 | The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes this a property of the multilingual columns rather than of the harness or the machine. ## Stale figures in files this PR touches The review listed four still quoting the raw temperature or the pre-refresh count; all four now match the table, with the raw and served values both given: | file | was | now | |---|---|---| | `README.md` Router table | `45 / 51` twice | `48 / 51`, the refreshed count | | `README.md` limits note | Khmer `0.952` unlabelled | `0.952` raw, `0.705` as served | | `README.md` multilingual section | Khmer `95.2%` | `95.2%` raw, `70.5%` as served | | `BENCHMARKS.md` | `0.952` and "never below `0.885`" | raw, with `0.705` and `0.621` as served | The served figures are the artifact's own: Khmer `0.9515` raw → `0.7051` served, and the minimum per-language mean confidence `0.8848` raw → `0.6209` served. ## What this PR does - **`research/results/cpu_51_language_sweep_refreshed.json`** (new): every per-language entry keeps the superseded committed row beside the new one, both temperature regimes are kept (`laya-multilingual` ships no fitted temperatures, so the two being byte-identical is the evidence for that), and `meta` is a strict superset of the committed file's — laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options, per_lang, dataset, split, instructions. - **`BENCHMARKS.md`**: the 51-language table is regenerated from the artifact rather than hand-edited — all 51 rows, both ECE columns and the sort order — with a note above it explaining why the multilingual columns are a re-run and keeping the committed row for comparison. - **`README.md`**: the four figures above. - **`research/README.md`**: `45 of 51` → `48 of 51`, `0.366` → `0.401`, Khmer labelled raw, and the re-run rows attributed to NandhaKishorM#222 (English half) and NandhaKishorM#208 (multilingual). - **`tests/test_doc_tables.py`**: every cell of every row is checked against the artifact, the `laya` column must still equal the committed file exactly, and the multilingual column must differ from it in more than 40 languages. 54 → 59 checks on the branch that added it, and the rebase kept both this block and main's new fp16-table block (114 checks together). Deliberate choices tested: - **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently. - **Both halves re-run rather than labelled as mixed.** Recording "English from NandhaKishorM#222 at 0.3.7" would have been honest but leaves a reader unable to compare the columns as a pair. Both were re-run here, and the equivalence at 0.3.7 vs 0.3.20 is asserted per language inside the build script over accuracy, ECE, macro-F1 and mean confidence. - **Raw *and* served everywhere the clamp moves a number**, rather than switching to served only — the raw figure is what the committed sweep records, so dropping it would break the comparison the file exists to make. ``` $ python tests/test_doc_tables.py 114 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` ## The gap is quantified but not explained The shift is not sampling noise: the multilingual mean moves **+0.0347** (3.8 pooled standard errors) while the `laya` column reproduces on 51 of 51 with sd 0.0000, and `mean_confidence` moves with it (+0.0369, up in 47 of 51, correlated only r = 0.349). Ruled out by measurement: the laya version (all three give 0.4008), the token budget (`max_len` 1024→256 bit-identical on the ten worst languages), the temperature clamp (no fitted temperatures), the checkpoint (same `model.safetensors` blob at every Hub revision), and sampling/harness/layout (`laya` reproduces exactly under all of them). The one variable not reproduced is `torch 2.8.0` (this box runs 2.14.0+cpu). A runtime numerics change would have to move the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or reduction-order difference. The committed file is left in place rather than deleted, so the discrepancy stays visible. Credit to @aashish254, whose M5 runs on both devices turned this from one machine's disagreement into a settled finding.
5bb8ab7 to
ad28be4
Compare
|
Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22. |
research: refresh the 51-language multilingual columns from the re-run (#208)
Implements the refresh the maintainer asked for in
#208: "A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."
Both halves are now one run at one version. The first version of this file took the English
column from the #222 re-run at laya 0.3.7, and the review asked me to record that or re-run it
here. I re-ran both halves on this machine at laya 0.3.20, and separately checked that the
choice does not matter:
Every comparison above is asserted inside the build script over accuracy, ECE, macro-F1 and mean
confidence, and the script fails if any language moves. So the artifact carries one version, and
provenancerecords per model where each half came from, as asked.Why the multilingual columns are a re-run
The committed
cpu_51_language_sweep.jsonwas measured at laya 0.2.0 and recordslaya-multilingualat 0.3661 macro accuracy. It does not reproduce:The
layahalf of the same file reproduces 51/51 exactly on all of them, which is what makesthis a property of the multilingual columns rather than of the harness or the machine.
What this PR does
research/results/cpu_51_language_sweep_refreshed.json(new). Every per-language entry keepsthe superseded committed row beside the new one so the before/after stays visible, and both
temperature regimes are kept —
laya-multilingualships no fitted temperatures, so the twocolumns being byte-identical is the evidence for that rather than an assertion of it.
metacarries laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options,
per_lang, dataset, split and instructions — a strict superset of what the committed file
recorded — and
provenancenames each half's source.research/README.md, which the refresh had missed entirely:45 of 51→48 of 51and0.366→0.401, with a pointer to the refreshed file, sothe headline finding no longer contradicts the table it links to;
95.2%is now labelled as the raw temperature, with 70.5% as served after theclamp at the same 0.000 accuracy;
cpu_51_language_sweep_clamped.jsonrow now attributes that file to research: re-run the 51-language sweep in both temperature regimes (the #208 ask) #222 and its Englishhalf, and credits The committed 51-language sweep's ECE and confidence columns predate the #42 temperature clamp and no longer reproduce #208 for the multilingual re-run.
BENCHMARKS.md. The 51-language table is regenerated from the artifact rather thanhand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so
enmoves to the bottom where the gain is negative) — and a note above it records why themultilingual columns are a re-run, keeping the committed row in the same table for comparison.
README.md. The stale45 of 51in the Jev comparison becomes 48 of 51, and themultilingual section states both checkpoints' macros with ECE quoted twice:
0.733as committedand
0.571as served, because the committed figure predates the temperature clamp.tests/test_doc_tables.py. The table is now checkable, so it is checked: every cell of everyrow against the refreshed artifact, that the
layacolumn still equals the committed fileexactly, and that the multilingual column differs from it in more than 40 languages. 54 → 59
checks.
Deliberate choices tested:
meta, so a reader of the JSON alone does not have to find this PR.laya's column is refreshed too, even though nothing changed. It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that.The gap is quantified but not explained
The multilingual shift is not sampling noise. Each committed accuracy is a count out of 100, so
one language moving 0.05 proves nothing; the direction across all 51 does. The multilingual mean
moves +0.0347 (median +0.0300, sd 0.0423) — 3.8 pooled standard errors — while the
layacolumn of the same file reproduces on 51 of 51 with sd 0.0000.
mean_confidencemoves with it(+0.0369, up in 47 of 51, only weakly correlated at r = 0.349), and the shift is largest where
the committed run was worst (+0.0506 hardest third, +0.0182 easiest).
Also ruled out by measurement rather than argument:
max_len1024/768/512/384/256 give bit-identical accuracy on the ten worst languages (bn0.45 at all five)laya-multilingualships no fitted temperatures, so both regimes are byte-identicalmodel.safetensorsblob at every revision in the Hub historylayareproduces exactly under all of them, in the same runThe one variable from the committed run not reproduced is
torch 2.8.0(this box runs2.14.0+cpu). I do not think it is worth installing: a runtime numerics change would have to move
the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English
checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or
reduction-order difference.
The committed file is left in place rather than deleted, so the discrepancy stays visible.
Credit to @aashish254, whose M5 run on both devices turned this from one machine's disagreement
into a settled finding.