Skip to content

research: refresh the 51-language multilingual columns from the re-run (#208) - #389

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:research-refresh-51-language-sweep
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:research-refresh-51-language-sweep

Conversation

@PerryLink

@PerryLink PerryLink commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

research: refresh the 51-language multilingual columns from the re-run (#208)

Implements the refresh the maintainer asked for in
#208: "A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."

Both halves are now one run at one version. The first version of this file took the English
column from the #222 re-run at laya 0.3.7, and the review asked me to record that or re-run it
here. I re-ran both halves on this machine at laya 0.3.20, and separately checked that the
choice does not matter:

english clamped          0.3.7 == 0.3.20 on all 51 languages
english unclamped        0.3.7 == 0.3.20 on all 51 languages
multilingual clamped     0.3.7 == 0.3.20 on all 51 languages
multilingual unclamped   0.3.7 == 0.3.20 on all 51 languages

Every comparison above is asserted inside the build script over accuracy, ECE, macro-F1 and mean
confidence, and the script fails if any language moves. So the artifact carries one version, and
provenance records per model where each half came from, as asked.

Why the multilingual columns are a re-run

The committed cpu_51_language_sweep.json was measured at laya 0.2.0 and records
laya-multilingual at 0.3661 macro accuracy. It does not reproduce:

run macro accuracy per-language accuracy matching the committed file
committed, laya 0.2.0, torch 2.8.0 0.3661 —
laya 0.2.0, x86 CPU 0.4008 6 / 51
laya 0.3.7, x86 CPU 0.4008 6 / 51
laya 0.3.20, x86 CPU 0.4008 6 / 51
laya 0.3.11, Apple M5, CPU (@aashish254) 0.4008 6 / 51
laya 0.3.11, Apple M5, MPS 0.4008 6 / 51

The laya half of the same file reproduces 51/51 exactly on all of them, which is what makes
this a property of the multilingual columns rather than of the harness or the machine.

What this PR does

research/results/cpu_51_language_sweep_refreshed.json (new). Every per-language entry keeps
the superseded committed row beside the new one so the before/after stays visible, and both
temperature regimes are kept — laya-multilingual ships no fitted temperatures, so the two
columns being byte-identical is the evidence for that rather than an assertion of it. meta
carries laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options,
per_lang, dataset, split and instructions — a strict superset of what the committed file
recorded — and provenance names each half's source.

research/README.md, which the refresh had missed entirely:

BENCHMARKS.md. The 51-language table is regenerated from the artifact rather than
hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so
en moves to the bottom where the gain is negative) — and a note above it records why the
multilingual columns are a re-run, keeping the committed row in the same table for comparison.

README.md. The stale 45 of 51 in the Jev comparison becomes 48 of 51, and the
multilingual section states both checkpoints' macros with ECE quoted twice: 0.733 as committed
and 0.571 as served, because the committed figure predates the temperature clamp.

tests/test_doc_tables.py. The table is now checkable, so it is checked: every cell of every
row against the refreshed artifact, that the laya column still equals the committed file
exactly, and that the multilingual column differs from it in more than 40 languages. 54 → 59
checks.

Deliberate choices tested:

  • Refresh into a new file rather than overwriting the committed one. The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently.
  • Both halves re-run rather than labelled as mixed. Recording "English from research: re-run the 51-language sweep in both temperature regimes (the #208 ask) #222 at 0.3.7" would have been honest but leaves a reader unable to compare the two columns as a pair; running both here and asserting the versions agree gives one version and no caveat.
  • The environment goes in the file, not only in the PR, and is a superset of the committed file's meta, so a reader of the JSON alone does not have to find this PR.
  • laya's column is refreshed too, even though nothing changed. It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that.
$ python tests/test_doc_tables.py              59 passed, 0 failed   (54 before)
$ python tests/test_packaging.py               all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/

The gap is quantified but not explained

The multilingual shift is not sampling noise. Each committed accuracy is a count out of 100, so
one language moving 0.05 proves nothing; the direction across all 51 does. The multilingual mean
moves +0.0347 (median +0.0300, sd 0.0423) — 3.8 pooled standard errors — while the laya
column of the same file reproduces on 51 of 51 with sd 0.0000. mean_confidence moves with it
(+0.0369, up in 47 of 51, only weakly correlated at r = 0.349), and the shift is largest where
the committed run was worst (+0.0506 hardest third, +0.0182 easiest).

Also ruled out by measurement rather than argument:

candidate result
the laya version 0.2.0, 0.3.7 and 0.3.20 all give 0.4008, asserted per language in the build
smaller effective token budget max_len 1024/768/512/384/256 give bit-identical accuracy on the ten worst languages (bn 0.45 at all five)
the temperature clamp laya-multilingual ships no fitted temperatures, so both regimes are byte-identical
the checkpoint same model.safetensors blob at every revision in the Hub history
sampling parameters, harness, layout laya reproduces exactly under all of them, in the same run

The one variable from the committed run not reproduced is torch 2.8.0 (this box runs
2.14.0+cpu). I do not think it is worth installing: a runtime numerics change would have to move
the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English
checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or
reduction-order difference.

The committed file is left in place rather than deleted, so the discrepancy stays visible.

Credit to @aashish254, whose M5 run on both devices turned this from one machine's disagreement
into a settled finding.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for chasing this down so thoroughly, and credit to @aashish254 for the cross-device runs. Keeping the committed rows beside the refreshed ones, and checking every cell in test_doc_tables, is exactly right. A few fixes before merge. The English column in the new file is identical to cpu_51_language_sweep_clamped.json, including the seconds, so please record per model where each column came from (English from the #222 run at 0.3.7) or re-run it on this machine. Please also update research/README.md, which still says 45 of 51 and 0.366, mark the Khmer 95.2% figure as raw temperature (it is 70.5% as served), and point the note about the other re-runs at #208, since only one of them is committed.

@PerryLink

Copy link
Copy Markdown
Contributor Author

All four fixed in e053986, rebased onto current main.

1. Column provenance. You were right, and it was worse than a missing label — meta.laya_version said 0.3.20 when both halves are 0.3.7. I read the version of the environment rather than the version recorded in each run's own config. The artifact now carries a provenance block, per model:

"english":      {"source": "cpu_51_language_sweep_clamped.json (PR #222, field `clamped_rerun`)",
                 "laya_version": "0.3.7", "max_len": 512,  "freshly_run_for_this_file": false},
"multilingual": {"source": "a fresh run on the machine described in `meta`",
                 "laya_version": "0.3.7", "max_len": 1024, "freshly_run_for_this_file": true}

and the English claim is now checked rather than asserted: the build compares it against #222's clamped_rerun across all 51 languages and fails if they diverge, which they do not (af ECE 0.5588, mean confidence 0.8488 in both). I took your first option rather than re-running English on this machine — the two halves having different histories is the informative part, so labelling it is better than hiding it behind a fresh run at yet another version. A 0.3.20 English run is in progress here and I can fold it in if you would rather have both halves on one version.

2. research/README.md — 45 of 51 → 48 of 51, 0.366 → 0.401, with a pointer to the refreshed file. I had missed this file entirely; it is the one place the old numbers were still stated as the headline.

3. Khmer — now labelled: 0.952 is the raw temperature, 0.705 as served, at the same 0.000 accuracy. Both figures were already in the data and I quoted only the raw one.

4. The other re-runs — the cpu_51_language_sweep_clamped.json row now attributes that file to #222 and says it covers the English half, and credits #208 for the multilingual re-run, since only one of the two is a re-run of the whole sweep. The README also distinguishes 0.733 as committed from 0.571 as served.

$ python tests/test_doc_tables.py      59 passed, 0 failed
$ python tests/test_packaging.py       all packaging tests passed

@PerryLink
PerryLink force-pushed the research-refresh-51-language-sweep branch from e053986 to 5bb8ab7 Compare September 24, 2026 19:04
@PerryLink

Copy link
Copy Markdown
Contributor Author

Both halves are now one run at one version, which removes the caveat you asked me to record rather than documenting it.

5bb8ab7, rebased onto current main.

On the provenance point. You were right that the English column was #222's, and right that it was worse than a missing label — meta.laya_version said 0.3.20 while both halves were 0.3.7, because I read the version of the environment rather than the version recorded in the run's own config. Rather than label the mixing, I re-ran both halves here at laya 0.3.20, and then checked that the choice could not matter:

english clamped          0.3.7 == 0.3.20 on all 51 languages
english unclamped        0.3.7 == 0.3.20 on all 51 languages
multilingual clamped     0.3.7 == 0.3.20 on all 51 languages
multilingual unclamped   0.3.7 == 0.3.20 on all 51 languages

Asserted per language over accuracy, ECE, macro-F1 and mean confidence, inside the build script, and it fails if anything moves. provenance now names each half's source and version, and meta is a strict superset of the committed file's — it gained cpu, platform, python, transformers, harness, seed, n_options, per_lang, dataset, split and instructions on top of device/laya/threads/timestamp/torch.

Worth noting for #208 itself: the multilingual column at 0.3.20 is 0.4008, the same as 0.3.7 and 0.2.0. The laya version is now ruled out on the multilingual half as well as the English one, so the committed row is not explained by a version difference in either direction.

Your other three points, all in 5bb8ab7:

$ python tests/test_doc_tables.py      59 passed, 0 failed
$ python tests/test_packaging.py       all packaging tests passed

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, re-running both halves at 0.3.20 and recording per-model provenance is exactly what was needed, and the cell-by-cell check in test_doc_tables is great. A few stale figures are left in files this PR touches. README.md line 354 still says 45 / 51 in the Router table. README.md lines 358 and 864 and BENCHMARKS.md line 129 still quote Khmer at 0.952 / 95.2% without saying it is the raw temperature. Please label those raw with 0.705 as served, or switch them to the served numbers. On BENCHMARKS.md line 129, the 'never drops below 0.885' figure is also raw (the served minimum is 0.621). Once those match the table, this is ready.

NandhaKishorM#208)

Implements the refresh the maintainer asked for in
[NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."*

**Both halves are one run at one version** — a fresh 0.3.20 run on this machine — and the
per-model `provenance` block records where each came from, as the review asked.

## Why the multilingual columns are a re-run

The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records
`laya-multilingual` at **0.3661** macro accuracy. It does not reproduce on any setup tried:

| run | macro accuracy | per-language accuracy matching the committed file |
|---|---|---|
| committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — |
| laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.20, x86 CPU (this file) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 |

The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes
this a property of the multilingual columns rather than of the harness or the machine.

## Stale figures in files this PR touches

The review listed four still quoting the raw temperature or the pre-refresh count; all four now
match the table, with the raw and served values both given:

| file | was | now |
|---|---|---|
| `README.md` Router table | `45 / 51` twice | `48 / 51`, the refreshed count |
| `README.md` limits note | Khmer `0.952` unlabelled | `0.952` raw, `0.705` as served |
| `README.md` multilingual section | Khmer `95.2%` | `95.2%` raw, `70.5%` as served |
| `BENCHMARKS.md` | `0.952` and "never below `0.885`" | raw, with `0.705` and `0.621` as served |

The served figures are the artifact's own: Khmer `0.9515` raw → `0.7051` served, and the minimum
per-language mean confidence `0.8848` raw → `0.6209` served.

## What this PR does

- **`research/results/cpu_51_language_sweep_refreshed.json`** (new): every per-language entry keeps the superseded committed row beside the new one, both temperature regimes are kept (`laya-multilingual` ships no fitted temperatures, so the two being byte-identical is the evidence for that), and `meta` is a strict superset of the committed file's — laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options, per_lang, dataset, split, instructions.
- **`BENCHMARKS.md`**: the 51-language table is regenerated from the artifact rather than hand-edited — all 51 rows, both ECE columns and the sort order — with a note above it explaining why the multilingual columns are a re-run and keeping the committed row for comparison.
- **`README.md`**: the four figures above.
- **`research/README.md`**: `45 of 51` → `48 of 51`, `0.366` → `0.401`, Khmer labelled raw, and the re-run rows attributed to NandhaKishorM#222 (English half) and NandhaKishorM#208 (multilingual).
- **`tests/test_doc_tables.py`**: every cell of every row is checked against the artifact, the `laya` column must still equal the committed file exactly, and the multilingual column must differ from it in more than 40 languages. 54 → 59 checks on the branch that added it, and the rebase kept both this block and main's new fp16-table block (114 checks together).

Deliberate choices tested:

- **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently.
- **Both halves re-run rather than labelled as mixed.** Recording "English from NandhaKishorM#222 at 0.3.7" would have been honest but leaves a reader unable to compare the columns as a pair. Both were re-run here, and the equivalence at 0.3.7 vs 0.3.20 is asserted per language inside the build script over accuracy, ECE, macro-F1 and mean confidence.
- **Raw *and* served everywhere the clamp moves a number**, rather than switching to served only — the raw figure is what the committed sweep records, so dropping it would break the comparison the file exists to make.

```
$ python tests/test_doc_tables.py              114 passed, 0 failed
$ python tests/test_packaging.py               all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

## The gap is quantified but not explained

The shift is not sampling noise: the multilingual mean moves **+0.0347** (3.8 pooled standard
errors) while the `laya` column reproduces on 51 of 51 with sd 0.0000, and `mean_confidence`
moves with it (+0.0369, up in 47 of 51, correlated only r = 0.349). Ruled out by measurement: the
laya version (all three give 0.4008), the token budget (`max_len` 1024→256 bit-identical on the
ten worst languages), the temperature clamp (no fitted temperatures), the checkpoint (same
`model.safetensors` blob at every Hub revision), and sampling/harness/layout (`laya` reproduces
exactly under all of them).

The one variable not reproduced is `torch 2.8.0` (this box runs 2.14.0+cpu). A runtime numerics
change would have to move the multilingual checkpoint by +0.035 uniformly across 51 languages
while leaving the English checkpoint on the same 51 states bit-identical, which is not a plausible
shape for a kernel or reduction-order difference.

The committed file is left in place rather than deleted, so the discrepancy stays visible.

Credit to @aashish254, whose M5 runs on both devices turned this from one machine's disagreement
into a settled finding.
@PerryLink
PerryLink force-pushed the research-refresh-51-language-sweep branch from 5bb8ab7 to ad28be4 Compare September 25, 2026 15:15
@NandhaKishorM
NandhaKishorM merged commit 14379cb into NandhaKishorM:main Sep 29, 2026
20 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @PerryLink! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants