Repository navigation
research: re-run the 51-language sweep in both temperature regimes (the #208 ask) - #222
Conversation
|
Adding the multilingual half, since it turns out to be the shortest part of the answer. The clamp is a no-op on Per language the two runs are identical in 51/51 on every column; the only difference between the two files is That is worth stating rather than leaving as "not measured", so I have changed the limitation in the description above into this result. One thing I could not reproduce, which is not about the clampWhile running this I tried to reproduce the committed multilingual columns and could not. Two candidate causes, both removed:
So I have no explanation yet, and I would rather report the number than a guess. This is the same |
|
Adding the two checks I said I would run, so the elimination record is complete rather than suggestive. Hypothesis: I evaluated the multilingual checkpoint at the wrong Hypothesis: the committed sweep measured
Same answer to four decimals, and the two configs are byte-identical anyway — Hypothesis: the checkpoint changed between the committed run and now. Eliminated, and this one is unambiguous. Every revision from the committed sweep's timestamp to today: The Hub commits in that window are So The practical point for this PR is unaffected: the clamp is a no-op on this checkpoint (empty I will open this as its own issue with the per-language table if you would rather have it out of the PR thread. It is also the |
2c5e85f to
3fd1453
Compare
NandhaKishorM#208 ask) The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so the before/after is visible. `research/results/cpu_51_language_sweep_clamped.json` holds the committed columns, an `--unclamped` re-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases. committed unclamped re-run clamped re-run 0.2269 0.2269 0.2269 macro accuracy 0.7331 0.7331 0.5709 macro ECE 0.2053 0.2053 0.2053 macro F1 Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in 51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and this one 2.14.0. The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower. `choice:11+` is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at all: a temperature-scaled softmax has the same argmax at every positive temperature. Worth recording because it cuts against the reading that the clamp only distorts the published number: `acc_at_50_coverage` is the one rank-quality column that consumes the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98. The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the committed columns stopped matching the unclamped re-run or if the clamped re-run started matching them. Deliberate choices tested: - `bench_local.py` was not modified. It produced the committed file, and rewriting it to reproduce its own pre-clamp output would make the old and new runs come from different scripts. `laya_eval.py` already has `--unclamped` for this, and the three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are recorded in the new file rather than quietly dropped. - The clamped re-run is committed alongside rather than replacing anything, so `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite. - `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is wired into both lanes, and this suite is currently in neither; adding it here would collide with that PR and with the verbatim `test` job in NandhaKishorM#212. Known limitation: only the `english` checkpoint was re-run. The committed file's `part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks are in `part_a`, so the multilingual half is untouched and unmeasured here. Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.
Three more candidates for the unreconciled multilingual row are now ruled out on re-measurement rather than by inspection: the shipped head_max_len (256, read from the checkpoint's own config and recorded in the run), which of the two multilingual copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51), and a checkpoint change since the committed sweep (weights and config are identical by size and hash at every revision in that window). Also corrects the test count in the same file, which the new NandhaKishorM#208 guard changes.
3fd1453 to
5719d6a
Compare
|
Rebased onto current One conflict, in
Re-verified after the rebase: So the PR description's
|
|
Thank you for doing the #208 re-run, and for keeping the original file beside the new one so the before/after stays visible. The follow-up showing the clamp is a no-op on |
NandhaKishorM#208 ask) (NandhaKishorM#222) * research: re-run the 51-language sweep in both temperature regimes (the NandhaKishorM#208 ask) The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so the before/after is visible. `research/results/cpu_51_language_sweep_clamped.json` holds the committed columns, an `--unclamped` re-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases. committed unclamped re-run clamped re-run 0.2269 0.2269 0.2269 macro accuracy 0.7331 0.7331 0.5709 macro ECE 0.2053 0.2053 0.2053 macro F1 Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in 51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and this one 2.14.0. The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower. `choice:11+` is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at all: a temperature-scaled softmax has the same argmax at every positive temperature. Worth recording because it cuts against the reading that the clamp only distorts the published number: `acc_at_50_coverage` is the one rank-quality column that consumes the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98. The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the committed columns stopped matching the unclamped re-run or if the clamped re-run started matching them. Deliberate choices tested: - `bench_local.py` was not modified. It produced the committed file, and rewriting it to reproduce its own pre-clamp output would make the old and new runs come from different scripts. `laya_eval.py` already has `--unclamped` for this, and the three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are recorded in the new file rather than quietly dropped. - The clamped re-run is committed alongside rather than replacing anything, so `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite. - `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is wired into both lanes, and this suite is currently in neither; adding it here would collide with that PR and with the verbatim `test` job in NandhaKishorM#212. Known limitation: only the `english` checkpoint was re-run. The committed file's `part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks are in `part_a`, so the multilingual half is untouched and unmeasured here. Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed. * research: record the multilingual eliminations for the committed-row gap Three more candidates for the unreconciled multilingual row are now ruled out on re-measurement rather than by inspection: the shipped head_max_len (256, read from the checkpoint's own config and recorded in the run), which of the two multilingual copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51), and a checkpoint change since the committed sweep (weights and config are identical by size and hash at every revision in that window). Also corrects the test count in the same file, which the new NandhaKishorM#208 guard changes. --------- Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
NandhaKishorM#208) Implements the refresh the maintainer asked for in [NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and the table from this re-run, with the environment recorded next to it, would be very welcome."* ## What did not reproduce The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records `laya-multilingual` at **0.3661** macro accuracy. Four independent runs on three machines do not reproduce it: | run | macro accuracy | per-language accuracy matching the committed file | |---|---|---| | committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — | | laya 0.2.0, x86 CPU (this PR's author) | **0.4008** | 6 / 51 | | laya 0.3.7, x86 CPU (this PR's author) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 | The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes this a property of the multilingual columns rather than of the harness or the machine. ## What this PR does **`research/results/cpu_51_language_sweep_refreshed.json`** (new). The refreshed numbers with the environment recorded next to them, in the shape of the file it replaces — `meta` carries laya, torch, transformers, CPU, platform, python, harness, seed, n_options, per_lang and dataset — and every per-language entry keeps the superseded committed row beside the new one so the before/after stays visible: ```json "af": { "committed": {"accuracy": 0.35, "ece": 0.4845, "mean_confidence": 0.8028, ...}, "refreshed_clamped": {"accuracy": 0.35, "ece": 0.4394, "mean_confidence": 0.7894, ...}, "refreshed_unclamped": { ... } } ``` Both regimes are kept for the same reason `NandhaKishorM#222` kept them: `laya-multilingual` ships no fitted temperatures, so the clamp cannot move it, and the two columns being byte-identical is the evidence for that claim rather than an assertion of it. **The macros, refreshed:** | | committed | refreshed | |---|---|---| | `laya-multilingual` macro accuracy | 0.3661 | **0.4008** | | `laya-multilingual` macro ECE | 0.3869 | **0.3911** | | `laya-multilingual` clearing 3x random | 45 / 51 | **48 / 51** | | `laya` macro accuracy / ECE | 0.2269 / 0.7331 | 0.2269 / **0.5709** | **`BENCHMARKS.md`.** The 51-language table is regenerated from the artifact rather than hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so `en` moves to the bottom where the gain is negative) — and a note above it records why the multilingual columns are a re-run, keeping the committed row in the same table for comparison. **`README.md`.** The stale `45 of 51` in the Jev comparison becomes **48 of 51**, and the multilingual section now states both checkpoints' macros with the ECE quoted twice: `0.733` as committed and `0.571` as served, because the committed figure predates the temperature clamp. **`tests/test_doc_tables.py`.** The table is now checkable, so it is checked: every cell of every row against the refreshed artifact, that the `laya` column still equals the committed file exactly, and that the multilingual column differs from it in more than 40 languages. A future edit that quietly restores a committed number, or a future refresh that forgets to update a row, fails the suite. 54 → 59 checks. Deliberate choices tested: - **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19, and overwriting it would erase the only evidence that the multilingual columns used to read differently. The refreshed file carries the old rows inline, so nothing is lost either way, and the table cites the artifact it now prints. - **The environment goes in the file, not only in the PR.** `laya` 0.3.20 / torch 2.14.0+cpu / transformers 5.17.0 / the CPU string / the platform, plus `why` naming NandhaKishorM#208. The maintainer asked for the environment next to the numbers, and a reader of the JSON alone should not have to find this PR. - **`laya`'s column is refreshed too, even though nothing changed.** It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that. A refresh that touched only one column would leave the other unverifiable. - **The `45 of 51` in the README is a real correction, not a rounding.** It is the "Languages usable" row of the Jev comparison table and it described the committed multilingual run. 48 is the refreshed count by the same criterion (>3x random). ``` $ python tests/test_doc_tables.py 59 passed, 0 failed (54 before) $ python tests/test_packaging.py all packaging tests passed $ python -m pytest tests/test_serve.py tests/test_router_batch.py \ tests/test_audit_regressions.py -q 50 passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: this does not explain *why* the committed multilingual run reads 0.3661. Four machines agree it does not come back, and the direction is consistent rather than noisy — of the 16 languages that move by 0.05 or more, every one moves up — but no cause is established here, and the committed file is left in place rather than deleted so the discrepancy stays visible. If the intent is that the committed columns are simply wrong, deleting them is a separate decision. Credit to @aashish254, whose M5 run on both devices is what turned this from one machine's disagreement into a settled finding.
NandhaKishorM#208) Implements the refresh the maintainer asked for in [NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and the table from this re-run, with the environment recorded next to it, would be very welcome."* ## Why the multilingual columns are a re-run The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records `laya-multilingual` at **0.3661** macro accuracy. It does not reproduce: | run | macro accuracy | per-language accuracy matching the committed file | |---|---|---| | committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — | | laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 | The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes this a property of the multilingual columns rather than of the harness or the machine. ## What this PR does **`research/results/cpu_51_language_sweep_refreshed.json`** (new). Every per-language entry keeps the superseded committed row beside the new one so the before/after stays visible, and both temperature regimes are kept — `laya-multilingual` ships no fitted temperatures, so the two columns being byte-identical is the evidence for that rather than an assertion of it. **`provenance` records where each half came from, per model.** The two halves are not from the same run and the first version of this file did not say so: ```json "english": {"source": "cpu_51_language_sweep_clamped.json (PR NandhaKishorM#222, field `clamped_rerun`)", "laya_version": "0.3.7", "max_len": 512, "freshly_run_for_this_file": false, "note": "identical to the NandhaKishorM#222 english re-run, which is why the `laya` column matches the committed sweep on all 51 languages; the recorded `seconds` belong to that run"}, "multilingual": {"source": "a fresh run on the machine described in `meta`", "laya_version": "0.3.7", "max_len": 1024, "freshly_run_for_this_file": true} ``` I checked this rather than asserting it: the English column is compared against NandhaKishorM#222's `clamped_rerun` field for all 51 languages inside the build, and the script fails if they diverge (`af` ECE 0.5588, mean confidence 0.8488 in both). **`meta.laya_version` was also wrong in the first version** — it said 0.3.20 while both halves are 0.3.7, because I read the version of the environment rather than the version recorded in the run's own `config`. That field is now replaced by the per-model `provenance.laya_version` above, taken from each run's `config`. **`research/README.md`**, which the refresh had missed entirely: - `45 of 51` → **`48 of 51`** and `0.366` → **`0.401`**, with a pointer to the refreshed file, so the headline finding no longer contradicts the table it links to; - Khmer's `95.2%` is now labelled as the **raw** temperature, with **70.5%** as served after the clamp at the same 0.000 accuracy. Both numbers were in the data and only the raw one was quoted; - the `cpu_51_language_sweep_clamped.json` row now attributes that file to NandhaKishorM#222 and its English half, and credits NandhaKishorM#208 for the multilingual re-run, since only one of the two is a re-run of the whole sweep. **`BENCHMARKS.md`.** The 51-language table is regenerated from the artifact rather than hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so `en` moves to the bottom where the gain is negative) — and a note above it records why the multilingual columns are a re-run, keeping the committed row in the same table for comparison. **`README.md`.** The stale `45 of 51` in the Jev comparison becomes **48 of 51**, and the multilingual section states both checkpoints' macros with ECE quoted twice: `0.733` as committed and `0.571` as served, because the committed figure predates the temperature clamp. **`tests/test_doc_tables.py`.** The table is now checkable, so it is checked: every cell of every row against the refreshed artifact, that the `laya` column still equals the committed file exactly, and that the multilingual column differs from it in more than 40 languages. A future edit that quietly restores a committed number, or a refresh that forgets a row, fails the suite. 54 → 59 checks. Deliberate choices tested: - **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently. The refreshed file carries the old rows inline, so nothing is lost either way. - **Both halves in one file, with provenance, rather than two files.** A reader of the table needs one artifact behind it, and the interesting fact is precisely that the two halves have different histories — which is only visible if they sit in the same file with their sources labelled. - **The environment goes in the file, not only in the PR.** laya, torch, transformers, CPU, platform, python, harness, seed, n_options, per_lang, dataset, split and instructions, plus `why` naming NandhaKishorM#208, and now per-model provenance. The maintainer asked for the environment next to the numbers, and a reader of the JSON alone should not have to find this PR. - **`laya`'s column is refreshed too, even though nothing changed.** It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that. - **`45 of 51` → `48 of 51` is a real correction, not a rounding.** It is the "Languages usable" row of the Jev comparison table and it described the committed multilingual run. ``` $ python tests/test_doc_tables.py 59 passed, 0 failed (54 before) $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` ## The gap is quantified but not explained The multilingual shift is not sampling noise. Each committed accuracy is a count out of 100, so one language moving 0.05 proves nothing; the direction across all 51 does. The multilingual mean moves **+0.0347** (median +0.0300, sd 0.0423) — **3.8 pooled standard errors** — while the `laya` column of the same file reproduces on 51 of 51 with sd 0.0000. `mean_confidence` moves with it (+0.0369, up in 47 of 51, only weakly correlated at r = 0.349), and the shift is largest where the committed run was worst (+0.0506 hardest third, +0.0182 easiest). Also ruled out by measurement rather than argument: | candidate | result | |---|---| | smaller effective token budget | `max_len` 1024/768/512/384/256 give bit-identical accuracy on the ten worst languages (`bn` 0.45 at all five) | | the temperature clamp | `laya-multilingual` ships no fitted temperatures, so both regimes are byte-identical | | the checkpoint | same `model.safetensors` blob at every revision in the Hub history | | sampling parameters, harness, layout | `laya` reproduces exactly under all of them, in the same run | The one variable from the committed run not reproduced is `torch 2.8.0` (this box runs 2.14.0+cpu). I do not think it is worth installing: a runtime numerics change would have to move the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or reduction-order difference. Stated rather than left implicit. The committed file is left in place rather than deleted, so the discrepancy stays visible. Credit to @aashish254, whose M5 run on both devices turned this from one machine's disagreement into a settled finding.
NandhaKishorM#208) Implements the refresh the maintainer asked for in [NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and the table from this re-run, with the environment recorded next to it, would be very welcome."* **Both halves are now one run at one version.** The first version of this file took the English column from the NandhaKishorM#222 re-run at laya 0.3.7, and the review asked me to record that or re-run it here. I re-ran both halves on this machine at **laya 0.3.20**, and separately checked that the choice does not matter: ``` english clamped 0.3.7 == 0.3.20 on all 51 languages english unclamped 0.3.7 == 0.3.20 on all 51 languages multilingual clamped 0.3.7 == 0.3.20 on all 51 languages multilingual unclamped 0.3.7 == 0.3.20 on all 51 languages ``` Every comparison above is asserted inside the build script over accuracy, ECE, macro-F1 and mean confidence, and the script fails if any language moves. So the artifact carries one version, and `provenance` records per model where each half came from, as asked. ## Why the multilingual columns are a re-run The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records `laya-multilingual` at **0.3661** macro accuracy. It does not reproduce: | run | macro accuracy | per-language accuracy matching the committed file | |---|---|---| | committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — | | laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 | | **laya 0.3.20, x86 CPU** | **0.4008** | **6 / 51** | | laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 | The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes this a property of the multilingual columns rather than of the harness or the machine. ## What this PR does **`research/results/cpu_51_language_sweep_refreshed.json`** (new). Every per-language entry keeps the superseded committed row beside the new one so the before/after stays visible, and both temperature regimes are kept — `laya-multilingual` ships no fitted temperatures, so the two columns being byte-identical is the evidence for that rather than an assertion of it. `meta` carries laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options, per_lang, dataset, split and instructions — a strict superset of what the committed file recorded — and `provenance` names each half's source. **`research/README.md`**, which the refresh had missed entirely: - `45 of 51` → **`48 of 51`** and `0.366` → **`0.401`**, with a pointer to the refreshed file, so the headline finding no longer contradicts the table it links to; - Khmer's `95.2%` is now labelled as the **raw** temperature, with **70.5%** as served after the clamp at the same 0.000 accuracy; - the `cpu_51_language_sweep_clamped.json` row now attributes that file to NandhaKishorM#222 and its English half, and credits NandhaKishorM#208 for the multilingual re-run. **`BENCHMARKS.md`.** The 51-language table is regenerated from the artifact rather than hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so `en` moves to the bottom where the gain is negative) — and a note above it records why the multilingual columns are a re-run, keeping the committed row in the same table for comparison. **`README.md`.** The stale `45 of 51` in the Jev comparison becomes **48 of 51**, and the multilingual section states both checkpoints' macros with ECE quoted twice: `0.733` as committed and `0.571` as served, because the committed figure predates the temperature clamp. **`tests/test_doc_tables.py`.** The table is now checkable, so it is checked: every cell of every row against the refreshed artifact, that the `laya` column still equals the committed file exactly, and that the multilingual column differs from it in more than 40 languages. 54 → 59 checks. Deliberate choices tested: - **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently. - **Both halves re-run rather than labelled as mixed.** Recording "English from NandhaKishorM#222 at 0.3.7" would have been honest but leaves a reader unable to compare the two columns as a pair; running both here and asserting the versions agree gives one version and no caveat. - **The environment goes in the file, not only in the PR**, and is a superset of the committed file's `meta`, so a reader of the JSON alone does not have to find this PR. - **`laya`'s column is refreshed too, even though nothing changed.** It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that. ``` $ python tests/test_doc_tables.py 59 passed, 0 failed (54 before) $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` ## The gap is quantified but not explained The multilingual shift is not sampling noise. Each committed accuracy is a count out of 100, so one language moving 0.05 proves nothing; the direction across all 51 does. The multilingual mean moves **+0.0347** (median +0.0300, sd 0.0423) — **3.8 pooled standard errors** — while the `laya` column of the same file reproduces on 51 of 51 with sd 0.0000. `mean_confidence` moves with it (+0.0369, up in 47 of 51, only weakly correlated at r = 0.349), and the shift is largest where the committed run was worst (+0.0506 hardest third, +0.0182 easiest). Also ruled out by measurement rather than argument: | candidate | result | |---|---| | the laya version | **0.2.0, 0.3.7 and 0.3.20 all give 0.4008**, asserted per language in the build | | smaller effective token budget | `max_len` 1024/768/512/384/256 give bit-identical accuracy on the ten worst languages (`bn` 0.45 at all five) | | the temperature clamp | `laya-multilingual` ships no fitted temperatures, so both regimes are byte-identical | | the checkpoint | same `model.safetensors` blob at every revision in the Hub history | | sampling parameters, harness, layout | `laya` reproduces exactly under all of them, in the same run | The one variable from the committed run not reproduced is `torch 2.8.0` (this box runs 2.14.0+cpu). I do not think it is worth installing: a runtime numerics change would have to move the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or reduction-order difference. The committed file is left in place rather than deleted, so the discrepancy stays visible. Credit to @aashish254, whose M5 run on both devices turned this from one machine's disagreement into a settled finding.
NandhaKishorM#208) Implements the refresh the maintainer asked for in [NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and the table from this re-run, with the environment recorded next to it, would be very welcome."* **Both halves are one run at one version** — a fresh 0.3.20 run on this machine — and the per-model `provenance` block records where each came from, as the review asked. ## Why the multilingual columns are a re-run The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records `laya-multilingual` at **0.3661** macro accuracy. It does not reproduce on any setup tried: | run | macro accuracy | per-language accuracy matching the committed file | |---|---|---| | committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — | | laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 | | laya 0.3.20, x86 CPU (this file) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 | | laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 | The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes this a property of the multilingual columns rather than of the harness or the machine. ## Stale figures in files this PR touches The review listed four still quoting the raw temperature or the pre-refresh count; all four now match the table, with the raw and served values both given: | file | was | now | |---|---|---| | `README.md` Router table | `45 / 51` twice | `48 / 51`, the refreshed count | | `README.md` limits note | Khmer `0.952` unlabelled | `0.952` raw, `0.705` as served | | `README.md` multilingual section | Khmer `95.2%` | `95.2%` raw, `70.5%` as served | | `BENCHMARKS.md` | `0.952` and "never below `0.885`" | raw, with `0.705` and `0.621` as served | The served figures are the artifact's own: Khmer `0.9515` raw → `0.7051` served, and the minimum per-language mean confidence `0.8848` raw → `0.6209` served. ## What this PR does - **`research/results/cpu_51_language_sweep_refreshed.json`** (new): every per-language entry keeps the superseded committed row beside the new one, both temperature regimes are kept (`laya-multilingual` ships no fitted temperatures, so the two being byte-identical is the evidence for that), and `meta` is a strict superset of the committed file's — laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options, per_lang, dataset, split, instructions. - **`BENCHMARKS.md`**: the 51-language table is regenerated from the artifact rather than hand-edited — all 51 rows, both ECE columns and the sort order — with a note above it explaining why the multilingual columns are a re-run and keeping the committed row for comparison. - **`README.md`**: the four figures above. - **`research/README.md`**: `45 of 51` → `48 of 51`, `0.366` → `0.401`, Khmer labelled raw, and the re-run rows attributed to NandhaKishorM#222 (English half) and NandhaKishorM#208 (multilingual). - **`tests/test_doc_tables.py`**: every cell of every row is checked against the artifact, the `laya` column must still equal the committed file exactly, and the multilingual column must differ from it in more than 40 languages. 54 → 59 checks on the branch that added it, and the rebase kept both this block and main's new fp16-table block (114 checks together). Deliberate choices tested: - **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently. - **Both halves re-run rather than labelled as mixed.** Recording "English from NandhaKishorM#222 at 0.3.7" would have been honest but leaves a reader unable to compare the columns as a pair. Both were re-run here, and the equivalence at 0.3.7 vs 0.3.20 is asserted per language inside the build script over accuracy, ECE, macro-F1 and mean confidence. - **Raw *and* served everywhere the clamp moves a number**, rather than switching to served only — the raw figure is what the committed sweep records, so dropping it would break the comparison the file exists to make. ``` $ python tests/test_doc_tables.py 114 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` ## The gap is quantified but not explained The shift is not sampling noise: the multilingual mean moves **+0.0347** (3.8 pooled standard errors) while the `laya` column reproduces on 51 of 51 with sd 0.0000, and `mean_confidence` moves with it (+0.0369, up in 47 of 51, correlated only r = 0.349). Ruled out by measurement: the laya version (all three give 0.4008), the token budget (`max_len` 1024→256 bit-identical on the ten worst languages), the temperature clamp (no fitted temperatures), the checkpoint (same `model.safetensors` blob at every Hub revision), and sampling/harness/layout (`laya` reproduces exactly under all of them). The one variable not reproduced is `torch 2.8.0` (this box runs 2.14.0+cpu). A runtime numerics change would have to move the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or reduction-order difference. The committed file is left in place rather than deleted, so the discrepancy stays visible. Credit to @aashish254, whose M5 runs on both devices turned this from one machine's disagreement into a settled finding.
The re-run #208 asked for.
research/results/cpu_51_language_sweep.jsonis left exactly as it is, so the before/after is visible; the new file sits beside it.research/results/cpu_51_language_sweep_clamped.jsonholds the committed columns, an--unclampedre-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases.enenMacro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on
accuracyandmacro_f1in 51/51, onecein 48/51 and onmean_confidencein 49/51. The three that differ do so by0.0001, the last stored digit — the committed run used torch 2.8.0 and this one 2.14.0. Since both files round to 4 decimals, that is the resolution at which agreement can be judged, and the macro figures match exactly.The clamped run differs on
eceandmean_confidencein 51/51, every one of them lower.choice:11+is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100.One result cuts against the reading that the clamp only distorts the published number.
acc_at_50_coverageis the one column in the sweep that consumes the confidence values rather than just reporting them, and it goes up: macro 0.3004 → 0.3020, anden0.94 → 0.98. The flatter distribution selects a slightly better half, not a worse one.accuracyandmacro_f1cannot move at all, because a temperature-scaled softmax has the same argmax at every positive temperature — so settling the calibration question does not reopen the accuracy numbers.Deliberate choices tested
bench_local.pyis not modified. I considered adding--unclampedthere, since it is the script that produced the committed file and it readsagent.temperature_by_options. I did not, because a script changed to reproduce its own pre-clamp output means the old and new runs no longer come from the same code.laya_eval.pyalready has--unclampedfor this. It computes six of the ten columns the committed file stores; the four it does not (brier,nll,seconds,dropped) are recorded in the new file rather than quietly dropped, and the file is committed alongside rather than replacing anything, so the columns other tables cite keep their source.ci.ymlis untouched, on purpose. The guard below is currently in neither CI lane. #184 is adding the check that every model-free suite is wired into both, so wiring this one by hand here would collide with that PR and would change thetestjob that #212 currently leaves byte-identical tomain. Happy to add it to either lane separately if you would rather not wait for #184.The multilingual checkpoint is not affected
I said I would run this half if you wanted it, so I ran it. It turns out to be the shortest part of the answer: the clamp is a no-op on
laya-multilingual, because it ships no fitted temperatures. Itstemperature_by_optionsis{}and its per-question-typetemperatureis[1.0, 1.0, 1.0], soclamp_temperaturehas nothing to act on and--unclampedand the default path are literally the same computation:Per language the two runs are identical in 51/51 on every column; the only difference between the two files is
summary.seconds(605.4 vs 631.2), wall-clock. So #208's calibration problem does not reach this checkpoint — there is no pre-clamp regime to re-measure, and the committed0.3869is not a stale calibration figure the wayenglish's0.7331is.A separate thing I could not reproduce, and I am not claiming to have explained it
While running that, I tried to reproduce the committed multilingual columns and could not.
laya-multilingualre-measures at 0.4008 macro accuracy where the committed file holds 0.3661, and per language the re-run is higher in 38 of the 45 that differ (6 of 51 match). Theenglishcheckpoint reproduces 51/51 in the same harness, in the same run, so the sampling, prompt, option construction and inference path are right.Two candidate causes, both removed:
docs:model-card commits only.multilingual/model.safetensorsis643835514bytes with identicalencoder/config.jsonandrl_agent_config.jsonat both revisions.head_max_len. I suspected I had evaluated it at the global default of 192, sinceenglishships 192 andmultilingualships 256. The harness reads it from the checkpoint's own config, and the run'sconfigblock recordshead_max_len: 256, max_len: 1024. Not it.I have no explanation yet and would rather report the number than a guess. This is the same
6/51disagreementresearch/eval/README.mdalready records from #210, and it is independent of #208: that README's note is about the clamp, this is about whether the committed multilingual run is reproducible at all. I have changed no numbers and I am not asserting the re-run is the correct one. If useful I will open it as its own issue with the per-language table instead of leaving it in a PR thread.Known limitation
Only
part_aof theenglishcheckpoint has a before/after in the new file. The committed file'spart_bcovers the English checkpoint only and holds no calibration columns the clamp touches, so nothing else in it needs re-measuring.Guard
research/eval/test_laya_eval.pynow fails if the committed columns stopped matching the unclamped re-run, or if the clamped re-run started matching them, so the file cannot silently drift out of agreement with the sweep it describes.