Skip to content

research: re-run the 51-language sweep in both temperature regimes (the #208 ask) - #222

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
PerryLink:research-208-temperature-clamp-both-regimes
Sep 23, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
PerryLink:research-208-temperature-clamp-both-regimes

Conversation

@PerryLink

@PerryLink PerryLink commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

The re-run #208 asked for. research/results/cpu_51_language_sweep.json is left exactly as it is, so the before/after is visible; the new file sits beside it.

research/results/cpu_51_language_sweep_clamped.json holds the committed columns, an --unclamped re-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases.

committed re-run, raw temperatures re-run, as served
macro accuracy 0.2269 0.2269 0.2269
macro ECE 0.7331 0.7331 0.5709
macro F1 0.2053 0.2053 0.2053
mean confidence, en 0.9989 0.9989 0.9582
ECE, en 0.1789 0.1789 0.1382

Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on accuracy and macro_f1 in 51/51, on ece in 48/51 and on mean_confidence in 49/51. The three that differ do so by 0.0001, the last stored digit — the committed run used torch 2.8.0 and this one 2.14.0. Since both files round to 4 decimals, that is the resolution at which agreement can be judged, and the macro figures match exactly.

The clamped run differs on ece and mean_confidence in 51/51, every one of them lower. choice:11+ is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100.

python research/eval/laya_eval.py --model convaiinnovations/laya --langs all \
    --per-lang 100 --n-opts 20 --device cpu --unclamped
python research/eval/laya_eval.py --model convaiinnovations/laya --langs all \
    --per-lang 100 --n-opts 20 --device cpu

One result cuts against the reading that the clamp only distorts the published number. acc_at_50_coverage is the one column in the sweep that consumes the confidence values rather than just reporting them, and it goes up: macro 0.3004 → 0.3020, and en 0.94 → 0.98. The flatter distribution selects a slightly better half, not a worse one. accuracy and macro_f1 cannot move at all, because a temperature-scaled softmax has the same argmax at every positive temperature — so settling the calibration question does not reopen the accuracy numbers.

Deliberate choices tested

bench_local.py is not modified. I considered adding --unclamped there, since it is the script that produced the committed file and it reads agent.temperature_by_options. I did not, because a script changed to reproduce its own pre-clamp output means the old and new runs no longer come from the same code. laya_eval.py already has --unclamped for this. It computes six of the ten columns the committed file stores; the four it does not (brier, nll, seconds, dropped) are recorded in the new file rather than quietly dropped, and the file is committed alongside rather than replacing anything, so the columns other tables cite keep their source.

ci.yml is untouched, on purpose. The guard below is currently in neither CI lane. #184 is adding the check that every model-free suite is wired into both, so wiring this one by hand here would collide with that PR and would change the test job that #212 currently leaves byte-identical to main. Happy to add it to either lane separately if you would rather not wait for #184.

The multilingual checkpoint is not affected

I said I would run this half if you wanted it, so I ran it. It turns out to be the shortest part of the answer: the clamp is a no-op on laya-multilingual, because it ships no fitted temperatures. Its temperature_by_options is {} and its per-question-type temperature is [1.0, 1.0, 1.0], so clamp_temperature has nothing to act on and --unclamped and the default path are literally the same computation:

--unclamped  ->  macro_accuracy 0.4008  macro_ece 0.3911  macro_f1 0.3737
default      ->  macro_accuracy 0.4008  macro_ece 0.3911  macro_f1 0.3737

Per language the two runs are identical in 51/51 on every column; the only difference between the two files is summary.seconds (605.4 vs 631.2), wall-clock. So #208's calibration problem does not reach this checkpoint — there is no pre-clamp regime to re-measure, and the committed 0.3869 is not a stale calibration figure the way english's 0.7331 is.

A separate thing I could not reproduce, and I am not claiming to have explained it

While running that, I tried to reproduce the committed multilingual columns and could not. laya-multilingual re-measures at 0.4008 macro accuracy where the committed file holds 0.3661, and per language the re-run is higher in 38 of the 45 that differ (6 of 51 match). The english checkpoint reproduces 51/51 in the same harness, in the same run, so the sampling, prompt, option construction and inference path are right.

Two candidate causes, both removed:

  • A checkpoint update. The Hub commits between the sweep and now are docs: model-card commits only. multilingual/model.safetensors is 643835514 bytes with identical encoder/config.json and rl_agent_config.json at both revisions.
  • head_max_len. I suspected I had evaluated it at the global default of 192, since english ships 192 and multilingual ships 256. The harness reads it from the checkpoint's own config, and the run's config block records head_max_len: 256, max_len: 1024. Not it.

I have no explanation yet and would rather report the number than a guess. This is the same 6/51 disagreement research/eval/README.md already records from #210, and it is independent of #208: that README's note is about the clamp, this is about whether the committed multilingual run is reproducible at all. I have changed no numbers and I am not asserting the re-run is the correct one. If useful I will open it as its own issue with the per-language table instead of leaving it in a PR thread.

Known limitation

Only part_a of the english checkpoint has a before/after in the new file. The committed file's part_b covers the English checkpoint only and holds no calibration columns the clamp touches, so nothing else in it needs re-measuring.

Guard

research/eval/test_laya_eval.py now fails if the committed columns stopped matching the unclamped re-run, or if the clamped re-run started matching them, so the file cannot silently drift out of agreement with the sweep it describes.

python research/eval/test_laya_eval.py
62 passed, 0 failed

@PerryLink

Copy link
Copy Markdown
Contributor Author

Adding the multilingual half, since it turns out to be the shortest part of the answer.

The clamp is a no-op on laya-multilingual, because it ships no fitted temperatures. Its temperature_by_options is {} and its per-question-type temperature is [1.0, 1.0, 1.0], so clamp_temperature has nothing to act on and --unclamped and the default path are the same computation:

temperature_by_options_raw  {}
temperature_by_options        {}          <- nothing for the clamp to move
--unclamped  ->  macro_accuracy 0.4008  macro_ece 0.3911  macro_f1 0.3737
default      ->  macro_accuracy 0.4008  macro_ece 0.3911  macro_f1 0.3737

Per language the two runs are identical in 51/51 on every column; the only difference between the two files is summary.seconds (605.4 vs 631.2), which is wall-clock. So #208's calibration problem does not reach this checkpoint at all — there is no pre-clamp regime to re-measure, and 0.3869 in the committed file is not a stale calibration figure the way english's 0.7331 is.

That is worth stating rather than leaving as "not measured", so I have changed the limitation in the description above into this result.

One thing I could not reproduce, which is not about the clamp

While running this I tried to reproduce the committed multilingual columns and could not. laya-multilingual re-measures at 0.4008 macro accuracy where the committed file holds 0.3661, and per language the re-run is higher in 38 of the 45 that differ (only 6 of 51 match). The english checkpoint reproduces 51/51, in the same harness, in the same run, so the sampling, prompt, option construction and inference path are right.

Two candidate causes, both removed:

  • A checkpoint update. The Hub commits between the sweep and now are docs: model-card commits only; multilingual/model.safetensors is 643835514 bytes with the same encoder/config.json and rl_agent_config.json at both revisions. Nothing changed.
  • head_max_len. I suspected I had evaluated it at the global default of 192, since english ships 192 and multilingual ships 256. The harness reads head_max_len from the checkpoint's own config, and the run's own config block records head_max_len: 256, max_len: 1024, so it was 256. Not it.

So I have no explanation yet, and I would rather report the number than a guess. This is the same 6/51 disagreement that research/eval/README.md already records from #210, and it is independent of #208 — that README's macro_ece note is about the clamp, but this accuracy gap is a separate, still-open question about whether the committed multilingual run is reproducible at all. I have not changed those numbers and I am not claiming the re-run is the correct one; if it would help I can open it as its own issue with the per-language table rather than folding it in here.

@PerryLink

Copy link
Copy Markdown
Contributor Author

Adding the two checks I said I would run, so the elimination record is complete rather than suggestive.

Hypothesis: I evaluated the multilingual checkpoint at the wrong head_max_len. Eliminated. english ships head_max_len=192 and multilingual ships 256, and build_sequence truncates the option block to that budget, so 192 would have cut 64 tokens the checkpoint expects. I re-ran it against the bundled multilingual/ subfolder with the value read from the checkpoint's own config, and the run's config block confirms it: head_max_len: 256, max_len: 1024. Result 0.4008 / macro_ece 0.3911, per-language accuracy identical to the committed file in 6/51. The first run and this one agree exactly, so the earlier numbers were not an artifact of the budget.

Hypothesis: the committed sweep measured laya-multilingual (the separate repo) while I measured the bundled subfolder, and the two are different checkpoints. Eliminated. I ran the separate repo through the same harness:

checkpoint head_max_len macro accuracy identical to committed
bundled multilingual/ subfolder 256 0.4008 6/51
convaiinnovations/laya-multilingual 256 0.4008 6/51
committed file — 0.3661 —

Same answer to four decimals, and the two configs are byte-identical anyway — model.safetensors is 643835514 bytes with sha256 b99c8bea… and rl_agent_config.json 472 bytes with the same hash in both repos. So the bundled copy is a copy, and neither repo reproduces the committed columns.

Hypothesis: the checkpoint changed between the committed run and now. Eliminated, and this one is unambiguous. Every revision from the committed sweep's timestamp to today:

multilingual/model.safetensors     643835514   sha256 b99c8bea239c53f6   (both revisions)
multilingual/encoder/config.json         1938   sha256 0de0e2d306388737   (both revisions)
multilingual/rl_agent_config.json         472   sha256 00e35f88bb731bb9   (both revisions)

The Hub commits in that window are docs:/assets: model-card commits only. Nothing that the model reads has changed.

So english reproduces 51/51 and multilingual reproduces 6/51, in one harness, in one process, on unchanged weights that I can name by hash. The sampler, prompt, option construction and inference path are therefore right — english proves it — and something specific to the multilingual measurement differs, which I have not found. I am leaving it as a reported measurement rather than a diagnosis, and I have not touched the committed numbers or claimed the re-run supersedes them.

The practical point for this PR is unaffected: the clamp is a no-op on this checkpoint (empty temperature_by_options, [1.0, 1.0, 1.0]), verified identical in 51/51 per language between the two regimes, so #208 does not reach it regardless of which of the two accuracy figures is the right one.

I will open this as its own issue with the per-language table if you would rather have it out of the PR thread. It is also the 6/51 line research/eval/README.md already carries from #210, which that README described as unresolved; these three runs are what I have been able to rule out since.

@PerryLink
PerryLink force-pushed the research-208-temperature-clamp-both-regimes branch from 2c5e85f to 3fd1453 Compare September 23, 2026 09:26
NandhaKishorM#208 ask)

The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from
before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer
reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so
the before/after is visible.

`research/results/cpu_51_language_sweep_clamped.json` holds the committed columns,
an `--unclamped` re-run and a default re-run side by side, per language, from the
same 51 languages and 5,100 cases.

    committed      unclamped re-run   clamped re-run
    0.2269         0.2269             0.2269     macro accuracy
    0.7331         0.7331             0.5709     macro ECE
    0.2053         0.2053             0.2053     macro F1

Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the
committed macro ECE at 0.7331, so the clamp is the only variable left. Per language
the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in
51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do
so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and
this one 2.14.0.

The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower.
`choice:11+` is the only bucket the clamp moves and every case here is a 20-option
question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at
all: a temperature-scaled softmax has the same argmax at every positive temperature.

Worth recording because it cuts against the reading that the clamp only distorts the
published number: `acc_at_50_coverage` is the one rank-quality column that consumes
the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98.

The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the
committed columns stopped matching the unclamped re-run or if the clamped re-run
started matching them.

Deliberate choices tested:

- `bench_local.py` was not modified. It produced the committed file, and rewriting it
  to reproduce its own pre-clamp output would make the old and new runs come from
  different scripts. `laya_eval.py` already has `--unclamped` for this, and the
  three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are
  recorded in the new file rather than quietly dropped.
- The clamped re-run is committed alongside rather than replacing anything, so
  `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite.
- `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is
  wired into both lanes, and this suite is currently in neither; adding it here
  would collide with that PR and with the verbatim `test` job in NandhaKishorM#212.

Known limitation: only the `english` checkpoint was re-run. The committed file's
`part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks
are in `part_a`, so the multilingual half is untouched and unmeasured here.

Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.
Three more candidates for the unreconciled multilingual row are now ruled out on
re-measurement rather than by inspection: the shipped head_max_len (256, read from
the checkpoint's own config and recorded in the run), which of the two multilingual
copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51),
and a checkpoint change since the committed sweep (weights and config are identical by
size and hash at every revision in that window).

Also corrects the test count in the same file, which the new NandhaKishorM#208 guard changes.
@PerryLink
PerryLink force-pushed the research-208-temperature-clamp-both-regimes branch from 3fd1453 to 5719d6a Compare September 23, 2026 13:30
@PerryLink

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (885ba788), so mergeable is MERGEABLE again.

One conflict, in research/eval/README.md, and it was with my own #232:

  • #232 corrected the confidence == 0.0 bullet to say the harness now bins the way laya.common.ece_score does. This PR was branched before that, so its copy of the bullet still said "is not binned". I kept main's corrected bullet and dropped the stale copy; the new "temperature clamp, measured both ways" section is unchanged.
  • The test count in the same file needed a decision rather than a merge: #232 left it at 49 and this PR had written 62. This branch adds the The committed 51-language sweep's ECE and confidence columns predate the #42 temperature clamp and no longer reproduce #208 guard checks on top, so the real number on the rebased tree is 64, verified by running it rather than by adding the two.

Re-verified after the rebase:

$ python research/eval/test_laya_eval.py
64 passed, 0 failed

# the committed artifact still reconciles against the committed sweep
$ python -  # reads cpu_51_language_sweep_clamped.json and the committed sweep
languages: 51 | n_cases: 5100
ece    unclamped_rerun   identical 48/51
ece    clamped_rerun     identical  0/51
mean_confidence unclamped_rerun identical 49/51
accuracy        unclamped_rerun identical 51/51
macro_f1        unclamped_rerun identical 51/51

So the PR description's 48/51 and 49/51 are right. I checked this again because a quick re-check appeared to give 51/51 for ece, and that turned out to be my own tolerance being too loose for a figure stored to four decimals: am, el and ro differ by 0.0001, which a 1.5e-4 threshold swallows but a 5e-5 one does not. The numbers above use 5e-5, and the three-way split is the same one the description already reports.

tests (windows) is now a required check on this repository, so this PR runs it too; it comes from #212, which landed today.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for doing the #208 re-run, and for keeping the original file beside the new one so the before/after stays visible. The follow-up showing the clamp is a no-op on laya-multilingual (it ships no fitted temperatures) and eliminating the head_max_len hypothesis makes the record complete. After #232 changed the same README, I merged main into your branch, which went through cleanly. Merging.

@NandhaKishorM
NandhaKishorM merged commit fcf1d7f into NandhaKishorM:main Sep 23, 2026
14 checks passed
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#208 ask) (NandhaKishorM#222)

* research: re-run the 51-language sweep in both temperature regimes (the NandhaKishorM#208 ask)

The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from
before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer
reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so
the before/after is visible.

`research/results/cpu_51_language_sweep_clamped.json` holds the committed columns,
an `--unclamped` re-run and a default re-run side by side, per language, from the
same 51 languages and 5,100 cases.

    committed      unclamped re-run   clamped re-run
    0.2269         0.2269             0.2269     macro accuracy
    0.7331         0.7331             0.5709     macro ECE
    0.2053         0.2053             0.2053     macro F1

Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the
committed macro ECE at 0.7331, so the clamp is the only variable left. Per language
the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in
51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do
so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and
this one 2.14.0.

The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower.
`choice:11+` is the only bucket the clamp moves and every case here is a 20-option
question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at
all: a temperature-scaled softmax has the same argmax at every positive temperature.

Worth recording because it cuts against the reading that the clamp only distorts the
published number: `acc_at_50_coverage` is the one rank-quality column that consumes
the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98.

The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the
committed columns stopped matching the unclamped re-run or if the clamped re-run
started matching them.

Deliberate choices tested:

- `bench_local.py` was not modified. It produced the committed file, and rewriting it
  to reproduce its own pre-clamp output would make the old and new runs come from
  different scripts. `laya_eval.py` already has `--unclamped` for this, and the
  three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are
  recorded in the new file rather than quietly dropped.
- The clamped re-run is committed alongside rather than replacing anything, so
  `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite.
- `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is
  wired into both lanes, and this suite is currently in neither; adding it here
  would collide with that PR and with the verbatim `test` job in NandhaKishorM#212.

Known limitation: only the `english` checkpoint was re-run. The committed file's
`part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks
are in `part_a`, so the multilingual half is untouched and unmeasured here.

Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.

* research: record the multilingual eliminations for the committed-row gap

Three more candidates for the unreconciled multilingual row are now ruled out on
re-measurement rather than by inspection: the shipped head_max_len (256, read from
the checkpoint's own config and recorded in the run), which of the two multilingual
copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51),
and a checkpoint change since the committed sweep (weights and config are identical by
size and hash at every revision in that window).

Also corrects the test count in the same file, which the new NandhaKishorM#208 guard changes.

---------

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#208)

Implements the refresh the maintainer asked for in
[NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."*

## What did not reproduce

The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records
`laya-multilingual` at **0.3661** macro accuracy. Four independent runs on three machines do not
reproduce it:

| run | macro accuracy | per-language accuracy matching the committed file |
|---|---|---|
| committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — |
| laya 0.2.0, x86 CPU (this PR's author) | **0.4008** | 6 / 51 |
| laya 0.3.7, x86 CPU (this PR's author) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 |

The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes
this a property of the multilingual columns rather than of the harness or the machine.

## What this PR does

**`research/results/cpu_51_language_sweep_refreshed.json`** (new). The refreshed numbers with
the environment recorded next to them, in the shape of the file it replaces — `meta` carries
laya, torch, transformers, CPU, platform, python, harness, seed, n_options, per_lang and
dataset — and every per-language entry keeps the superseded committed row beside the new one so
the before/after stays visible:

```json
"af": {
  "committed":         {"accuracy": 0.35, "ece": 0.4845, "mean_confidence": 0.8028, ...},
  "refreshed_clamped": {"accuracy": 0.35, "ece": 0.4394, "mean_confidence": 0.7894, ...},
  "refreshed_unclamped": { ... }
}
```

Both regimes are kept for the same reason `NandhaKishorM#222` kept them: `laya-multilingual` ships no fitted
temperatures, so the clamp cannot move it, and the two columns being byte-identical is the
evidence for that claim rather than an assertion of it.

**The macros, refreshed:**

| | committed | refreshed |
|---|---|---|
| `laya-multilingual` macro accuracy | 0.3661 | **0.4008** |
| `laya-multilingual` macro ECE | 0.3869 | **0.3911** |
| `laya-multilingual` clearing 3x random | 45 / 51 | **48 / 51** |
| `laya` macro accuracy / ECE | 0.2269 / 0.7331 | 0.2269 / **0.5709** |

**`BENCHMARKS.md`.** The 51-language table is regenerated from the artifact rather than
hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain,
so `en` moves to the bottom where the gain is negative) — and a note above it records why the
multilingual columns are a re-run, keeping the committed row in the same table for comparison.

**`README.md`.** The stale `45 of 51` in the Jev comparison becomes **48 of 51**, and the
multilingual section now states both checkpoints' macros with the ECE quoted twice: `0.733` as
committed and `0.571` as served, because the committed figure predates the temperature clamp.

**`tests/test_doc_tables.py`.** The table is now checkable, so it is checked: every cell of every
row against the refreshed artifact, that the `laya` column still equals the committed file
exactly, and that the multilingual column differs from it in more than 40 languages. A future
edit that quietly restores a committed number, or a future refresh that forgets to update a row,
fails the suite. 54 → 59 checks.

Deliberate choices tested:

- **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19, and overwriting it would erase the only evidence that the multilingual columns used to read differently. The refreshed file carries the old rows inline, so nothing is lost either way, and the table cites the artifact it now prints.
- **The environment goes in the file, not only in the PR.** `laya` 0.3.20 / torch 2.14.0+cpu / transformers 5.17.0 / the CPU string / the platform, plus `why` naming NandhaKishorM#208. The maintainer asked for the environment next to the numbers, and a reader of the JSON alone should not have to find this PR.
- **`laya`'s column is refreshed too, even though nothing changed.** It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that. A refresh that touched only one column would leave the other unverifiable.
- **The `45 of 51` in the README is a real correction, not a rounding.** It is the "Languages usable" row of the Jev comparison table and it described the committed multilingual run. 48 is the refreshed count by the same criterion (>3x random).

```
$ python tests/test_doc_tables.py              59 passed, 0 failed   (54 before)
$ python tests/test_packaging.py               all packaging tests passed
$ python -m pytest tests/test_serve.py tests/test_router_batch.py \
                   tests/test_audit_regressions.py -q
50 passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: this does not explain *why* the committed multilingual run reads 0.3661. Four
machines agree it does not come back, and the direction is consistent rather than noisy — of the
16 languages that move by 0.05 or more, every one moves up — but no cause is established here,
and the committed file is left in place rather than deleted so the discrepancy stays visible. If
the intent is that the committed columns are simply wrong, deleting them is a separate decision.

Credit to @aashish254, whose M5 run on both devices is what turned this from one machine's
disagreement into a settled finding.
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#208)

Implements the refresh the maintainer asked for in
[NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."*

## Why the multilingual columns are a re-run

The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records
`laya-multilingual` at **0.3661** macro accuracy. It does not reproduce:

| run | macro accuracy | per-language accuracy matching the committed file |
|---|---|---|
| committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — |
| laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 |

The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes
this a property of the multilingual columns rather than of the harness or the machine.

## What this PR does

**`research/results/cpu_51_language_sweep_refreshed.json`** (new). Every per-language entry keeps
the superseded committed row beside the new one so the before/after stays visible, and both
temperature regimes are kept — `laya-multilingual` ships no fitted temperatures, so the two
columns being byte-identical is the evidence for that rather than an assertion of it.

**`provenance` records where each half came from, per model.** The two halves are not from the
same run and the first version of this file did not say so:

```json
"english":      {"source": "cpu_51_language_sweep_clamped.json (PR NandhaKishorM#222, field `clamped_rerun`)",
                 "laya_version": "0.3.7", "max_len": 512,  "freshly_run_for_this_file": false,
                 "note": "identical to the NandhaKishorM#222 english re-run, which is why the `laya` column
                          matches the committed sweep on all 51 languages; the recorded `seconds`
                          belong to that run"},
"multilingual": {"source": "a fresh run on the machine described in `meta`",
                 "laya_version": "0.3.7", "max_len": 1024, "freshly_run_for_this_file": true}
```

I checked this rather than asserting it: the English column is compared against NandhaKishorM#222's
`clamped_rerun` field for all 51 languages inside the build, and the script fails if they
diverge (`af` ECE 0.5588, mean confidence 0.8488 in both). **`meta.laya_version` was also wrong
in the first version** — it said 0.3.20 while both halves are 0.3.7, because I read the version
of the environment rather than the version recorded in the run's own `config`. That field is now
replaced by the per-model `provenance.laya_version` above, taken from each run's `config`.

**`research/README.md`**, which the refresh had missed entirely:

- `45 of 51` → **`48 of 51`** and `0.366` → **`0.401`**, with a pointer to the refreshed file, so
  the headline finding no longer contradicts the table it links to;
- Khmer's `95.2%` is now labelled as the **raw** temperature, with **70.5%** as served after the
  clamp at the same 0.000 accuracy. Both numbers were in the data and only the raw one was
  quoted;
- the `cpu_51_language_sweep_clamped.json` row now attributes that file to NandhaKishorM#222 and its English
  half, and credits NandhaKishorM#208 for the multilingual re-run, since only one of the two is a re-run of
  the whole sweep.

**`BENCHMARKS.md`.** The 51-language table is regenerated from the artifact rather than
hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so
`en` moves to the bottom where the gain is negative) — and a note above it records why the
multilingual columns are a re-run, keeping the committed row in the same table for comparison.

**`README.md`.** The stale `45 of 51` in the Jev comparison becomes **48 of 51**, and the
multilingual section states both checkpoints' macros with ECE quoted twice: `0.733` as committed
and `0.571` as served, because the committed figure predates the temperature clamp.

**`tests/test_doc_tables.py`.** The table is now checkable, so it is checked: every cell of every
row against the refreshed artifact, that the `laya` column still equals the committed file
exactly, and that the multilingual column differs from it in more than 40 languages. A future
edit that quietly restores a committed number, or a refresh that forgets a row, fails the suite.
54 → 59 checks.

Deliberate choices tested:

- **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently. The refreshed file carries the old rows inline, so nothing is lost either way.
- **Both halves in one file, with provenance, rather than two files.** A reader of the table needs one artifact behind it, and the interesting fact is precisely that the two halves have different histories — which is only visible if they sit in the same file with their sources labelled.
- **The environment goes in the file, not only in the PR.** laya, torch, transformers, CPU, platform, python, harness, seed, n_options, per_lang, dataset, split and instructions, plus `why` naming NandhaKishorM#208, and now per-model provenance. The maintainer asked for the environment next to the numbers, and a reader of the JSON alone should not have to find this PR.
- **`laya`'s column is refreshed too, even though nothing changed.** It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that.
- **`45 of 51` → `48 of 51` is a real correction, not a rounding.** It is the "Languages usable" row of the Jev comparison table and it described the committed multilingual run.

```
$ python tests/test_doc_tables.py              59 passed, 0 failed   (54 before)
$ python tests/test_packaging.py               all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

## The gap is quantified but not explained

The multilingual shift is not sampling noise. Each committed accuracy is a count out of 100, so
one language moving 0.05 proves nothing; the direction across all 51 does. The multilingual mean
moves **+0.0347** (median +0.0300, sd 0.0423) — **3.8 pooled standard errors** — while the `laya`
column of the same file reproduces on 51 of 51 with sd 0.0000. `mean_confidence` moves with it
(+0.0369, up in 47 of 51, only weakly correlated at r = 0.349), and the shift is largest where
the committed run was worst (+0.0506 hardest third, +0.0182 easiest).

Also ruled out by measurement rather than argument:

| candidate | result |
|---|---|
| smaller effective token budget | `max_len` 1024/768/512/384/256 give bit-identical accuracy on the ten worst languages (`bn` 0.45 at all five) |
| the temperature clamp | `laya-multilingual` ships no fitted temperatures, so both regimes are byte-identical |
| the checkpoint | same `model.safetensors` blob at every revision in the Hub history |
| sampling parameters, harness, layout | `laya` reproduces exactly under all of them, in the same run |

The one variable from the committed run not reproduced is `torch 2.8.0` (this box runs
2.14.0+cpu). I do not think it is worth installing: a runtime numerics change would have to move
the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English
checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or
reduction-order difference. Stated rather than left implicit.

The committed file is left in place rather than deleted, so the discrepancy stays visible.

Credit to @aashish254, whose M5 run on both devices turned this from one machine's disagreement
into a settled finding.
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#208)

Implements the refresh the maintainer asked for in
[NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."*

**Both halves are now one run at one version.** The first version of this file took the English
column from the NandhaKishorM#222 re-run at laya 0.3.7, and the review asked me to record that or re-run it
here. I re-ran both halves on this machine at **laya 0.3.20**, and separately checked that the
choice does not matter:

```
english clamped          0.3.7 == 0.3.20 on all 51 languages
english unclamped        0.3.7 == 0.3.20 on all 51 languages
multilingual clamped     0.3.7 == 0.3.20 on all 51 languages
multilingual unclamped   0.3.7 == 0.3.20 on all 51 languages
```

Every comparison above is asserted inside the build script over accuracy, ECE, macro-F1 and mean
confidence, and the script fails if any language moves. So the artifact carries one version, and
`provenance` records per model where each half came from, as asked.

## Why the multilingual columns are a re-run

The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records
`laya-multilingual` at **0.3661** macro accuracy. It does not reproduce:

| run | macro accuracy | per-language accuracy matching the committed file |
|---|---|---|
| committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — |
| laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 |
| **laya 0.3.20, x86 CPU** | **0.4008** | **6 / 51** |
| laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 |

The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes
this a property of the multilingual columns rather than of the harness or the machine.

## What this PR does

**`research/results/cpu_51_language_sweep_refreshed.json`** (new). Every per-language entry keeps
the superseded committed row beside the new one so the before/after stays visible, and both
temperature regimes are kept — `laya-multilingual` ships no fitted temperatures, so the two
columns being byte-identical is the evidence for that rather than an assertion of it. `meta`
carries laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options,
per_lang, dataset, split and instructions — a strict superset of what the committed file
recorded — and `provenance` names each half's source.

**`research/README.md`**, which the refresh had missed entirely:

- `45 of 51` → **`48 of 51`** and `0.366` → **`0.401`**, with a pointer to the refreshed file, so
  the headline finding no longer contradicts the table it links to;
- Khmer's `95.2%` is now labelled as the **raw** temperature, with **70.5%** as served after the
  clamp at the same 0.000 accuracy;
- the `cpu_51_language_sweep_clamped.json` row now attributes that file to NandhaKishorM#222 and its English
  half, and credits NandhaKishorM#208 for the multilingual re-run.

**`BENCHMARKS.md`.** The 51-language table is regenerated from the artifact rather than
hand-edited — all 51 per-language rows, both ECE columns and the sort order (by routing gain, so
`en` moves to the bottom where the gain is negative) — and a note above it records why the
multilingual columns are a re-run, keeping the committed row in the same table for comparison.

**`README.md`.** The stale `45 of 51` in the Jev comparison becomes **48 of 51**, and the
multilingual section states both checkpoints' macros with ECE quoted twice: `0.733` as committed
and `0.571` as served, because the committed figure predates the temperature clamp.

**`tests/test_doc_tables.py`.** The table is now checkable, so it is checked: every cell of every
row against the refreshed artifact, that the `laya` column still equals the committed file
exactly, and that the multilingual column differs from it in more than 40 languages. 54 → 59
checks.

Deliberate choices tested:

- **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently.
- **Both halves re-run rather than labelled as mixed.** Recording "English from NandhaKishorM#222 at 0.3.7" would have been honest but leaves a reader unable to compare the two columns as a pair; running both here and asserting the versions agree gives one version and no caveat.
- **The environment goes in the file, not only in the PR**, and is a superset of the committed file's `meta`, so a reader of the JSON alone does not have to find this PR.
- **`laya`'s column is refreshed too, even though nothing changed.** It reproduces the committed file exactly, and printing it from the same artifact is what lets the new test assert that.

```
$ python tests/test_doc_tables.py              59 passed, 0 failed   (54 before)
$ python tests/test_packaging.py               all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

## The gap is quantified but not explained

The multilingual shift is not sampling noise. Each committed accuracy is a count out of 100, so
one language moving 0.05 proves nothing; the direction across all 51 does. The multilingual mean
moves **+0.0347** (median +0.0300, sd 0.0423) — **3.8 pooled standard errors** — while the `laya`
column of the same file reproduces on 51 of 51 with sd 0.0000. `mean_confidence` moves with it
(+0.0369, up in 47 of 51, only weakly correlated at r = 0.349), and the shift is largest where
the committed run was worst (+0.0506 hardest third, +0.0182 easiest).

Also ruled out by measurement rather than argument:

| candidate | result |
|---|---|
| the laya version | **0.2.0, 0.3.7 and 0.3.20 all give 0.4008**, asserted per language in the build |
| smaller effective token budget | `max_len` 1024/768/512/384/256 give bit-identical accuracy on the ten worst languages (`bn` 0.45 at all five) |
| the temperature clamp | `laya-multilingual` ships no fitted temperatures, so both regimes are byte-identical |
| the checkpoint | same `model.safetensors` blob at every revision in the Hub history |
| sampling parameters, harness, layout | `laya` reproduces exactly under all of them, in the same run |

The one variable from the committed run not reproduced is `torch 2.8.0` (this box runs
2.14.0+cpu). I do not think it is worth installing: a runtime numerics change would have to move
the multilingual checkpoint by +0.035 uniformly across 51 languages while leaving the English
checkpoint on the same 51 states bit-identical, which is not a plausible shape for a kernel or
reduction-order difference.

The committed file is left in place rather than deleted, so the discrepancy stays visible.

Credit to @aashish254, whose M5 run on both devices turned this from one machine's disagreement
into a settled finding.
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 25, 2026
NandhaKishorM#208)

Implements the refresh the maintainer asked for in
[NandhaKishorM#208](NandhaKishorM#208): *"A PR that refreshes that JSON and
the table from this re-run, with the environment recorded next to it, would be very welcome."*

**Both halves are one run at one version** — a fresh 0.3.20 run on this machine — and the
per-model `provenance` block records where each came from, as the review asked.

## Why the multilingual columns are a re-run

The committed `cpu_51_language_sweep.json` was measured at laya 0.2.0 and records
`laya-multilingual` at **0.3661** macro accuracy. It does not reproduce on any setup tried:

| run | macro accuracy | per-language accuracy matching the committed file |
|---|---|---|
| committed, laya 0.2.0, torch 2.8.0 | 0.3661 | — |
| laya 0.2.0, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.7, x86 CPU | **0.4008** | 6 / 51 |
| laya 0.3.20, x86 CPU (this file) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, CPU ([@aashish254](https://github.com/aashish254)) | **0.4008** | 6 / 51 |
| laya 0.3.11, Apple M5, MPS | **0.4008** | 6 / 51 |

The `laya` half of the same file reproduces **51/51 exactly** on all of them, which is what makes
this a property of the multilingual columns rather than of the harness or the machine.

## Stale figures in files this PR touches

The review listed four still quoting the raw temperature or the pre-refresh count; all four now
match the table, with the raw and served values both given:

| file | was | now |
|---|---|---|
| `README.md` Router table | `45 / 51` twice | `48 / 51`, the refreshed count |
| `README.md` limits note | Khmer `0.952` unlabelled | `0.952` raw, `0.705` as served |
| `README.md` multilingual section | Khmer `95.2%` | `95.2%` raw, `70.5%` as served |
| `BENCHMARKS.md` | `0.952` and "never below `0.885`" | raw, with `0.705` and `0.621` as served |

The served figures are the artifact's own: Khmer `0.9515` raw → `0.7051` served, and the minimum
per-language mean confidence `0.8848` raw → `0.6209` served.

## What this PR does

- **`research/results/cpu_51_language_sweep_refreshed.json`** (new): every per-language entry keeps the superseded committed row beside the new one, both temperature regimes are kept (`laya-multilingual` ships no fitted temperatures, so the two being byte-identical is the evidence for that), and `meta` is a strict superset of the committed file's — laya, torch, transformers, CPU, platform, python, threads, harness, seed, n_options, per_lang, dataset, split, instructions.
- **`BENCHMARKS.md`**: the 51-language table is regenerated from the artifact rather than hand-edited — all 51 rows, both ECE columns and the sort order — with a note above it explaining why the multilingual columns are a re-run and keeping the committed row for comparison.
- **`README.md`**: the four figures above.
- **`research/README.md`**: `45 of 51` → `48 of 51`, `0.366` → `0.401`, Khmer labelled raw, and the re-run rows attributed to NandhaKishorM#222 (English half) and NandhaKishorM#208 (multilingual).
- **`tests/test_doc_tables.py`**: every cell of every row is checked against the artifact, the `laya` column must still equal the committed file exactly, and the multilingual column must differ from it in more than 40 languages. 54 → 59 checks on the branch that added it, and the rebase kept both this block and main's new fp16-table block (114 checks together).

Deliberate choices tested:

- **Refresh into a new file rather than overwriting the committed one.** The committed JSON is the record of what was run on 2026-09-19; overwriting it would erase the only evidence that the multilingual columns once read differently.
- **Both halves re-run rather than labelled as mixed.** Recording "English from NandhaKishorM#222 at 0.3.7" would have been honest but leaves a reader unable to compare the columns as a pair. Both were re-run here, and the equivalence at 0.3.7 vs 0.3.20 is asserted per language inside the build script over accuracy, ECE, macro-F1 and mean confidence.
- **Raw *and* served everywhere the clamp moves a number**, rather than switching to served only — the raw figure is what the committed sweep records, so dropping it would break the comparison the file exists to make.

```
$ python tests/test_doc_tables.py              114 passed, 0 failed
$ python tests/test_packaging.py               all packaging tests passed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

## The gap is quantified but not explained

The shift is not sampling noise: the multilingual mean moves **+0.0347** (3.8 pooled standard
errors) while the `laya` column reproduces on 51 of 51 with sd 0.0000, and `mean_confidence`
moves with it (+0.0369, up in 47 of 51, correlated only r = 0.349). Ruled out by measurement: the
laya version (all three give 0.4008), the token budget (`max_len` 1024→256 bit-identical on the
ten worst languages), the temperature clamp (no fitted temperatures), the checkpoint (same
`model.safetensors` blob at every Hub revision), and sampling/harness/layout (`laya` reproduces
exactly under all of them).

The one variable not reproduced is `torch 2.8.0` (this box runs 2.14.0+cpu). A runtime numerics
change would have to move the multilingual checkpoint by +0.035 uniformly across 51 languages
while leaving the English checkpoint on the same 51 states bit-identical, which is not a plausible
shape for a kernel or reduction-order difference.

The committed file is left in place rather than deleted, so the discrepancy stays visible.

Credit to @aashish254, whose M5 runs on both devices turned this from one machine's disagreement
into a settled finding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants