Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,19 @@ Every checkpoint answered **byte-identical questions** in each run (fixed seed).
| Applications | the seven workflow themes + the datasets where Jev numbers exist, all three checkpoints (laya 0.2.1, CPU, 400 cases per task, seed 13, 2026-09-19) | `research/results/app_benchmark_results.json` |


**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets (`choice:11+` is now served at 0.5, not 0.1006). Accuracy columns are unaffected. A re-run with the current package is tracked in #208.
**Calibration columns in the CPU sweep predate the temperature clamp.** The 51-language ECE and mean-confidence figures were produced before #42 clamped temperatures to `[0.5, 5]`, so today's package reports different confidence for the affected buckets. Accuracy columns are unaffected, because a temperature-scaled softmax has the same argmax at every positive temperature.

The same 51 languages and 5,100 cases have now been re-run with 0.3.7 in both regimes (`research/results/cpu_51_language_sweep_clamped.json`, [#208](https://github.com/NandhaKishorM/laya/issues/208)). Macro accuracy reproduces at **0.2269** exactly, and macro ECE moves **0.7331 → 0.5709**:

| | committed | re-run, raw temperatures | re-run, as served |
|---|---|---|---|
| macro accuracy | 0.2269 | 0.2269 | 0.2269 |
| macro ECE | 0.7331 | **0.7331** | 0.5709 |
| macro F1 | 0.2053 | 0.2053 | 0.2053 |
| mean confidence, `en` | 0.9989 | **0.9989** | 0.9582 |
| ECE, `en` | 0.1789 | **0.1789** | 0.1382 |

The raw-temperature column reproduces the committed file, so the only variable left is the clamp. `choice:11+` is the sole bucket it moves, and every case in this sweep is a 20-option question, so the clamp applies to all 5,100 — and lowers ECE in all 51 languages. `acc_at_50_coverage` is the one rank-quality column that uses the confidence values: macro 0.3004 → 0.3020, and `en` 0.94 → 0.98, so the flatter distribution selects a slightly better half rather than a worse one.

---

Expand Down
1 change: 1 addition & 0 deletions research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ installed its abseil runtime can deadlock model construction on macOS/Python 3.9
|---|---|
| `results/t4_colab_benchmark.json` | 17,416 questions on one T4, both checkpoints, identical questions per model |
| `results/cpu_51_language_sweep.json` | 51 languages x 2 checkpoints, MASSIVE intent, 20 options |
| `results/cpu_51_language_sweep_clamped.json` | the same 51 languages and 5,100 cases re-run with 0.3.7, raw temperatures and served temperatures side by side ([#208](https://github.com/NandhaKishorM/laya/issues/208)) |

## Headline findings

Expand Down
44 changes: 41 additions & 3 deletions research/eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,24 @@ having an empty `temperature_by_options`, so the clamp cannot explain it.
Ruled out: the option sets (identical digest to the english run), the weights
(bundled and standalone multilingual are byte-identical, all 170 tensors
`torch.equal`), the dataset (revision `940fd47a`, last modified 2026-02-24), and
`build_sequence` (unchanged since `v0.2.0`). It is in the multilingual inference path
between `laya 0.2.0` and `0.3.6` and is **not** reconciled. Flagged rather than hidden.
`build_sequence` (unchanged since `v0.2.0`). Also ruled out, on re-measurement:

* **the shipped `head_max_len`**, which matters here because this checkpoint ships
`256` and english ships `192`. The harness reads it from the checkpoint's own
config and the run's `config` block records `head_max_len: 256, max_len: 1024`, so
the multilingual numbers above were not taken at english's budget. Re-running with
the value read from config gives the same `0.4008`, and `6/51` again.
* **which of the two multilingual copies was measured.** The bundled `multilingual/`
subfolder and the standalone `convaiinnovations/laya-multilingual` repo were each
run end to end over all 51 languages and both give `macro_accuracy 0.4008`,
`macro_ece 0.3911`, `6/51`.
* **a checkpoint change since the committed sweep.** `multilingual/model.safetensors`
is `643835514` bytes at `sha256 b99c8bea…` and `multilingual/rl_agent_config.json`
is `472` bytes at `sha256 00e35f88…` at every revision from the sweep's timestamp to
today; the Hub commits in that window are model-card `docs:`/`assets:` only.

It is in the multilingual inference path between `laya 0.2.0` and `0.3.6` and is
**not** reconciled. Flagged rather than hidden.

Related: **`head_max_len` is load-bearing for accuracy**, not just for option
truncation. The english checkpoint at its shipped `head_max_len=192` scores 0.82;
Expand All @@ -150,7 +166,7 @@ forcing 256 or 512 drops it to 0.79.
checkpoint, no network:

```bash
python research/eval/test_laya_eval.py # 49 passed, 0 failed
python research/eval/test_laya_eval.py # 64 passed, 0 failed
```

It pins the upstream constants (seed 13, 20 options, the exact instruction string),
Expand All @@ -176,3 +192,25 @@ not just against this harness's own arithmetic.
binned differently. It now matches `laya.common.ece_score`,
`research/scripts/bench_local.py` and `research/scripts/build_benchmark_nb.py`, and
`test_laya_eval.py` asserts that agreement.

### The temperature clamp, measured both ways

`research/results/cpu_51_language_sweep_clamped.json` carries the same re-run twice, once per
regime, against the committed columns. Macro accuracy reproduces the committed file exactly
and macro ECE is the only macro figure that moves:

| | committed | `--unclamped` | default |
|---|---|---|---|
| `macro_accuracy` | 0.2269 | **0.2269** | 0.2269 |
| `macro_ece` | 0.7331 | **0.7331** | 0.5709 |
| `macro_f1` | 0.2053 | **0.2053** | 0.2053 |

Per language, the unclamped run agrees with the committed file on `accuracy` and `macro_f1`
in **51/51**, on `ece` in **48/51** and on `mean_confidence` in **49/51**. The handful that
differ do so by `0.0001`, the last stored digit: the committed run used torch 2.8.0 and this
one 2.14.0. The clamped run differs from the committed file on `ece` and `mean_confidence` in
**51/51**, every one of them lower, because it is the only column the clamp can move.

`accuracy`, `macro_f1` and `n` are identical in all three columns by construction: scaling
logits by any positive temperature does not change the argmax. That is why a re-run can settle
the calibration question without reopening the accuracy numbers.
84 changes: 84 additions & 0 deletions research/eval/test_laya_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,90 @@ class _FakeAgent:
INSTRUCTIONS, "What is the user asking for in `utterance`?")


# ------------------------------------------- the #208 before/after re-run file
# research/results/cpu_51_language_sweep_clamped.json records the committed sweep,
# the pre-clamp re-run and the served-temperature re-run side by side. It is only
# useful if it still agrees with the committed file, so that agreement is a test.
import json # noqa: E402

_RESULTS = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))),
"research", "results")


def _load(name):
with open(os.path.join(_RESULTS, name), encoding="utf-8") as fh:
return json.load(fh)


_rerun = _load("cpu_51_language_sweep_clamped.json")
_sweep = _load("cpu_51_language_sweep.json")["part_a"]["by_model"]["english"]
_langs = _sweep["per_language"]

# Both files round each per-language figure to 4 decimals (bench_local.py:138-144), so
# agreement has to be judged at that resolution: two files can disagree by 1 in the
# last stored digit for reasons that have nothing to do with the temperatures, and
# three languages do (am, el, ro) because the committed run used torch 2.8.0 and this
# one 2.14.0. The macro figures match exactly, which is why that is the headline.
_QUANT = 1.5e-4


def _close(a, b, tol=_QUANT):
return abs(a - b) <= tol

check("208/one entry per language", len(_rerun["per_language"]), len(_langs))
check("208/case count is languages x per_lang",
_rerun["config"]["n_cases"],
_rerun["config"]["languages"] * _rerun["config"]["per_lang"])
check("208/macro accuracy copied from the committed sweep",
_rerun["macro"]["committed"]["accuracy"], _sweep["macro_accuracy"])
check("208/macro ece copied from the committed sweep",
_rerun["macro"]["committed"]["ece"], _sweep["macro_ece"])

# accuracy is argmax of a temperature-scaled softmax, so it cannot move with T
check_true("208/accuracy identical in all three regimes",
all(len({_rerun["per_language"][lg][r]["accuracy"] for r in
("committed", "unclamped_rerun", "clamped_rerun")}) == 1
for lg in _langs))
check_true("208/macro_f1 identical in all three regimes",
all(len({_rerun["per_language"][lg][r]["macro_f1"] for r in
("committed", "unclamped_rerun", "clamped_rerun")}) == 1
for lg in _langs))

# the committed calibration columns must match the unclamped re-run, and only the
# clamped re-run may differ -- that is the whole claim
check_true("208/ece matches the committed file in the unclamped re-run",
all(_close(_rerun["per_language"][lg]["unclamped_rerun"]["ece"],
_langs[lg]["ece"]) for lg in _langs))
check_true("208/ece differs from the committed file in the clamped re-run",
all(not _close(_rerun["per_language"][lg]["clamped_rerun"]["ece"],
_langs[lg]["ece"]) for lg in _langs))
check_true("208/mean_confidence matches in the unclamped re-run",
all(_close(_rerun["per_language"][lg]["unclamped_rerun"]["mean_confidence"],
_langs[lg]["mean_confidence"]) for lg in _langs))
check_true("208/mean_confidence differs in the clamped re-run",
all(not _close(_rerun["per_language"][lg]["clamped_rerun"]["mean_confidence"],
_langs[lg]["mean_confidence"]) for lg in _langs))

# the unclamped re-run is a reproduction, not an approximation: name the tolerance
# so a future change that widens it has to say so
check_true("208/unclamped reproduction agrees in at least 48 of 51 languages",
sum(1 for lg in _langs
if _close(_rerun["per_language"][lg]["unclamped_rerun"]["ece"], _langs[lg]["ece"])) >= 48)

# the clamp can lower ECE everywhere without lowering rank quality, so this is a
# guard against the misleading "the clamp makes it worse" reading
check_true("208/the clamp lowers macro ece",
_rerun["macro"]["clamped_rerun"]["macro_ece"]
< _rerun["macro"]["unclamped_rerun"]["macro_ece"])
check_true("208/every per-language delta is reported",
all("delta_ece" in v and "delta_mean_confidence" in v
for v in _rerun["per_language"].values()))
check("208/only choice:11+ is the clamped bucket",
_rerun["temperature_choice_11_plus"]["unclamped_rerun"], 0.10058280825614929)
check("208/the served bucket is 0.5",
_rerun["temperature_choice_11_plus"]["clamped_rerun"], 0.5)


print("\n%d passed, %d failed" % (len(PASS), len(FAIL)))
for f in FAIL:
print(" FAIL " + f)
Expand Down
Loading
Loading