Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions research/eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,13 +150,15 @@ forcing 256 or 512 drops it to 0.79.
checkpoint, no network:

```bash
python research/eval/test_laya_eval.py # 47 passed, 0 failed
python research/eval/test_laya_eval.py # 49 passed, 0 failed
```

It pins the upstream constants (seed 13, 20 options, the exact instruction string),
the determinism of the sampler, that a fresh RNG per language is used, and the
metric arithmetic, including the `confidence == 0.0` bin boundary that this harness
shares with `bench_local.py` and `laya.common.ece_score`.
shares with `laya.common.ece_score`, `research/scripts/bench_local.py` and
`research/scripts/build_benchmark_nb.py`. That boundary is asserted against all four,
not just against this harness's own arithmetic.

## Limits

Expand All @@ -167,6 +169,10 @@ shares with `bench_local.py` and `laya.common.ece_score`.
* The English checkpoint collapses on non-Latin scripts (see `BENCHMARKS.md`), so a
low score in one language is not by itself evidence of a misroute — check
`laya.lang.analyse` for the script before concluding which checkpoint was used.
* `confidence == 0.0` is not binned, matching `laya.common.ece_score` and
`bench_local.py`. If [#39](https://github.com/NandhaKishorM/laya/pull/39) lands
with a different boundary, this should follow it.
* The `confidence == 0.0` bin boundary is the one
[#39](https://github.com/NandhaKishorM/laya/pull/39) settled: the first bin is closed
at the bottom, so `0.0` is counted. This harness used `conf > lo` for every bin until
the divergence was found, which made it the only one of the four implementations that
binned differently. It now matches `laya.common.ece_score`,
`research/scripts/bench_local.py` and `research/scripts/build_benchmark_nb.py`, and
`test_laya_eval.py` asserts that agreement.
13 changes: 10 additions & 3 deletions research/eval/laya_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,15 +175,22 @@ def temperature_for(agent, qtype: int, k: int, unclamped: bool = False) -> float


def ece(confidence, correct, bins: int = ECE_BINS) -> float:
"""Expected calibration error over equal-width confidence bins."""
"""Expected calibration error over equal-width confidence bins.

The first bin is closed at the bottom (`>= lo`), so `confidence == 0.0` is counted.
That is the boundary #39 settled in `laya.common.ece_score`,
`research/scripts/bench_local.py` and `research/scripts/build_benchmark_nb.py`. This
harness kept the pre-#39 test until the divergence was found, which made it the only
one of the four implementations that binned differently.
"""
import numpy as np
confidence = np.asarray(confidence, dtype=float)
correct = np.asarray(correct, dtype=float)
if not len(confidence):
return float("nan")
total, edges = 0.0, np.linspace(0.0, 1.0, bins + 1)
for lo, hi in zip(edges[:-1], edges[1:]):
sel = (confidence > lo) & (confidence <= hi)
for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi)
if sel.any():
total += sel.mean() * abs(confidence[sel].mean() - correct[sel].mean())
return float(total)
Expand Down
18 changes: 11 additions & 7 deletions research/eval/test_laya_eval.py
Original file line number Diff line number Diff line change
Expand Up @@ -103,13 +103,17 @@ def check_true(name, cond, detail=""):
round(ece([0.9] * 1000, [1.0] * 1000), 4))
check("ece/one sample", round(ece([1.0], [1.0]), 4), 0.0)

# Known boundary: the bin test is `conf > lo`, so conf == 0.0 falls in no bin and
# contributes nothing. `laya.common.ece_score` and `research/scripts/bench_local.py`
# bin identically, so this harness deliberately matches them rather than diverging --
# comparability with the published tables is the point. PR #39 addresses the same
# boundary in `laya.common`; if it lands, this harness should follow it.
check("ece/conf==0.0 is not binned (matches upstream)",
round(ece([0.0, 0.0], [1.0, 1.0]), 4), 0.0)
# Bin boundary: the first bin is closed at the bottom, so conf == 0.0 IS counted. This
# harness tested `conf > lo` for every bin until the divergence was found, which made it
# the only one of the four implementations that binned differently --
# `laya.common.ece_score`, `research/scripts/bench_local.py` and
# `research/scripts/build_benchmark_nb.py` all settled on this boundary in #39.
check("ece/conf==0.0 is binned (matches the other three)",
round(ece([0.0, 0.0], [1.0, 1.0]), 4), 1.0)
check("ece/conf==0.0 carries its bin weight",
round(ece([0.0, 1.0], [1.0, 1.0]), 4), 0.5)
check("ece/conf==0.0 and correct costs nothing",
round(ece([0.0, 0.0], [0.0, 0.0]), 4), 0.0)
check_true("ece/conf slightly above 0 IS binned",
ece([1e-9, 1e-9], [1.0, 1.0]) > 0.0)

Expand Down
Loading