Skip to content

fix(research): the eval harness is the one ECE that binned differently - #232

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/ece-boundary-matches-upstream
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/ece-boundary-matches-upstream

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

fix(research): the eval harness is the one ECE that binned differently

research/eval/README.md claimed this harness shared its confidence == 0.0 bin
boundary with laya.common.ece_score and bench_local.py, and test_laya_eval.py
asserted it under the name "matches upstream". It did not:

laya.common.ece_score([0,0],[1,1])          = 1.0
research/scripts/bench_local.py             = 1.0
research/scripts/build_benchmark_nb.py      = 1.0
research/eval/laya_eval.py                  = 0.0   <-- this one

#39 ("fix: include zero confidence in ECE") landed the conf >= lo boundary in
laya/common.py, bench_local.py and build_benchmark_nb.py at 07:03 today. The
harness was merged in #210 at 07:07 with the pre-#39 test, sel = (confidence > lo) & (confidence <= hi), so zero-confidence cases fell in no bin and contributed nothing.
That is this PR's own mistake, and the README documented it as agreement.

The fix is the first bin:

for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
    sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi)

All four implementations now agree on every case:

case common laya_eval bench_local build_nb
conf exactly 0, wrong 1.0000 1.0000 1.0000 1.0000
conf 0 and 1 0.5000 0.5000 0.5000 0.5000
ordinary 0.1000 0.1000 0.1000 0.1000
just above 0 1.0000 1.0000 1.0000 1.0000
one bin 0.0500 0.0500 0.0500 0.0500
all wrong, confident 0.9500 0.9500 0.9500 0.9500

(build_benchmark_nb.py defines its ece_score inside a generated notebook cell, so it
cannot be imported; the function text is extracted and executed to compare it.)

Deliberate choices tested:

  • Fixed the harness rather than the doc. #39 was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim.

  • Regenerating the committed data is not needed, and I checked rather than assumed. The change can only move a language that has a confidence == 0.0 case. Six languages re-run at the committed settings (--unclamped, 100 cases, 20 options) have zero such cases each, and every one of their ECE values matches cpu_51_language_sweep.json exactly under both boundaries:

    lang cases with confidence == 0.0 old ECE new ECE committed
    en 0 0.1789 0.1789 0.1789
    de 0 0.5584 0.5584 0.5584
    km 0 0.9515 0.9515 0.9515
    am 0 0.8253 0.8253 0.8252
    my 0 0.8605 0.8605 0.8605
    th 0 0.8814 0.8814 0.8814

    am differs from the committed 0.8252 by 0.0001 under both boundaries, which is the torch-version quantisation research/eval/README.md already documents, not this change.

  • The test now asserts the boundary by value instead of asserting the harness matches itself. The old case was written against the harness's own arithmetic (check(..., 0.0)), so it passed for exactly the reason the divergence went unnoticed.

  • No CI wiring added here. research/eval/test_laya_eval.py runs in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix.

Known limitation: the committed research/results/eval_english_51_languages.json was
generated under the old boundary, so it is an artefact of the pre-fix harness. It is
unaffected for the same reason the table above holds — no case has zero confidence — and
I have left it byte-identical rather than regenerating 51 languages to reproduce the same
numbers.

Verified:

$ python research/eval/test_laya_eval.py     49 passed, 0 failed     (47 before)
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/ research/    # clean

`research/eval/README.md` claimed this harness shared its `confidence == 0.0` bin
boundary with `laya.common.ece_score` and `bench_local.py`, and `test_laya_eval.py`
asserted it under the name "matches upstream". It did not:

```
laya.common.ece_score([0,0],[1,1])          = 1.0
research/scripts/bench_local.py             = 1.0
research/scripts/build_benchmark_nb.py      = 1.0
research/eval/laya_eval.py                  = 0.0   <-- this one
```

`NandhaKishorM#39` ("fix: include zero confidence in ECE") landed the `conf >= lo` boundary in
`laya/common.py`, `bench_local.py` and `build_benchmark_nb.py` at 07:03 today. The
harness was merged in `NandhaKishorM#210` at 07:07 with the pre-`NandhaKishorM#39` test, `sel = (confidence > lo)
& (confidence <= hi)`, so zero-confidence cases fell in no bin and contributed nothing.
That is this PR's own mistake, and the README documented it as agreement.

The fix is the first bin:

```python
for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
    sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi)
```

All four implementations now agree on every case:

| case | common | laya_eval | bench_local | build_nb |
|---|---|---|---|---|
| conf exactly 0, wrong | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| conf 0 and 1 | 0.5000 | 0.5000 | 0.5000 | 0.5000 |
| ordinary | 0.1000 | 0.1000 | 0.1000 | 0.1000 |
| just above 0 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| one bin | 0.0500 | 0.0500 | 0.0500 | 0.0500 |
| all wrong, confident | 0.9500 | 0.9500 | 0.9500 | 0.9500 |

(`build_benchmark_nb.py` defines its `ece_score` inside a generated notebook cell, so it
cannot be imported; the function text is extracted and executed to compare it.)

Deliberate choices tested:

- **Fixed the harness rather than the doc.** `NandhaKishorM#39` was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim.
- **Regenerating the committed data is not needed, and I checked rather than assumed.** The change can only move a language that has a `confidence == 0.0` case. Six languages re-run at the committed settings (`--unclamped`, 100 cases, 20 options) have **zero** such cases each, and every one of their ECE values matches `cpu_51_language_sweep.json` exactly under both boundaries:

  | lang | cases with `confidence == 0.0` | old ECE | new ECE | committed |
  |---|---|---|---|---|
  | en | 0 | 0.1789 | 0.1789 | 0.1789 |
  | de | 0 | 0.5584 | 0.5584 | 0.5584 |
  | km | 0 | 0.9515 | 0.9515 | 0.9515 |
  | am | 0 | 0.8253 | 0.8253 | 0.8252 |
  | my | 0 | 0.8605 | 0.8605 | 0.8605 |
  | th | 0 | 0.8814 | 0.8814 | 0.8814 |

  `am` differs from the committed 0.8252 by 0.0001 under *both* boundaries, which is the torch-version quantisation `research/eval/README.md` already documents, not this change.
- **The test now asserts the boundary by value instead of asserting the harness matches itself.** The old case was written against the harness's own arithmetic (`check(..., 0.0)`), so it passed for exactly the reason the divergence went unnoticed.
- **No CI wiring added here.** `research/eval/test_laya_eval.py` runs in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix.

Known limitation: the committed `research/results/eval_english_51_languages.json` was
generated under the old boundary, so it is an artefact of the pre-fix harness. It is
unaffected for the same reason the table above holds — no case has zero confidence — and
I have left it byte-identical rather than regenerating 51 languages to reproduce the same
numbers.

Verified:

```
$ python research/eval/test_laya_eval.py     49 passed, 0 failed     (47 before)
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/ research/    # clean
```
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, and for tracing it across all four ECE implementations. A harness test called "matches upstream" that did not match upstream is the worst kind of green. The eval harness now bins confidence == 0.0 the same way as laya.common.ece_score. Merging.

@NandhaKishorM
NandhaKishorM merged commit b9d1305 into NandhaKishorM:main Sep 23, 2026
13 checks passed
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#232)

`research/eval/README.md` claimed this harness shared its `confidence == 0.0` bin
boundary with `laya.common.ece_score` and `bench_local.py`, and `test_laya_eval.py`
asserted it under the name "matches upstream". It did not:

```
laya.common.ece_score([0,0],[1,1])          = 1.0
research/scripts/bench_local.py             = 1.0
research/scripts/build_benchmark_nb.py      = 1.0
research/eval/laya_eval.py                  = 0.0   <-- this one
```

`NandhaKishorM#39` ("fix: include zero confidence in ECE") landed the `conf >= lo` boundary in
`laya/common.py`, `bench_local.py` and `build_benchmark_nb.py` at 07:03 today. The
harness was merged in `NandhaKishorM#210` at 07:07 with the pre-`NandhaKishorM#39` test, `sel = (confidence > lo)
& (confidence <= hi)`, so zero-confidence cases fell in no bin and contributed nothing.
That is this PR's own mistake, and the README documented it as agreement.

The fix is the first bin:

```python
for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])):
    sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi)
```

All four implementations now agree on every case:

| case | common | laya_eval | bench_local | build_nb |
|---|---|---|---|---|
| conf exactly 0, wrong | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| conf 0 and 1 | 0.5000 | 0.5000 | 0.5000 | 0.5000 |
| ordinary | 0.1000 | 0.1000 | 0.1000 | 0.1000 |
| just above 0 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| one bin | 0.0500 | 0.0500 | 0.0500 | 0.0500 |
| all wrong, confident | 0.9500 | 0.9500 | 0.9500 | 0.9500 |

(`build_benchmark_nb.py` defines its `ece_score` inside a generated notebook cell, so it
cannot be imported; the function text is extracted and executed to compare it.)

Deliberate choices tested:

- **Fixed the harness rather than the doc.** `NandhaKishorM#39` was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim.
- **Regenerating the committed data is not needed, and I checked rather than assumed.** The change can only move a language that has a `confidence == 0.0` case. Six languages re-run at the committed settings (`--unclamped`, 100 cases, 20 options) have **zero** such cases each, and every one of their ECE values matches `cpu_51_language_sweep.json` exactly under both boundaries:

  | lang | cases with `confidence == 0.0` | old ECE | new ECE | committed |
  |---|---|---|---|---|
  | en | 0 | 0.1789 | 0.1789 | 0.1789 |
  | de | 0 | 0.5584 | 0.5584 | 0.5584 |
  | km | 0 | 0.9515 | 0.9515 | 0.9515 |
  | am | 0 | 0.8253 | 0.8253 | 0.8252 |
  | my | 0 | 0.8605 | 0.8605 | 0.8605 |
  | th | 0 | 0.8814 | 0.8814 | 0.8814 |

  `am` differs from the committed 0.8252 by 0.0001 under *both* boundaries, which is the torch-version quantisation `research/eval/README.md` already documents, not this change.
- **The test now asserts the boundary by value instead of asserting the harness matches itself.** The old case was written against the harness's own arithmetic (`check(..., 0.0)`), so it passed for exactly the reason the divergence went unnoticed.
- **No CI wiring added here.** `research/eval/test_laya_eval.py` runs in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix.

Known limitation: the committed `research/results/eval_english_51_languages.json` was
generated under the old boundary, so it is an artefact of the pre-fix harness. It is
unaffected for the same reason the table above holds — no case has zero confidence — and
I have left it byte-identical rather than regenerating 51 languages to reproduce the same
numbers.

Verified:

```
$ python research/eval/test_laya_eval.py     49 passed, 0 failed     (47 before)
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/ research/    # clean
```

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants