Repository navigation
fix(research): the eval harness is the one ECE that binned differently - #232
Merged
NandhaKishorM merged 1 commit intoSep 23, 2026
Merged
NandhaKishorM merged 1 commit into
NandhaKishorM merged 1 commit into
Conversation
`research/eval/README.md` claimed this harness shared its `confidence == 0.0` bin boundary with `laya.common.ece_score` and `bench_local.py`, and `test_laya_eval.py` asserted it under the name "matches upstream". It did not: ``` laya.common.ece_score([0,0],[1,1]) = 1.0 research/scripts/bench_local.py = 1.0 research/scripts/build_benchmark_nb.py = 1.0 research/eval/laya_eval.py = 0.0 <-- this one ``` `NandhaKishorM#39` ("fix: include zero confidence in ECE") landed the `conf >= lo` boundary in `laya/common.py`, `bench_local.py` and `build_benchmark_nb.py` at 07:03 today. The harness was merged in `NandhaKishorM#210` at 07:07 with the pre-`NandhaKishorM#39` test, `sel = (confidence > lo) & (confidence <= hi)`, so zero-confidence cases fell in no bin and contributed nothing. That is this PR's own mistake, and the README documented it as agreement. The fix is the first bin: ```python for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])): sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi) ``` All four implementations now agree on every case: | case | common | laya_eval | bench_local | build_nb | |---|---|---|---|---| | conf exactly 0, wrong | 1.0000 | 1.0000 | 1.0000 | 1.0000 | | conf 0 and 1 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | | ordinary | 0.1000 | 0.1000 | 0.1000 | 0.1000 | | just above 0 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | | one bin | 0.0500 | 0.0500 | 0.0500 | 0.0500 | | all wrong, confident | 0.9500 | 0.9500 | 0.9500 | 0.9500 | (`build_benchmark_nb.py` defines its `ece_score` inside a generated notebook cell, so it cannot be imported; the function text is extracted and executed to compare it.) Deliberate choices tested: - **Fixed the harness rather than the doc.** `NandhaKishorM#39` was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim. - **Regenerating the committed data is not needed, and I checked rather than assumed.** The change can only move a language that has a `confidence == 0.0` case. Six languages re-run at the committed settings (`--unclamped`, 100 cases, 20 options) have **zero** such cases each, and every one of their ECE values matches `cpu_51_language_sweep.json` exactly under both boundaries: | lang | cases with `confidence == 0.0` | old ECE | new ECE | committed | |---|---|---|---|---| | en | 0 | 0.1789 | 0.1789 | 0.1789 | | de | 0 | 0.5584 | 0.5584 | 0.5584 | | km | 0 | 0.9515 | 0.9515 | 0.9515 | | am | 0 | 0.8253 | 0.8253 | 0.8252 | | my | 0 | 0.8605 | 0.8605 | 0.8605 | | th | 0 | 0.8814 | 0.8814 | 0.8814 | `am` differs from the committed 0.8252 by 0.0001 under *both* boundaries, which is the torch-version quantisation `research/eval/README.md` already documents, not this change. - **The test now asserts the boundary by value instead of asserting the harness matches itself.** The old case was written against the harness's own arithmetic (`check(..., 0.0)`), so it passed for exactly the reason the divergence went unnoticed. - **No CI wiring added here.** `research/eval/test_laya_eval.py` runs in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix. Known limitation: the committed `research/results/eval_english_51_languages.json` was generated under the old boundary, so it is an artefact of the pre-fix harness. It is unaffected for the same reason the table above holds — no case has zero confidence — and I have left it byte-identical rather than regenerating 51 languages to reproduce the same numbers. Verified: ``` $ python research/eval/test_laya_eval.py 49 passed, 0 failed (47 before) $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ research/ # clean ```
Owner
|
Thank you, and for tracing it across all four ECE implementations. A harness test called "matches upstream" that did not match upstream is the worst kind of green. The eval harness now bins |
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
NandhaKishorM#232) `research/eval/README.md` claimed this harness shared its `confidence == 0.0` bin boundary with `laya.common.ece_score` and `bench_local.py`, and `test_laya_eval.py` asserted it under the name "matches upstream". It did not: ``` laya.common.ece_score([0,0],[1,1]) = 1.0 research/scripts/bench_local.py = 1.0 research/scripts/build_benchmark_nb.py = 1.0 research/eval/laya_eval.py = 0.0 <-- this one ``` `NandhaKishorM#39` ("fix: include zero confidence in ECE") landed the `conf >= lo` boundary in `laya/common.py`, `bench_local.py` and `build_benchmark_nb.py` at 07:03 today. The harness was merged in `NandhaKishorM#210` at 07:07 with the pre-`NandhaKishorM#39` test, `sel = (confidence > lo) & (confidence <= hi)`, so zero-confidence cases fell in no bin and contributed nothing. That is this PR's own mistake, and the README documented it as agreement. The fix is the first bin: ```python for i, (lo, hi) in enumerate(zip(edges[:-1], edges[1:])): sel = (confidence >= lo if i == 0 else confidence > lo) & (confidence <= hi) ``` All four implementations now agree on every case: | case | common | laya_eval | bench_local | build_nb | |---|---|---|---|---| | conf exactly 0, wrong | 1.0000 | 1.0000 | 1.0000 | 1.0000 | | conf 0 and 1 | 0.5000 | 0.5000 | 0.5000 | 0.5000 | | ordinary | 0.1000 | 0.1000 | 0.1000 | 0.1000 | | just above 0 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | | one bin | 0.0500 | 0.0500 | 0.0500 | 0.0500 | | all wrong, confident | 0.9500 | 0.9500 | 0.9500 | 0.9500 | (`build_benchmark_nb.py` defines its `ece_score` inside a generated notebook cell, so it cannot be imported; the function text is extracted and executed to compare it.) Deliberate choices tested: - **Fixed the harness rather than the doc.** `NandhaKishorM#39` was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim. - **Regenerating the committed data is not needed, and I checked rather than assumed.** The change can only move a language that has a `confidence == 0.0` case. Six languages re-run at the committed settings (`--unclamped`, 100 cases, 20 options) have **zero** such cases each, and every one of their ECE values matches `cpu_51_language_sweep.json` exactly under both boundaries: | lang | cases with `confidence == 0.0` | old ECE | new ECE | committed | |---|---|---|---|---| | en | 0 | 0.1789 | 0.1789 | 0.1789 | | de | 0 | 0.5584 | 0.5584 | 0.5584 | | km | 0 | 0.9515 | 0.9515 | 0.9515 | | am | 0 | 0.8253 | 0.8253 | 0.8252 | | my | 0 | 0.8605 | 0.8605 | 0.8605 | | th | 0 | 0.8814 | 0.8814 | 0.8814 | `am` differs from the committed 0.8252 by 0.0001 under *both* boundaries, which is the torch-version quantisation `research/eval/README.md` already documents, not this change. - **The test now asserts the boundary by value instead of asserting the harness matches itself.** The old case was written against the harness's own arithmetic (`check(..., 0.0)`), so it passed for exactly the reason the divergence went unnoticed. - **No CI wiring added here.** `research/eval/test_laya_eval.py` runs in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix. Known limitation: the committed `research/results/eval_english_51_languages.json` was generated under the old boundary, so it is an artefact of the pre-fix harness. It is unaffected for the same reason the table above holds — no case has zero confidence — and I have left it byte-identical rather than regenerating 51 languages to reproduce the same numbers. Verified: ``` $ python research/eval/test_laya_eval.py 49 passed, 0 failed (47 before) $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ research/ # clean ``` Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(research): the eval harness is the one ECE that binned differently
research/eval/README.mdclaimed this harness shared itsconfidence == 0.0binboundary with
laya.common.ece_scoreandbench_local.py, andtest_laya_eval.pyasserted it under the name "matches upstream". It did not:
#39("fix: include zero confidence in ECE") landed theconf >= loboundary inlaya/common.py,bench_local.pyandbuild_benchmark_nb.pyat 07:03 today. Theharness was merged in
#210at 07:07 with the pre-#39test,sel = (confidence > lo) & (confidence <= hi), so zero-confidence cases fell in no bin and contributed nothing.That is this PR's own mistake, and the README documented it as agreement.
The fix is the first bin:
All four implementations now agree on every case:
(
build_benchmark_nb.pydefines itsece_scoreinside a generated notebook cell, so itcannot be imported; the function text is extracted and executed to compare it.)
Deliberate choices tested:
Fixed the harness rather than the doc.
#39was a deliberate change to include zero confidence, and it is the majority position in the repo — three implementations against one. The alternative was to reword the README to say the harness bins alone, which would leave the harness incomparable with the published numbers for no reason. Comparability was the stated point of the original claim.Regenerating the committed data is not needed, and I checked rather than assumed. The change can only move a language that has a
confidence == 0.0case. Six languages re-run at the committed settings (--unclamped, 100 cases, 20 options) have zero such cases each, and every one of their ECE values matchescpu_51_language_sweep.jsonexactly under both boundaries:confidence == 0.0amdiffers from the committed 0.8252 by 0.0001 under both boundaries, which is the torch-version quantisationresearch/eval/README.mdalready documents, not this change.The test now asserts the boundary by value instead of asserting the harness matches itself. The old case was written against the harness's own arithmetic (
check(..., 0.0)), so it passed for exactly the reason the divergence went unnoticed.No CI wiring added here.
research/eval/test_laya_eval.pyruns in no workflow, so this suite is not gated. Wiring it needs an install step in both lanes and belongs with the other lane changes, not in a correctness fix.Known limitation: the committed
research/results/eval_english_51_languages.jsonwasgenerated under the old boundary, so it is an artefact of the pre-fix harness. It is
unaffected for the same reason the table above holds — no case has zero confidence — and
I have left it byte-identical rather than regenerating 51 languages to reproduce the same
numbers.
Verified: