Repository navigation
fix(benchmarks): two parity cells did not match the JSON they cite - #299
Merged
NandhaKishorM merged 1 commit intoSep 24, 2026
Merged
Conversation
`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells did not match: | row | column | table said | JSON says | |---|---|---|---| | `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) | | `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) | Both round down, so this is not a formatting difference. The other 16 numeric cells and all twelve argmax agreement counts do match. The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and `README.md:488` both said the fast path *is at least as close to the fp32 reference as the stock bf16 path is*, and on `laya-multilingual` / `noul` it is not: ``` laya-multilingual noul fast_vs_fp32 = 0.045488 stock_vs_fp32 = 0.044593 ``` which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports — within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and name the row that is the exception, rather than asserting something a reader can falsify from the table in front of them. Deliberate choices tested: - **A check, not just two corrections.** `NandhaKishorM#170`, `NandhaKishorM#134` and `NandhaKishorM#208` were all a published number that no longer traced to a run, and `NandhaKishorM#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `NandhaKishorM#285`'s proposal that can run in CI today. - **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite: ``` 52 passed, 2 failed FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043' FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045' ``` With them in place, `54 passed, 0 failed`. - **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero. - **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted. - **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence. - **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `NandhaKishorM#212` added the Windows lane. - Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model. ``` $ python tests/test_doc_tables.py 54 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_load_errors.py 17 passed, 0 failed $ python tests/test_email.py 90 passed, 0 failed ``` Known limitation: this verifies the tables that cite a committed JSON, and it cannot check prose — it would have caught the two cells but not the sentence above them, which is why both were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that is `NandhaKishorM#285`'s half and needs the hardware.
Owner
|
Thank you. Two cells that rounded the wrong way against the JSON they cite, plus a test that checks the tables against the JSON, is exactly the kind of guard that keeps the numbers honest. Merging. |
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…andhaKishorM#299) `BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells did not match: | row | column | table said | JSON says | |---|---|---|---| | `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) | | `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) | Both round down, so this is not a formatting difference. The other 16 numeric cells and all twelve argmax agreement counts do match. The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and `README.md:488` both said the fast path *is at least as close to the fp32 reference as the stock bf16 path is*, and on `laya-multilingual` / `noul` it is not: ``` laya-multilingual noul fast_vs_fp32 = 0.045488 stock_vs_fp32 = 0.044593 ``` which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports — within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and name the row that is the exception, rather than asserting something a reader can falsify from the table in front of them. Deliberate choices tested: - **A check, not just two corrections.** `NandhaKishorM#170`, `NandhaKishorM#134` and `NandhaKishorM#208` were all a published number that no longer traced to a run, and `NandhaKishorM#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `NandhaKishorM#285`'s proposal that can run in CI today. - **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite: ``` 52 passed, 2 failed FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043' FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045' ``` With them in place, `54 passed, 0 failed`. - **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero. - **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted. - **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence. - **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `NandhaKishorM#212` added the Windows lane. - Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model. ``` $ python tests/test_doc_tables.py 54 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_load_errors.py 17 passed, 0 failed $ python tests/test_email.py 90 passed, 0 failed ``` Known limitation: this verifies the tables that cite a committed JSON, and it cannot check prose — it would have caught the two cells but not the sentence above them, which is why both were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that is `NandhaKishorM#285`'s half and needs the hardware. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(benchmarks): two parity cells did not match the JSON they cite
BENCHMARKS.md:274says the fast-path comparison "can be re-checked without a GPU" becauseevery per-option probability is in
benchmarks/results/parity_*.json. It can, and two cellsdid not match:
laya/noulmax |p_fast - p_fp32|0.044d_fast_fp32 = 0.0434565544128418)laya-multilingual/noulmax |p_fast - p_fp32|0.046d_fast_fp32 = 0.04548829793930054)Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.
The sentence under that table was falsified by its own numbers.
BENCHMARKS.md:286andREADME.md:488both said the fast path is at least as close to the fp32 reference as thestock bf16 path is, and on
laya-multilingual/noulit is not:which is the
0.045vs0.045printed two lines above it. The claim held on the other fiverows (e.g.
laya/noul0.0435 vs 0.0578). Both places now state the bound the data supports— within
0.046of fp32, within0.076of stock, argmax agreement at least47/48— andname the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.
Deliberate choices tested:
#170,#134and#208were all a published number that no longer traced to a run, and#285proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is.tests/test_doc_tables.pyreads the published table out ofBENCHMARKS.mdand compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of#285's proposal that can run in CI today.54 passed, 0 failed.103-332 q/sthroughput range, the fast-path latency table whosefast_*.jsoncarry noevalkey, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.round(0.009501, 3)is0.010under banker's rounding but the table's own0.010from0.009501shows the ordinary convention was used, so the comparison quantizes withROUND_HALF_UPrather thanround(). Usinground()would report that row as drifted.laya-multilingual/noul, so a future edit cannot quietly restore the stronger sentence.newline=""and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because#212added the Windows lane.test_packaging.py, which is the other suite that reads repository files rather than running the model.Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is
#285's half and needs the hardware.