Skip to content

fix(benchmarks): two parity cells did not match the JSON they cite - #299

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/parity-table-matches-json
Sep 24, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:fix/parity-table-matches-json

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

fix(benchmarks): two parity cells did not match the JSON they cite

BENCHMARKS.md:274 says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in benchmarks/results/parity_*.json. It can, and two cells
did not match:

row column table said JSON says
laya / noul max |p_fast - p_fp32| 0.044 0.043 (d_fast_fp32 = 0.0434565544128418)
laya-multilingual / noul max |p_fast - p_fp32| 0.046 0.045 (d_fast_fp32 = 0.04548829793930054)

Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.

The sentence under that table was falsified by its own numbers. BENCHMARKS.md:286 and
README.md:488 both said the fast path is at least as close to the fp32 reference as the
stock bf16 path is
, and on laya-multilingual / noul it is not:

laya-multilingual noul   fast_vs_fp32 = 0.045488   stock_vs_fp32 = 0.044593

which is the 0.045 vs 0.045 printed two lines above it. The claim held on the other five
rows (e.g. laya/noul 0.0435 vs 0.0578). Both places now state the bound the data supports
— within 0.046 of fp32, within 0.076 of stock, argmax agreement at least 47/48 — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.

Deliberate choices tested:

  • A check, not just two corrections. #170, #134 and #208 were all a published number that no longer traced to a run, and #285 proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. tests/test_doc_tables.py reads the published table out of BENCHMARKS.md and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of #285's proposal that can run in CI today.
  • It fails loudly on drift instead of skipping. Removing the two corrections and running the suite:
    52 passed, 2 failed
      FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
      FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
    
    With them in place, 54 passed, 0 failed.
  • It reports what it cannot check rather than implying coverage. Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the 103-332 q/s throughput range, the fast-path latency table whose fast_*.json carry no eval key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
  • Rounding matches how the tables were written. round(0.009501, 3) is 0.010 under banker's rounding but the table's own 0.010 from 0.009501 shows the ordinary convention was used, so the comparison quantizes with ROUND_HALF_UP rather than round(). Using round() would report that row as drifted.
  • The old claim is pinned as false, not just deleted. The suite asserts that the set of rows where the stock path is closer to fp32 is exactly laya-multilingual/noul, so a future edit cannot quietly restore the stronger sentence.
  • Line endings. The suite opens markdown with newline="" and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because #212 added the Windows lane.
  • Wired into both lanes beside test_packaging.py, which is the other suite that reads repository files rather than running the model.
$ python tests/test_doc_tables.py               54 passed, 0 failed
$ python tests/test_packaging.py                all packaging tests passed
$ python tests/test_criteria.py                 128 passed, 0 failed
$ python tests/test_router.py                   all routing tests passed
$ python tests/test_load_errors.py              17 passed, 0 failed
$ python tests/test_email.py                    90 passed, 0 failed

Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is #285's half and needs the hardware.

`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells
did not match:

| row | column | table said | JSON says |
|---|---|---|---|
| `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) |
| `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) |

Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.

The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and
`README.md:488` both said the fast path *is at least as close to the fp32 reference as the
stock bf16 path is*, and on `laya-multilingual` / `noul` it is not:

```
laya-multilingual noul   fast_vs_fp32 = 0.045488   stock_vs_fp32 = 0.044593
```

which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five
rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports
— within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.

Deliberate choices tested:

- **A check, not just two corrections.** `NandhaKishorM#170`, `NandhaKishorM#134` and `NandhaKishorM#208` were all a published number that no longer traced to a run, and `NandhaKishorM#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `NandhaKishorM#285`'s proposal that can run in CI today.
- **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite:
  ```
  52 passed, 2 failed
    FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
    FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
  ```
  With them in place, `54 passed, 0 failed`.
- **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
- **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted.
- **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence.
- **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `NandhaKishorM#212` added the Windows lane.
- Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model.

```
$ python tests/test_doc_tables.py               54 passed, 0 failed
$ python tests/test_packaging.py                all packaging tests passed
$ python tests/test_criteria.py                 128 passed, 0 failed
$ python tests/test_router.py                   all routing tests passed
$ python tests/test_load_errors.py              17 passed, 0 failed
$ python tests/test_email.py                    90 passed, 0 failed
```

Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is `NandhaKishorM#285`'s half and needs the hardware.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you. Two cells that rounded the wrong way against the JSON they cite, plus a test that checks the tables against the JSON, is exactly the kind of guard that keeps the numbers honest. Merging.

@NandhaKishorM
NandhaKishorM merged commit d17414d into NandhaKishorM:main Sep 24, 2026
17 checks passed
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…andhaKishorM#299)

`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells
did not match:

| row | column | table said | JSON says |
|---|---|---|---|
| `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) |
| `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) |

Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.

The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and
`README.md:488` both said the fast path *is at least as close to the fp32 reference as the
stock bf16 path is*, and on `laya-multilingual` / `noul` it is not:

```
laya-multilingual noul   fast_vs_fp32 = 0.045488   stock_vs_fp32 = 0.044593
```

which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five
rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports
— within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.

Deliberate choices tested:

- **A check, not just two corrections.** `NandhaKishorM#170`, `NandhaKishorM#134` and `NandhaKishorM#208` were all a published number that no longer traced to a run, and `NandhaKishorM#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `NandhaKishorM#285`'s proposal that can run in CI today.
- **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite:
  ```
  52 passed, 2 failed
    FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
    FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
  ```
  With them in place, `54 passed, 0 failed`.
- **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
- **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted.
- **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence.
- **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `NandhaKishorM#212` added the Windows lane.
- Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model.

```
$ python tests/test_doc_tables.py               54 passed, 0 failed
$ python tests/test_packaging.py                all packaging tests passed
$ python tests/test_criteria.py                 128 passed, 0 failed
$ python tests/test_router.py                   all routing tests passed
$ python tests/test_load_errors.py              17 passed, 0 failed
$ python tests/test_email.py                    90 passed, 0 failed
```

Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is `NandhaKishorM#285`'s half and needs the hardware.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants