Skip to content

fix(evals): a dataset, report or agreement run saved with a BOM is read, not refused - #1415

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:evals-bom-utf8sig
Oct 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:evals-bom-utf8sig

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

Three readers of a caller's JSON in the evals tool opened encoding="utf-8" and are now opening encoding="utf-8-sig":

  • laya/evals.py Dataset.from_jsonl -- the evaluation dataset.
  • laya/evals_cli.py _load_report -- every report read (compare's report and baseline, run's baseline).
  • laya/evals_cli.py the agreement subcommand's three saved-run reads.

Writers are untouched (--json / --markdown still emit plain utf-8), and tests: one new BOM case each in tests/test_evals.py (dataset + report) and tests/test_evals_agreement.py (agreement CLI).

Why

A file written by Notepad, Excel or most Windows editors starts with a three-byte UTF-8 BOM. Read as plain utf-8, those bytes survive as U+FEFF at the start of the document, and json refuses them -- with a message that names the fix this library could have applied itself:

EvalError: /tmp/d.jsonl:1 is not valid JSON: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
EvalError: cannot read report '/tmp/report.json': Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
laya-evals agreement: cannot read agreement run '.../reference.json': Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)   (exit 2)

So a dataset exported from a spreadsheet is unrunnable, and the refusal tells the user to change the encoding of the file rather than changing three characters of the reader. The repository already decided this question everywhere else a caller hands over a file: laya/cli.py opens --questions, --batch and stdin with utf-8-sig and its comment says why -- "a questions file written by Notepad or Excel starts with a UTF-8 BOM, and a plain utf-8 read leaves it as U+FEFF, so json.load refuses a file that is valid JSON -- with its own 'decode using utf-8-sig' hint. The other file readers in this repo open the same way for the same reason" -- and laya/train.py (whose docstring says it uses utf-8-sig "so UTF-8 BOM headers from Excel are handled transparently"), laya/evidence.py and docker/entrypoint.py do. The evals doors were the exception to a rule the rest of the codebase already follows, which is what makes a dataset read and a --questions read disagree about the same file on the same machine.

Behaviour is otherwise identical: utf-8-sig strips a leading BOM and decodes every BOM-less file exactly as utf-8 did, so all the refusal paths already tested (not-JSON, non-UTF-8 bytes, a directory, an unreadable volume, unclosable depth) are unchanged -- the new tests assert one of them still refuses in the same words.

How it was verified

python -m pytest tests/test_evals.py tests/test_evals_agreement.py -q
# base (fe9a6c35): 278 passed        ->  this branch: 281 passed
python -m pytest tests/test_evals_api.py                       # 69 passed, 0 failed (script runner; unchanged)
python tests/test_doc_tables.py                                # 181 passed, 0 failed (unchanged from base)
python tests/test_cli.py                                       # 213 passed, 0 failed (unchanged from base)
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/                                     # clean
python laya-java/scripts/gen_fixtures.py --check
# the same two pre-existing stales (`lang_detect.json`, `python_json.json`) as at base

All three leaks were reproduced at base before the fix (the three EvalError texts above are that output, verbatim). Then laya/evals.py and laya/evals_cli.py were reverted to base with the new tests in place:

FAILED tests/test_evals.py::test_dataset_from_jsonl_reads_a_file_that_starts_with_a_bom
FAILED tests/test_evals.py::test_load_report_reads_a_file_that_starts_with_a_bom
FAILED tests/test_evals_agreement.py::test_cli_reads_bom_prefixed_saved_runs
3 failed, 278 deselected in 0.75s

The agreement case is a real subprocess run of laya-evals agreement (returncode == 2 at base, 0 with the fix) and asserts the published report equals compare(runs) byte for byte -- the BOM changes nothing about the numbers, so it must change nothing about the verdict. Both files were restored byte-identical from /tmp copies and the suites re-run from disk afterwards.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Three readers of a caller's JSON opened `utf-8`, so a UTF-8 BOM survived as U+FEFF on line 1 and
`json` refused a document that is valid JSON -- quoting its own "decode using utf-8-sig" hint back
at the user as the reason the file could not be read. `laya/cli.py`, `laya/train.py` and
`laya/evidence.py` already open every caller-supplied file with `utf-8-sig`, and the comment in
`laya/cli.py` says the repo's other readers do; the evals doors were the exception.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.18. utf-8-sig is the correct reader here, and the detail that makes this a bug rather than a preference is that the file is valid JSON: the BOM survives as U+FEFF on line 1 and json refuses a document that is well formed, while suggesting utf-8-sig in its own error text. Being told to do the thing the reader could have done is a poor experience.

Covering all three readers in one change is what I would have asked for. A dataset that loads and an agreement run that does not would have been a worse state than the one you started from, since the inconsistency is harder to diagnose than a uniform refusal.

Citing laya/cli.py, laya/train.py and laya/evidence.py as already reading caller files this way is the part that makes this obviously correct: it closes a gap in a convention the tree had already chosen, rather than introducing one.

Note for anyone reading the diff later: utf-8-sig strips a leading BOM if present and is otherwise identical to utf-8, so this does not change how any existing well-formed file is read. That is why it is safe to apply to all three readers at once.

Verified: 706 tests across 93 suites.

@NandhaKishorM
NandhaKishorM merged commit 7f7671f into NandhaKishorM:main Oct 29, 2026
63 checks passed
NandhaKishorM added a commit that referenced this pull request Oct 29, 2026
Twelve pull requests: #1364, #1409, #1410, #1412, #1413, #1414, #1415, #1416,
#1417, #1418, #1419, #1420.

Test suites: 706 passing across 93 suites, up from 640 across 92. laya-cpp
registers 423 ctest tests, up from 213, and its MIN_TESTS floor is raised to match.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants