Repository navigation
fix(evals): a dataset, report or agreement run saved with a BOM is read, not refused - #1415
Conversation
Three readers of a caller's JSON opened `utf-8`, so a UTF-8 BOM survived as U+FEFF on line 1 and `json` refused a document that is valid JSON -- quoting its own "decode using utf-8-sig" hint back at the user as the reason the file could not be read. `laya/cli.py`, `laya/train.py` and `laya/evidence.py` already open every caller-supplied file with `utf-8-sig`, and the comment in `laya/cli.py` says the repo's other readers do; the evals doors were the exception.
|
Merged in 0.4.18. Covering all three readers in one change is what I would have asked for. A dataset that loads and an agreement run that does not would have been a worse state than the one you started from, since the inconsistency is harder to diagnose than a uniform refusal. Citing Note for anyone reading the diff later: Verified: 706 tests across 93 suites. |
Twelve pull requests: #1364, #1409, #1410, #1412, #1413, #1414, #1415, #1416, #1417, #1418, #1419, #1420. Test suites: 706 passing across 93 suites, up from 640 across 92. laya-cpp registers 423 ctest tests, up from 213, and its MIN_TESTS floor is raised to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What
Three readers of a caller's JSON in the evals tool opened
encoding="utf-8"and are now openingencoding="utf-8-sig":laya/evals.pyDataset.from_jsonl-- the evaluation dataset.laya/evals_cli.py_load_report-- every report read (compare's report and baseline,run's baseline).laya/evals_cli.pytheagreementsubcommand's three saved-run reads.Writers are untouched (
--json/--markdownstill emit plainutf-8), and tests: one new BOM case each intests/test_evals.py(dataset + report) andtests/test_evals_agreement.py(agreement CLI).Why
A file written by Notepad, Excel or most Windows editors starts with a three-byte UTF-8 BOM. Read as plain
utf-8, those bytes survive as U+FEFF at the start of the document, andjsonrefuses them -- with a message that names the fix this library could have applied itself:So a dataset exported from a spreadsheet is unrunnable, and the refusal tells the user to change the encoding of the file rather than changing three characters of the reader. The repository already decided this question everywhere else a caller hands over a file:
laya/cli.pyopens--questions,--batchand stdin withutf-8-sigand its comment says why -- "a questions file written by Notepad or Excel starts with a UTF-8 BOM, and a plainutf-8read leaves it as U+FEFF, sojson.loadrefuses a file that is valid JSON -- with its own 'decode using utf-8-sig' hint. The other file readers in this repo open the same way for the same reason" -- andlaya/train.py(whose docstring says it usesutf-8-sig"so UTF-8 BOM headers from Excel are handled transparently"),laya/evidence.pyanddocker/entrypoint.pydo. The evals doors were the exception to a rule the rest of the codebase already follows, which is what makes a dataset read and a--questionsread disagree about the same file on the same machine.Behaviour is otherwise identical:
utf-8-sigstrips a leading BOM and decodes every BOM-less file exactly asutf-8did, so all the refusal paths already tested (not-JSON, non-UTF-8 bytes, a directory, an unreadable volume, unclosable depth) are unchanged -- the new tests assert one of them still refuses in the same words.How it was verified
All three leaks were reproduced at base before the fix (the three
EvalErrortexts above are that output, verbatim). Thenlaya/evals.pyandlaya/evals_cli.pywere reverted to base with the new tests in place:The agreement case is a real subprocess run of
laya-evals agreement(returncode == 2at base,0with the fix) and asserts the published report equalscompare(runs)byte for byte -- the BOM changes nothing about the numbers, so it must change nothing about the verdict. Both files were restored byte-identical from/tmpcopies and the suites re-run from disk afterwards.Checklist
main