Skip to content

fix(evals): write redirected output as utf-8 - #1045

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
plox-sumit:fix/evals-stdout-utf8
Oct 8, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
plox-sumit:fix/evals-stdout-utf8

Conversation

@plox-sumit

Copy link
Copy Markdown
Contributor

What

laya-evals and laya eval now write stdout as utf-8, so output sent to a file or a pipe can hold a non-Latin question id, tag or language. This is the follow-up named at the end of #893, which did the same for laya's own commands.

Why

Redirected output is encoded with the locale's codec by default. On Windows that is cp1252 (Python before 3.15), which cannot encode most non-Latin text. validate prints the dataset's question ids, and run --slice prints tags, languages and question ids, all of which come from the user's dataset. #893 did not cover this path: laya eval returns from cli.main() before the line that sets stdout, and laya-evals has its own main().

Reproduced on Windows 11 with Python 3.12 and a one-row dataset whose question id is Hindi (श्रेणी):

laya-evals validate ds.jsonl > out.txt

before:  exit 1, out.txt is empty, and a traceback ending in
         UnicodeEncodeError: 'charmap' codec can't encode characters in position 30-35: character maps to <undefined>
after:   exit 0, out.txt holds
         1 examples, 1 question id(s): श्रेणी

laya eval validate ds.jsonl > out.txt fails the same way before and writes the same line after. The exit code matters here: docs/evals.md keeps 1 for a failed threshold or baseline tolerance, so a CI job on a Windows runner reads this crash as "the model got worse" when the dataset is valid.

The fix is sys.stdout.reconfigure(encoding="utf-8") at the top of evals_cli.main(). It is guarded with hasattr, like the one in cli.main(), because tests replace stdout.

One behaviour change to know about, the same as in #893: on Windows, redirected output with accented Latin text (é, ü) used to be written as cp1252 and is now utf-8.

How it was verified

python -m pytest tests/test_evals.py -q
python -m pytest tests/test_evals_shortlist.py -q
python tests/test_evals_api.py
python tests/test_evals_onnx.py
python tests/test_cli.py
python tests/test_cli_lang_guess.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
  • tests/test_evals.py: 138 passed. On main without this PR: 137 passed. With only the evals_cli.py change removed, the new test fails with a UnicodeEncodeError at the print in _cmd_validate.
  • The new test runs validate through both entry points with stdout wrapped in a latin-1 writer, so it fails on any OS without the fix, not only on Windows.
  • tests/test_evals_shortlist.py: 7 passed. tests/test_evals_onnx.py: 33 passed, 0 failed. tests/test_evals_api.py, tests/test_cli.py and tests/test_cli_lang_guess.py: all passed.
  • ruff: all checks passed. compileall: clean.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed (no API change)

Left out on purpose:

  • I did not run laya-evals run --slice tag end to end with a non-Latin tag, because that needs a checkpoint. It prints through the same stdout, so the same line covers it.
  • laya-train prints the output folder at the end (Training complete! Model saved to ...). A non-Latin folder name could hit the same problem when the output is redirected. I have not reproduced it, so it is not in this PR.

Redirected stdout is encoded with the locale's codec by default (cp1252 on
Windows), which cannot hold a non-Latin question id, tag or language, so
`laya-evals validate data.jsonl > out.txt` died with a traceback and exit 1,
the code kept for a quality failure. evals_cli.main() now reconfigures stdout
to utf-8, as cli.main() does; `laya eval` returns before that line.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@plox-sumit

Copy link
Copy Markdown
Contributor Author

A note on CI, since six checks are red and this PR only touches laya/evals_cli.py and tests/test_evals.py:

  • The Python jobs that exercise this change all pass: tests on 3.10 to 3.13 and on Windows, evals, lint and both smoke jobs.
  • The four JVM jobs and the .NET build + model-free tests job are also red on main at 3cf26cb. I compared the jdk 21 log on both sides: the same two tests fail there and here (DecoderTest > empty action logits produce neutral probabilities and LanguageDetectionParityTest > ... english-note-with-acronym-line-and-foreign-line). The .NET job fails here on seven LayaRouterTests.EveryRouteProbeEntry cases that expect multilingual.
  • parity (multilingual) passed on main a few hours earlier. Here the .NET tokenizer tests failed with Failed to load tokenizer: EOF while parsing a value at line 1 column 0, which reads like an empty tokenizer download. A re-run should clear it; I cannot start one from the fork.

@NandhaKishorM
NandhaKishorM merged commit b955f82 into NandhaKishorM:main Oct 8, 2026
30 of 36 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged.

The exit code is the part that makes this more than a cosmetic encoding fix, and you are right to lead with it. docs/evals.md keeps exit 1 for a failed threshold or a baseline tolerance, so a Windows CI job reading a UnicodeEncodeError crash sees "the model got worse" when the dataset is simply valid and has a non-Latin question id. That is a false regression signal, and it is the kind that gets a good change reverted.

Your reproduction is the right one: a one-row dataset whose question id is Hindi, redirected to a file, before and after, with both the exit code and the file contents shown.

Explaining why #893 did not already cover this is what made it quick to review: laya eval returns from cli.main() before the line that sets stdout, and laya-evals has its own main(). Two entry points, one of them bypassing the fix, is exactly the shape that leaves a follow-up.

This lands in the same release as #1001, which stopped tests/ reading repo pages with the runner's codec. Same root cause from the other direction: Python's default encoding is the locale's, and the Windows locale is cp1252.

All 85 suites pass on the merged tree.

NandhaKishorM added a commit that referenced this pull request Oct 8, 2026
35 pull requests from 8 contributors.

* LAYA_EXTRA_MODELS registers extra checkpoints on laya-serve (#1047, #919)
* fine-tuning prints its optimizer-update budget before a run (#1014, #963)
* the Kaggle DDP notebook converges on the shared training primitives (#1017, #887)
* evals writes redirected output as utf-8 (#1045)
* the server serves again after a lifespan restart (#1049)
* Java router lifecycle hooks, plus an OSV dependency scan (#1046)
* twenty-four documentation corrections, each verified against the code

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants