Skip to content

fix(research): make the Chinese benchmark audit platform-independent - #216

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HankGrimm:fix/feishu-audit-cross-platform
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
HankGrimm:fix/feishu-audit-cross-platform

Conversation

@HankGrimm

@HankGrimm HankGrimm commented Sep 23, 2026 •

Copy link
Copy Markdown

The feishu-benchmark CI job runs audit.py and its unittest suite with no dependencies, which makes it the one suite a contributor can run anywhere. It cannot run on a platform whose locale encoding is not UTF-8, for two independent reasons.

1. Text IO without an explicit encoding. read_text() / write_text() fall back to the locale encoding, so reading the Chinese data/cases.jsonl raises UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d on Windows (cp1252). On Linux nothing changes — encoding='utf-8' is what the locale already supplied.

2. No .gitattributes. SOURCE.json pins a sha256 of the raw bytes of the frozen dataset and of every archived run, and verify_archive() checks them with read_bytes(). With git's default core.autocrlf=true on Windows those files check out as CRLF, so the integrity check fails on a clean clone: ValueError: Archived source bytes changed: data/cases.jsonl. All nine archived files hash correctly only with LF, so the archive is pinned to eol=lf. The stored blobs are already LF; this only stops a checkout from rewriting them.

Writers (run.py, and audit.py --write-summary) also get newline='\n', so a run produced off Linux stays byte-comparable with the committed archive instead of silently differing by line ending.

No logic and no numbers changed.

Verification

On Windows 11, Python 3.11, cp1252 locale — both CI steps fail on main and pass after the change:

python research/benchmarks/feishu_zh/audit.py
  jev / choice: 64/64 · jev / four_noul: 63/64
  laya / choice: 20/64 · laya / four_noul: 18/64

python -m unittest discover -s research/benchmarks/feishu_zh/tests
  Ran 6 tests — OK

And the strongest check: audit.py --write-summary on Windows now reproduces results/v1/summary.json byte-for-byte (sha256 ec4a60b4…), identical to the committed Linux-produced archive. Before the change it raised UnicodeDecodeError.

py_compile clean on all four touched scripts. git ls-files --eol reports i/lf w/lf for every archived file.

Related but separate: #212 adds a Windows lane for tests/, which does not cover this job.

The feishu-benchmark CI job runs audit.py and its unittest suite with no
dependencies, but it cannot run on a platform whose locale encoding is not
UTF-8, for two independent reasons.

1. read_text()/write_text() without an explicit encoding fall back to the
   locale encoding, so reading the Chinese data/cases.jsonl raises
   UnicodeDecodeError on Windows (cp1252).

2. SOURCE.json pins a sha256 of the raw bytes of the frozen dataset and of
   every archived run, and verify_archive() checks them with read_bytes().
   With git's default core.autocrlf=true on Windows those files check out as
   CRLF, so the integrity check fails on a clean clone. All nine archived
   files hash correctly only with LF, so the archive is pinned to eol=lf.

Writers also get newline='\n', so a run produced off Linux stays
byte-comparable with the committed archive. No logic and no numbers changed.

Verified on Windows 11, Python 3.11, cp1252: both CI steps fail on main and
pass after the change, and audit.py --write-summary now reproduces
results/v1/summary.json byte-for-byte.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@aashish254

aashish254 commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Short answer: on a UTF-8-locale platform nothing changes, exactly as the PR describes - from the macOS side this is good to merge.

Independent check on macOS (Darwin arm64, Python 3.14.4), checked out at 033d013 on top of 010bace (v0.3.7):

  • python research/benchmarks/feishu_zh/audit.py - exit 0, 0 failed requests across all four suites
  • python -m unittest discover -s research/benchmarks/feishu_zh/tests - 6/6 OK
  • git check-attr confirms text eol=lf applies to the frozen dataset, and the sha256 archive verification inside audit.py passes on a fresh checkout

The explicit encoding='utf-8' reads and the LF pinning behave as advertised, and the .gitattributes comments document why.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, and thanks @aashish254 for the independent macOS check. The feishu audit is the one suite that runs with no dependencies, so it should run on every platform. Explicit UTF-8 plus .gitattributes for the frozen data does that. Merging.

@NandhaKishorM
NandhaKishorM merged commit 885ba78 into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…andhaKishorM#216)

The feishu-benchmark CI job runs audit.py and its unittest suite with no
dependencies, but it cannot run on a platform whose locale encoding is not
UTF-8, for two independent reasons.

1. read_text()/write_text() without an explicit encoding fall back to the
   locale encoding, so reading the Chinese data/cases.jsonl raises
   UnicodeDecodeError on Windows (cp1252).

2. SOURCE.json pins a sha256 of the raw bytes of the frozen dataset and of
   every archived run, and verify_archive() checks them with read_bytes().
   With git's default core.autocrlf=true on Windows those files check out as
   CRLF, so the integrity check fails on a clean clone. All nine archived
   files hash correctly only with LF, so the archive is pinned to eol=lf.

Writers also get newline='\n', so a run produced off Linux stays
byte-comparable with the committed archive. No logic and no numbers changed.

Verified on Windows 11, Python 3.11, cp1252: both CI steps fail on main and
pass after the change, and audit.py --write-summary now reproduces
results/v1/summary.json byte-for-byte.

Co-authored-by: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants