Repository navigation
fix(agent): don't rewrite tokenizer_config through the HuggingFace cache symlink - #107
Conversation
|
Thank you, this is a real bug: writing through the snapshot symlink mutates the shared blob that other revisions and processes read. Today's |
HuggingFace snapshots are symlinks into a shared blobs/ store, so open(path, "w") truncated the shared blob: it mutated a file shared with other revisions/processes, raced the first concurrent load, and desynced the hub cache metadata. Detach the local file before writing. Adds tests/test_tokenizer_cache.py and runs it in CI.
The broad 'except Exception: pass' hid failures: if the patch did not apply, AutoTokenizer would fail later with a confusing error and no hint the config was the cause. Emit a RuntimeWarning with the path and reason. Covered by a new malformed-JSON case in tests/test_tokenizer_cache.py.
79036e2 to
6ae53c8
Compare
|
Thank you for rebasing. Writing through the snapshot symlink truncates a blob shared with other revisions and processes, so detaching the local file first is the right fix, and a warning instead of a silent |
|
thanks! good call on the gap between unlink and write, i'll do the temp file + os.replace follow-up in a small PR. |
…che symlink (NandhaKishorM#107) * fix(agent): don't rewrite tokenizer_config through the hub-cache symlink HuggingFace snapshots are symlinks into a shared blobs/ store, so open(path, "w") truncated the shared blob: it mutated a file shared with other revisions/processes, raced the first concurrent load, and desynced the hub cache metadata. Detach the local file before writing. Adds tests/test_tokenizer_cache.py and runs it in CI. * fix(agent): warn when tokenizer_config cannot be patched The broad 'except Exception: pass' hid failures: if the patch did not apply, AutoTokenizer would fail later with a confusing error and no hint the config was the cause. Emit a RuntimeWarning with the path and reason. Covered by a new malformed-JSON case in tests/test_tokenizer_cache.py. --------- Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
What
_fix_tokenizer_configpatchestokenizer/tokenizer_config.jsonin place to keep thecheckpoint loadable across transformers versions.
huggingface_hub.snapshot_downloadlays snapshots out as symlinks into a shared
blobs/store, soopen(path, "w")truncates the shared blob rather than a local file.
Why it matters
Writing through the link mutates a file shared with other revisions and other processes,
races the first concurrent load (e.g. multiple server workers all calling
Agent(...)atonce), and leaves the hub cache metadata (
refs/etag) out of sync with the file on disk.Reproduction
Change
Also: the function wrapped everything in
except Exception: pass. If the patch fails (forexample an unreadable or malformed config), that silently skips it and
AutoTokenizerthenfails later with no hint that the config was the cause. It now emits a
RuntimeWarningnaming the file and the reason, and leaves the file untouched.
Verification
python tests/test_tokenizer_cache.py(new): asserts the blob is byte-identical, thesymlink is replaced by a regular file, the local file carries the patched value, and a
malformed config warns instead of raising.
python tests/test_router.py(150),python tests/test_criteria.py(34).ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120,compileall.