Skip to content

perf: parse each checkpoint's tokenizer once per process - #225

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
fwt1888:perf/cache-tokenizers
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
fwt1888:perf/cache-tokenizers

Conversation

@fwt1888

@fwt1888 fwt1888 commented Sep 23, 2026

Copy link
Copy Markdown

Follow-up to #141, rebased onto main: keeps only the tokenizer caching.

huggingface_hub caches the download but not the parsed tokenizer, so every Agent() re-parsed tokenizer.json — 34 MB / 256k vocab on the multilingual checkpoint, several seconds, and now the largest term in a cold load. This parses it once per tokenizer directory (keyed on the directory and its config mtime, so a re-download or rewritten config still re-parses) and shares it across Agents, including one the Router rebuilds after eviction. Hub ids are not cached, so a caller cannot pin a stale revision.

The no_init_weights half of #141 landed separately as #195, so it is dropped here. Adds a regression test in tests/test_local_e2e.py.

`huggingface_hub` caches the download but not the parsed tokenizer, so every
`Agent()` re-parsed `tokenizer.json`. For the multilingual checkpoint that file
is 34 MB / 256k vocab and costs several seconds to load -- several times the
cost of applying its weights, and now the largest remaining term in a cold load.

A tokenizer is read-only during inference, so one instance is shared per
tokenizer directory, keyed on the directory and its mtime so a rewritten config
or a re-download still triggers a fresh parse. Remote ids are not cached, so a
caller cannot pin a stale revision.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for splitting this out of #141. Keying the cache on the directory and the config mtime means a re-download or a rewritten config still re-parses, and not caching hub ids avoids pinning a stale revision. I checked it on the English checkpoint: a second Agent loads in 0.5 s instead of 2.9 s, and its answers on six states are identical to main. Merging.

@NandhaKishorM
NandhaKishorM merged commit 819d81d into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…rM#225)

`huggingface_hub` caches the download but not the parsed tokenizer, so every
`Agent()` re-parsed `tokenizer.json`. For the multilingual checkpoint that file
is 34 MB / 256k vocab and costs several seconds to load -- several times the
cost of applying its weights, and now the largest remaining term in a cold load.

A tokenizer is read-only during inference, so one instance is shared per
tokenizer directory, keyed on the directory and its mtime so a rewritten config
or a re-download still triggers a fresh parse. Remote ids are not cached, so a
caller cannot pin a stale revision.

Co-authored-by: wangtiantian <wang.tiantian2@zte.com.cn>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants