Repository navigation
perf: parse each checkpoint's tokenizer once per process - #225
Merged
Merged
Conversation
`huggingface_hub` caches the download but not the parsed tokenizer, so every `Agent()` re-parsed `tokenizer.json`. For the multilingual checkpoint that file is 34 MB / 256k vocab and costs several seconds to load -- several times the cost of applying its weights, and now the largest remaining term in a cold load. A tokenizer is read-only during inference, so one instance is shared per tokenizer directory, keyed on the directory and its mtime so a rewritten config or a re-download still triggers a fresh parse. Remote ids are not cached, so a caller cannot pin a stale revision.
Owner
|
Thank you for splitting this out of #141. Keying the cache on the directory and the config mtime means a re-download or a rewritten config still re-parses, and not caching hub ids avoids pinning a stale revision. I checked it on the English checkpoint: a second Agent loads in 0.5 s instead of 2.9 s, and its answers on six states are identical to |
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…rM#225) `huggingface_hub` caches the download but not the parsed tokenizer, so every `Agent()` re-parsed `tokenizer.json`. For the multilingual checkpoint that file is 34 MB / 256k vocab and costs several seconds to load -- several times the cost of applying its weights, and now the largest remaining term in a cold load. A tokenizer is read-only during inference, so one instance is shared per tokenizer directory, keyed on the directory and its mtime so a rewritten config or a re-download still triggers a fresh parse. Remote ids are not cached, so a caller cannot pin a stale revision. Co-authored-by: wangtiantian <wang.tiantian2@zte.com.cn>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #141, rebased onto
main: keeps only the tokenizer caching.huggingface_hubcaches the download but not the parsed tokenizer, so everyAgent()re-parsedtokenizer.json— 34 MB / 256k vocab on the multilingual checkpoint, several seconds, and now the largest term in a cold load. This parses it once per tokenizer directory (keyed on the directory and its config mtime, so a re-download or rewritten config still re-parses) and shares it across Agents, including one the Router rebuilds after eviction. Hub ids are not cached, so a caller cannot pin a stale revision.The
no_init_weightshalf of #141 landed separately as #195, so it is dropped here. Adds a regression test intests/test_local_e2e.py.