Repository navigation
Conversation
added 2 commits
September 22, 2026 16:19
A cold laya.load() spent most of its time on work that is thrown away: the model was constructed with AutoModel.from_config(), which random-initialises every parameter, and load_state_dict() then overwrote all of them. On a 3-thread CPU box the ~19 s cold load broke down as ~17 s of construction against ~1.1 s to read 0.84 GB of safetensors and ~0.3 s to apply them. Pinning the parameters to the meta device during construction removes the wasted initialisation while leaving every buffer intact. Buffers must NOT be skipped: ModernBERT computes its non-persistent RoPE inv_freq buffers in __init__, they are absent from every checkpoint, and forward() reads them. Materialising with nn.Module.to_empty() zeroes them, which leaves the encoder silently attending with a dead rotary embedding, so materialise_meta_parameters() converts parameters only.
huggingface_hub caches the download but not the parsed tokenizer, so every Agent() re-parsed tokenizer.json. For the multilingual checkpoint that file is 34 MB / 256k vocab and costs ~4.5 s to load -- several times the cost of applying its weights, and the single largest remaining term in a cold load. A tokenizer is read-only during inference, so one instance is shared per tokenizer directory, keyed on the directory and its mtime so a rewritten config or a re-download still triggers a fresh parse. Remote ids are not cached, so a caller cannot pin a stale revision. Measured cold Agent() construction on the same CPU box, after both changes: english 0.71 s -> 0.27 s multilingual 4.36 s -> 0.28 s typed-decisions 0.73 s -> 0.39 s
Owner
|
Thank you for measuring where cold loads actually go. Your first finding landed today in another form: #195 builds the model under |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A cold
laya.load()was spending most of its time on work that gets thrown away. Two things, both measured on the same 3-thread CPU box:1. Random initialisation of weights the checkpoint immediately overwrites.
build_model()callsAutoModel.from_config(), which allocates and randomly initialises every parameter.load_state_dict()then overwrites all of them. On a 421M-parameter checkpoint that init was ~17 s of a ~19 s load, against ~1.1 s to read the 0.84 GB of safetensors and ~0.3 s to apply them.The fix pins parameters to the
metadevice during construction so no storage is allocated and no initialisation runs.2. Re-parsing
tokenizer.jsonon everyAgent().huggingface_hubcaches the download but not the parsed tokenizer, so every construction re-parsed it. That is negligible for English (3.6 MB) but the multilingual checkpoint ships a 34 MB, 256k-vocabulary file costing ~4.5 s — several times the cost of applying its weights, and the largest remaining term once (1) lands.The fix parses once per tokenizer directory per process. A tokenizer is read-only during inference, so one instance is safely shared — including by an Agent the
Routerhas just evicted and is about to rebuild.Why this matters
With the
Router's defaultmax_loaded=1, traffic that alternates languages rebuilds an Agent on every request. That is the reload the README warns about in "Production Preload & Memory". Preloading still removes the reload entirely and remains the right production answer; this makes the reload itself cheap, which also helpsmax_loaded=2..3deployments and cold starts.Measurements
Agent(work, device="cpu"), median of 3, after a warmup construction:english(ModernBERT-large, 421M)multilingual(mmBERT-base, 322M)typed-decisions(ModernBERT-large, 421M)Reproduce with the new
research/scripts/bench_load_latency.py.Per-step breakdown of the English checkpoint, before → after:
AutoTokenizer.from_pretrainedbuild_model(random init)safetensors.load_file(0.84 GB)_verify_compatibilityload_state_dict(strict=True)Correctness
Verified against a pre-change reference on the real English checkpoint:
system_one()output is identical field by field, including probabilities and token counts, and all three shipped checkpoints produce their expected argmax after the change.The
metapath is only safe because of a subtlety worth calling out: buffers must not be skipped. ModernBERT computes its non-persistent RoPEinv_freqbuffers in__init__, they are absent from every checkpoint, andforward()reads them. The obvious implementation — build onmeta, thennn.Module.to_empty()— re-creates the buffers as zeros, silently leaving the encoder attending with a dead rotary embedding. No exception, just wrong logits. Somaterialise_meta_parameters()converts parameters only and leaves buffers untouched.Three tests pin this down, all offline and fast (
tests/test_decision_model.py):empty=Truedefers parameters tometawhile buffers are real, non-zero, and bit-identical to a normal buildto_empty()implementation — I mutation-tested it)load_state_dictreproduces a normal build's logits exactlytests/test_local_e2e.pygains a tokenizer-reuse check covering per-checkpoint sharing, cross-checkpoint isolation, and cache invalidation when the on-disk config is edited.Notes
build_model(..., empty=True)is opt-in; the defaultempty=Falsepath is unchanged, so this does not alter any existing caller.tokenizer_config.jsonmtime), so_fix_tokenizer_config()rewriting it or a re-download still triggers a fresh parse. Remote ids are deliberately never cached, so a caller cannot pin a stale revision.perf: parse each checkpoint's tokenizer once per process), so it can be dropped with a one-line rebase if you would rather not carry process-level state.test_router,test_criteria,test_download,test_shortlist,test_decision_model,test_packaging,test_email),ruff checkwith the repo's selectors is clean,compileallclean.