Skip to content

perf: make lazy Router language switches cheap — skip throwaway weight init, cache tokenizers - #141

Closed
fwt1888 wants to merge 2 commits into
NandhaKishorM:mainfrom
fwt1888:perf/skip-random-init
Closed

fwt1888 wants to merge 2 commits into
NandhaKishorM:mainfrom
fwt1888:perf/skip-random-init

Conversation

@fwt1888

@fwt1888 fwt1888 commented Sep 22, 2026

Copy link
Copy Markdown

What

A cold laya.load() was spending most of its time on work that gets thrown away. Two things, both measured on the same 3-thread CPU box:

1. Random initialisation of weights the checkpoint immediately overwrites.

build_model() calls AutoModel.from_config(), which allocates and randomly initialises every parameter. load_state_dict() then overwrites all of them. On a 421M-parameter checkpoint that init was ~17 s of a ~19 s load, against ~1.1 s to read the 0.84 GB of safetensors and ~0.3 s to apply them.

The fix pins parameters to the meta device during construction so no storage is allocated and no initialisation runs.

2. Re-parsing tokenizer.json on every Agent().

huggingface_hub caches the download but not the parsed tokenizer, so every construction re-parsed it. That is negligible for English (3.6 MB) but the multilingual checkpoint ships a 34 MB, 256k-vocabulary file costing ~4.5 s — several times the cost of applying its weights, and the largest remaining term once (1) lands.

The fix parses once per tokenizer directory per process. A tokenizer is read-only during inference, so one instance is safely shared — including by an Agent the Router has just evicted and is about to rebuild.

Why this matters

With the Router's default max_loaded=1, traffic that alternates languages rebuilds an Agent on every request. That is the reload the README warns about in "Production Preload & Memory". Preloading still removes the reload entirely and remains the right production answer; this makes the reload itself cheap, which also helps max_loaded=2..3 deployments and cold starts.

Measurements

Agent(work, device="cpu"), median of 3, after a warmup construction:

checkpoint before after
english (ModernBERT-large, 421M) 20.34 s 0.27 s 75x
multilingual (mmBERT-base, 322M) 22.20 s 0.28 s 79x
typed-decisions (ModernBERT-large, 421M) 17.45 s 0.39 s 45x

Reproduce with the new research/scripts/bench_load_latency.py.

Per-step breakdown of the English checkpoint, before → after:

step before after
AutoTokenizer.from_pretrained 0.29 s ~0 s (cached)
build_model (random init) 17.20 s 0.05 s (meta)
safetensors.load_file (0.84 GB) 1.03 s 0.76 s
_verify_compatibility 0.00 s 0.00 s
load_state_dict(strict=True) 0.33 s 0.40 s
total 18.85 s ~0.3 s

Correctness

Verified against a pre-change reference on the real English checkpoint: system_one() output is identical field by field, including probabilities and token counts, and all three shipped checkpoints produce their expected argmax after the change.

The meta path is only safe because of a subtlety worth calling out: buffers must not be skipped. ModernBERT computes its non-persistent RoPE inv_freq buffers in __init__, they are absent from every checkpoint, and forward() reads them. The obvious implementation — build on meta, then nn.Module.to_empty() — re-creates the buffers as zeros, silently leaving the encoder attending with a dead rotary embedding. No exception, just wrong logits. So materialise_meta_parameters() converts parameters only and leaves buffers untouched.

Three tests pin this down, all offline and fast (tests/test_decision_model.py):

  • empty=True defers parameters to meta while buffers are real, non-zero, and bit-identical to a normal build
  • materialisation does not disturb the buffers (fails loudly under the to_empty() implementation — I mutation-tested it)
  • build-empty → materialise → load_state_dict reproduces a normal build's logits exactly

tests/test_local_e2e.py gains a tokenizer-reuse check covering per-checkpoint sharing, cross-checkpoint isolation, and cache invalidation when the on-disk config is edited.

Notes

  • build_model(..., empty=True) is opt-in; the default empty=False path is unchanged, so this does not alter any existing caller.
  • The tokenizer cache key is (absolute directory, tokenizer_config.json mtime), so _fix_tokenizer_config() rewriting it or a re-download still triggers a fresh parse. Remote ids are deliberately never cached, so a caller cannot pin a stale revision.
  • The cache holds one tokenizer per checkpoint for the process lifetime (~12 MB serialised for the 256k one). It is a separate commit (perf: parse each checkpoint's tokenizer once per process), so it can be dropped with a one-line rebase if you would rather not carry process-level state.
  • Full CI suite passes (test_router, test_criteria, test_download, test_shortlist, test_decision_model, test_packaging, test_email), ruff check with the repo's selectors is clean, compileall clean.

wangtiantian added 2 commits September 22, 2026 16:19
A cold laya.load() spent most of its time on work that is thrown away: the
model was constructed with AutoModel.from_config(), which random-initialises
every parameter, and load_state_dict() then overwrote all of them.

On a 3-thread CPU box the ~19 s cold load broke down as ~17 s of construction
against ~1.1 s to read 0.84 GB of safetensors and ~0.3 s to apply them.
Pinning the parameters to the meta device during construction removes the
wasted initialisation while leaving every buffer intact.

Buffers must NOT be skipped: ModernBERT computes its non-persistent RoPE
inv_freq buffers in __init__, they are absent from every checkpoint, and
forward() reads them. Materialising with nn.Module.to_empty() zeroes them,
which leaves the encoder silently attending with a dead rotary embedding, so
materialise_meta_parameters() converts parameters only.
huggingface_hub caches the download but not the parsed tokenizer, so every
Agent() re-parsed tokenizer.json. For the multilingual checkpoint that file is
34 MB / 256k vocab and costs ~4.5 s to load -- several times the cost of
applying its weights, and the single largest remaining term in a cold load.

A tokenizer is read-only during inference, so one instance is shared per
tokenizer directory, keyed on the directory and its mtime so a rewritten
config or a re-download still triggers a fresh parse. Remote ids are not
cached, so a caller cannot pin a stale revision.

Measured cold Agent() construction on the same CPU box, after both changes:

  english           0.71 s -> 0.27 s
  multilingual      4.36 s -> 0.28 s
  typed-decisions   0.73 s -> 0.39 s
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for measuring where cold loads actually go. Your first finding landed today in another form: #195 builds the model under no_init_weights (22 s to 1.5 s, outputs identical). The tokenizer caching is still valuable on its own, so if you rebase onto main and keep just that part, I'd be glad to review it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants