Repository navigation
fix(router): keep two checkpoints by default, not one - #180
Merged
NandhaKishorM merged 1 commit intoSep 23, 2026
Merged
NandhaKishorM merged 1 commit into
NandhaKishorM merged 1 commit into
Conversation
`Router` defaults to `max_loaded=1`, so a workload whose requests alternate languages evicts and rebuilds a checkpoint on every request. NandhaKishorM#172 measured that at 20-23 s per request on CPU against 49-136 ms with both resident, and reported it upstream. Automatic routing only ever chooses between `english` and `multilingual`: `typed-decisions` needs `auto_task_detection=True`, an explicit `model=` or an explicit `task=`. A cap of one therefore guarantees a full rebuild on exactly the mixed-script traffic the Router exists for, while a cap of two costs nothing to a deployment that only ever sees one language, because checkpoints load lazily and the second is never built. `max_loaded=1` stays available for a memory-constrained host, `3`/`preload` for traffic that also reaches `typed-decisions`; all three are documented in the class docstring and the README's deployment table, which both described the old default. Verification (offline, no weights): * tests/test_router.py 154 passed, 0 failed (150 before). The new checks drive a stub loader over the reported alternating workload: cap=1 builds 20 checkpoints for 20 requests, cap=2 builds 2, and five English-only requests build one checkpoint, which is the memory argument for the default. * Other model-free suites unchanged and passing; ruff with CI's rule set and python -m compileall clean. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
Owner
|
Thanks @devloper961-maker. This is the second half of #172, and the reasoning holds up: automatic routing only ever picks between two checkpoints, and a single-language deployment never builds the second, so the new default only costs memory for traffic that needs it. All model-free suites pass locally, alone and together with the other routing PRs landing today. Merging. |
This was referenced Sep 23, 2026
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
) `Router` defaults to `max_loaded=1`, so a workload whose requests alternate languages evicts and rebuilds a checkpoint on every request. NandhaKishorM#172 measured that at 20-23 s per request on CPU against 49-136 ms with both resident, and reported it upstream. Automatic routing only ever chooses between `english` and `multilingual`: `typed-decisions` needs `auto_task_detection=True`, an explicit `model=` or an explicit `task=`. A cap of one therefore guarantees a full rebuild on exactly the mixed-script traffic the Router exists for, while a cap of two costs nothing to a deployment that only ever sees one language, because checkpoints load lazily and the second is never built. `max_loaded=1` stays available for a memory-constrained host, `3`/`preload` for traffic that also reaches `typed-decisions`; all three are documented in the class docstring and the README's deployment table, which both described the old default. Verification (offline, no weights): * tests/test_router.py 154 passed, 0 failed (150 before). The new checks drive a stub loader over the reported alternating workload: cap=1 builds 20 checkpoints for 20 requests, cap=2 builds 2, and five English-only requests build one checkpoint, which is the memory argument for the default. * Other model-free suites unchanged and passing; ruff with CI's rule set and python -m compileall clean. 🤖 Generated with Codebuff Co-authored-by: Nikhi <nikhi@example.com> Co-authored-by: Codebuff <noreply@codebuff.com>
NandhaKishorM
pushed a commit
that referenced
this pull request
Sep 27, 2026
examples/server.py built its Router with max_loaded=1, a copy of a default laya/router.py retired in #180. Unset LAYA_MAX_LOADED now means "not asked for": the key is left out of the constructor so the library owns the default, an explicit value (env or --max-loaded) is still honoured, /health reports the cap the running Router holds rather than the requested one, and the --reload env push pops the variable instead of writing str(None) for the reimported process to read with int().
4 tasks done
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 2, 2026
… defaults to The banner opened with "At the default `max_loaded=1`". laya/router.py takes max_loaded: int = 2 (the old default was retired in #180) and README.md documents "up to two checkpoints resident by default", so the page that teaches residency taught the retired number -- and taught readers that they had to raise the cap to avoid the churn it was demonstrating. The default is now read from a Router instance instead of typed in here: building one loads no checkpoint, so it costs nothing and cannot drift again. Section 1 labels the cap it pins as pinned, next to the library's default. The closing rule of thumb also claimed the reload is "thousands of times more expensive than the routing itself" with nothing measured behind it. Both sides are timed now -- the cold build in seconds and a route() through _common.timed(), which loads nothing -- and the page prints their ratio: 0.8 s against 0.031 ms, 27199 times on this machine.
4 tasks done
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 11, 2026
…nglish examples/server.py constructed Router(default="english") whenever LAYA_DEFAULT_MODEL was unset. 3cf26cb moved Router's own default to multilingual in 0.4.0, and laya.serve follows it by sending no default keyword at all -- which laya/mcp/server.py reuses. One environment, three answers: a bare laya-serve and a bare laya mcp route a state with no language evidence to the multilingual checkpoint, a bare `python examples/server.py` still sent it to the English arm. Unset now means "not asked for", the same rule this file already follows for LAYA_MAX_LOADED (#180 retired its copy of that number for the same reason). The /models payload reports the built Router's answer instead of the requested one, and the --reload push leaves the variable unpushed rather than writing a checkpoint literally named "None" into the child. tests/test_example_server_limits.py gains 32 checks: the unset and blank spellings at both surfaces, the running Router's default, the route() decision for the three ambiguous states 3cf26cb names, /models in JSON and HTML, four operator spellings that must still be honored, the laya.serve parity claim, and the --reload round trip. 122 -> 154 checks.
This was referenced Oct 20, 2026
Merged
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 21, 2026
Python's Router defaults to max_loaded=2 (#180), so a router that alternates two languages keeps both resident. LayaRouterOptions' max_loaded defaulted to 1, which made every alternation evict and rebuild a checkpoint.
NandhaKishorM
pushed a commit
that referenced
this pull request
Oct 21, 2026
Python's Router defaults to max_loaded=2 (#180), so a router that alternates two languages keeps both resident. LayaRouterOptions.MaxLoaded defaulted to 1 while its doc comment claimed to match Python, which made every alternation evict and reload a checkpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of #172 (the routing half is #178).
Routerdefaults tomax_loaded=1, so a workload whose requests alternate languages evicts and rebuilds a checkpoint on every request. #172 measured that at 20-23 s per request on CPU against 49-136 ms with both resident — a 150-400× difference on exactly the traffic a router exists for.Why 2
Automatic routing only ever chooses between two checkpoints:
englishandmultilingual.typed-decisionsis reached only throughauto_task_detection=True, an explicitmodel=or an explicittask=, as the module docstring already says. So at the default, a mixed-script request stream is guaranteed to thrash, and nothing about the payload can avoid it.Two is also the default that costs nothing to the deployments the old default was protecting: checkpoints load lazily, so a service that only ever sees one language never builds the second. The memory argument for
1only applies to traffic that actually uses a second checkpoint — which is the traffic1makes unusable.max_loaded=1stays for a memory-constrained host and3/preloadfor traffic that also reachestyped-decisions. Both the class docstring and the README's deployment table described the old default, so all three are named there now, including what the default does and does not cost:Router()(lazy,max_loaded=2)Router(max_loaded=1)Router(preload=True)Preloading is still the recommendation for a server, and this does not change that — the first load of each language is still seconds.
Verification
Offline, no weights.
tests/test_router.pygoes from 150 passed / 0 failed to 154 passed / 0 failed: the new checks drive a stub loader over the reported alternating workload and assert 20 builds atmax_loaded=1against 2 at the default, then assert that English-only traffic builds one checkpoint. The other model-free suites are unchanged and pass,ruffwith CI's rule set is clean,python -m compileall -q laya/ tests/passes.Touches
laya/router.py,README.mdandtests/test_router.pyonly; it does not touchlaya/lang.py, so it does not interact with #178/#179. I merged all three branches locally to check: they apply cleanly, the combined routing suite is 221 passed / 0 failed, andRouter().max_loadedis 2 with all three applied.Not addressed: #141 (make a lazy language switch cheaper — no weight re-read) and #138 (group a heterogeneous batch by checkpoint) reduce the same symptom from the other side and are complementary; with either of those,
max_loaded=1becomes less painful but still rebuilds.🤖 Generated with Codebuff
Co-Authored-By: Codebuff noreply@codebuff.com