Skip to content

fix(router): keep two checkpoints by default, not one - #180

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
devloper961-maker:fix/router-default-cap-172
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
devloper961-maker:fix/router-default-cap-172

Conversation

@devloper961-maker

@devloper961-maker devloper961-maker commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Second half of #172 (the routing half is #178).

Router defaults to max_loaded=1, so a workload whose requests alternate languages evicts and rebuilds a checkpoint on every request. #172 measured that at 20-23 s per request on CPU against 49-136 ms with both resident — a 150-400× difference on exactly the traffic a router exists for.

Why 2

Automatic routing only ever chooses between two checkpoints: english and multilingual. typed-decisions is reached only through auto_task_detection=True, an explicit model= or an explicit task=, as the module docstring already says. So at the default, a mixed-script request stream is guaranteed to thrash, and nothing about the payload can avoid it.

Two is also the default that costs nothing to the deployments the old default was protecting: checkpoints load lazily, so a service that only ever sees one language never builds the second. The memory argument for 1 only applies to traffic that actually uses a second checkpoint — which is the traffic 1 makes unusable.

stub loader over the regression workload        checkpoints built
20 alternating requests, max_loaded=1           20   (every request)
20 alternating requests, max_loaded=2            2
5 English-only requests, max_loaded=2            1

max_loaded=1 stays for a memory-constrained host and 3/preload for traffic that also reaches typed-decisions. Both the class docstring and the README's deployment table described the old default, so all three are named there now, including what the default does and does not cost:

Deployment Latency Reloads
Router() (lazy, max_loaded=2) detection only (<1 ms) after each language's first load 1 the first time a language appears
Router(max_loaded=1) 7-10 s on every switch 1 per switch
Router(preload=True) 32.8 ms (GPU) / 193-464 ms (CPU) none

Preloading is still the recommendation for a server, and this does not change that — the first load of each language is still seconds.

Verification

Offline, no weights. tests/test_router.py goes from 150 passed / 0 failed to 154 passed / 0 failed: the new checks drive a stub loader over the reported alternating workload and assert 20 builds at max_loaded=1 against 2 at the default, then assert that English-only traffic builds one checkpoint. The other model-free suites are unchanged and pass, ruff with CI's rule set is clean, python -m compileall -q laya/ tests/ passes.

Touches laya/router.py, README.md and tests/test_router.py only; it does not touch laya/lang.py, so it does not interact with #178/#179. I merged all three branches locally to check: they apply cleanly, the combined routing suite is 221 passed / 0 failed, and Router().max_loaded is 2 with all three applied.

Not addressed: #141 (make a lazy language switch cheaper — no weight re-read) and #138 (group a heterogeneous batch by checkpoint) reduce the same symptom from the other side and are complementary; with either of those, max_loaded=1 becomes less painful but still rebuilds.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff noreply@codebuff.com

`Router` defaults to `max_loaded=1`, so a workload whose requests alternate
languages evicts and rebuilds a checkpoint on every request. NandhaKishorM#172 measured that
at 20-23 s per request on CPU against 49-136 ms with both resident, and reported
it upstream.

Automatic routing only ever chooses between `english` and `multilingual`:
`typed-decisions` needs `auto_task_detection=True`, an explicit `model=` or an
explicit `task=`. A cap of one therefore guarantees a full rebuild on exactly
the mixed-script traffic the Router exists for, while a cap of two costs nothing
to a deployment that only ever sees one language, because checkpoints load
lazily and the second is never built.

`max_loaded=1` stays available for a memory-constrained host, `3`/`preload` for
traffic that also reaches `typed-decisions`; all three are documented in the
class docstring and the README's deployment table, which both described the old
default.

Verification (offline, no weights):
* tests/test_router.py 154 passed, 0 failed (150 before). The new checks drive a
  stub loader over the reported alternating workload: cap=1 builds 20
  checkpoints for 20 requests, cap=2 builds 2, and five English-only requests
  build one checkpoint, which is the memory argument for the default.
* Other model-free suites unchanged and passing; ruff with CI's rule set and
  python -m compileall clean.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @devloper961-maker. This is the second half of #172, and the reasoning holds up: automatic routing only ever picks between two checkpoints, and a single-language deployment never builds the second, so the new default only costs memory for traffic that needs it.

All model-free suites pass locally, alone and together with the other routing PRs landing today. Merging.

@NandhaKishorM
NandhaKishorM merged commit 270e1bc into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
)

`Router` defaults to `max_loaded=1`, so a workload whose requests alternate
languages evicts and rebuilds a checkpoint on every request. NandhaKishorM#172 measured that
at 20-23 s per request on CPU against 49-136 ms with both resident, and reported
it upstream.

Automatic routing only ever chooses between `english` and `multilingual`:
`typed-decisions` needs `auto_task_detection=True`, an explicit `model=` or an
explicit `task=`. A cap of one therefore guarantees a full rebuild on exactly
the mixed-script traffic the Router exists for, while a cap of two costs nothing
to a deployment that only ever sees one language, because checkpoints load
lazily and the second is never built.

`max_loaded=1` stays available for a memory-constrained host, `3`/`preload` for
traffic that also reaches `typed-decisions`; all three are documented in the
class docstring and the README's deployment table, which both described the old
default.

Verification (offline, no weights):
* tests/test_router.py 154 passed, 0 failed (150 before). The new checks drive a
  stub loader over the reported alternating workload: cap=1 builds 20
  checkpoints for 20 requests, cap=2 builds 2, and five English-only requests
  build one checkpoint, which is the memory argument for the default.
* Other model-free suites unchanged and passing; ruff with CI's rule set and
  python -m compileall clean.

🤖 Generated with Codebuff

Co-authored-by: Nikhi <nikhi@example.com>
Co-authored-by: Codebuff <noreply@codebuff.com>
NandhaKishorM pushed a commit that referenced this pull request Sep 27, 2026
examples/server.py built its Router with max_loaded=1, a copy of a default
laya/router.py retired in #180. Unset LAYA_MAX_LOADED now means "not asked
for": the key is left out of the constructor so the library owns the default,
an explicit value (env or --max-loaded) is still honoured, /health reports the
cap the running Router holds rather than the requested one, and the --reload
env push pops the variable instead of writing str(None) for the reimported
process to read with int().
NandhaKishorM pushed a commit that referenced this pull request Oct 2, 2026
… defaults to

The banner opened with "At the default `max_loaded=1`". laya/router.py takes
max_loaded: int = 2 (the old default was retired in #180) and README.md documents
"up to two checkpoints resident by default", so the page that teaches residency
taught the retired number -- and taught readers that they had to raise the cap to
avoid the churn it was demonstrating.

The default is now read from a Router instance instead of typed in here: building one
loads no checkpoint, so it costs nothing and cannot drift again. Section 1 labels the
cap it pins as pinned, next to the library's default.

The closing rule of thumb also claimed the reload is "thousands of times more expensive
than the routing itself" with nothing measured behind it. Both sides are timed now --
the cold build in seconds and a route() through _common.timed(), which loads nothing --
and the page prints their ratio: 0.8 s against 0.031 ms, 27199 times on this machine.
NandhaKishorM pushed a commit that referenced this pull request Oct 11, 2026
…nglish

examples/server.py constructed Router(default="english") whenever
LAYA_DEFAULT_MODEL was unset. 3cf26cb moved Router's own default to
multilingual in 0.4.0, and laya.serve follows it by sending no default
keyword at all -- which laya/mcp/server.py reuses. One environment, three
answers: a bare laya-serve and a bare laya mcp route a state with no
language evidence to the multilingual checkpoint, a bare
`python examples/server.py` still sent it to the English arm.

Unset now means "not asked for", the same rule this file already follows
for LAYA_MAX_LOADED (#180 retired its copy of that number for the same
reason). The /models payload reports the built Router's answer instead of
the requested one, and the --reload push leaves the variable unpushed
rather than writing a checkpoint literally named "None" into the child.

tests/test_example_server_limits.py gains 32 checks: the unset and blank
spellings at both surfaces, the running Router's default, the route()
decision for the three ambiguous states 3cf26cb names, /models in JSON
and HTML, four operator spellings that must still be honored, the
laya.serve parity claim, and the --reload round trip. 122 -> 154 checks.
NandhaKishorM pushed a commit that referenced this pull request Oct 21, 2026
Python's Router defaults to max_loaded=2 (#180), so a router that
alternates two languages keeps both resident. LayaRouterOptions'
max_loaded defaulted to 1, which made every alternation evict and
rebuild a checkpoint.
NandhaKishorM pushed a commit that referenced this pull request Oct 21, 2026
Python's Router defaults to max_loaded=2 (#180), so a router that
alternates two languages keeps both resident. LayaRouterOptions.MaxLoaded
defaulted to 1 while its doc comment claimed to match Python, which made
every alternation evict and reload a checkpoint.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants