Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ The English checkpoint collapses on non-Latin scripts (Khmer scores **0.000 accu

### Production Preload & Memory

A cold checkpoint build costs seconds; language detection costs microseconds. At the default `max_loaded=1`, traffic that alternates languages rebuilds a model on *every* request (measured at a 7.4 s median reload on CPU and 10.3 s on T4).
A cold checkpoint build costs seconds; language detection costs microseconds. The lazy default keeps **two** checkpoints resident — `english` and `multilingual`, the only two automatic routing chooses between — so a language flip costs detection only once each has been built. `max_loaded=1` rebuilds the checkpoint it just evicted on *every* switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4), and traffic that only ever sees one language never builds the second, so the default costs a single-language deployment nothing.

For a server or production app, preload:

Expand All @@ -156,14 +156,16 @@ router.preload(["english", "multilingual"])
# If your app already built an agent, attach it to avoid duplicate VRAM:
router.attach("english", existing_agent)

# Manage resident memory (default keeps 1 hot, LRU eviction)
router = Router(max_loaded=2) # keep two hot
# Manage resident memory (default keeps two hot: english + multilingual, LRU eviction)
router = Router(max_loaded=3) # keep all three hot, e.g. with auto_task_detection
router = Router(max_loaded=1) # memory-constrained host, reloads on every switch
router.unload() # free memory
```

| Deployment Mode | Per-Request Latency | Model Reloads |
|---|---|---|
| `Router()` (lazy, `max_loaded=1`) | 7 to 10 s on every language switch | 1 per switch |
| `Router()` (lazy, `max_loaded=2`) | detection only (<1 ms) on a switch, after each language's first load | 1 the first time a language appears |
| `Router(max_loaded=1)` | 7 to 10 s on every language switch | 1 per switch |
| `Router(preload=True)` | **32.8 ms (GPU) / 193–464 ms (CPU)** | **none** |

---
Expand Down
13 changes: 10 additions & 3 deletions laya/router.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,9 +133,16 @@ class Router:
Models are downloaded and built on first use. `max_loaded` caps how many stay resident
(least-recently-used is evicted), because all three together are ~1.16B parameters.

The default is 2, because automatic routing only ever chooses between `english` and
`multilingual`: a cap of one rebuilds the checkpoint it just evicted on every script switch,
which is seconds per request on exactly the traffic the Router exists for. Traffic that only
ever sees one language never builds the second checkpoint, so the default costs it nothing.
Lower it to 1 for a memory-constrained host, and raise it to 3 (or preload) when
`auto_task_detection`, an explicit `model=` or an explicit `task=` can reach
`typed-decisions` as well.

For a server or a demo, preload instead: a cold load costs seconds, while detection costs
microseconds, so anything that alternates languages at `max_loaded=1` reloads on every
request.
microseconds, so even the default still pays a load the first time a language appears.

r = Router(preload=True) # all three resident, routing is free
r = Router(preload=True, device="cuda")
Expand All @@ -147,7 +154,7 @@ def __init__(
models: Optional[Dict[str, str]] = None,
device: Optional[str] = None,
token: Optional[str] = None,
max_loaded: int = 1,
max_loaded: int = 2,
default: str = "english",
auto_task_detection: bool = False,
standalone_repos: bool = False,
Expand Down
41 changes: 41 additions & 0 deletions tests/test_router.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,47 @@ def _load_stub(rr, name):
check("lru/unload all", rr.loaded, [])


# --------------------------------------------------------------------- default cap (#172)
# #172 measured a workload that alternates languages at 20-23 s per request on CPU (reloading a
# checkpoint every request) against 49-136 ms with both resident. Automatic routing only ever
# chooses between `english` and `multilingual`, so the default holds both, and a deployment that
# never alternates never builds the second.
def counting_router(cap=None):
"""Router whose loader records which checkpoints it had to build."""
rr = Router() if cap is None else Router(max_loaded=cap)
built = []

def load(name, _rr=rr, _built=built):
key = normalise_name(name)
if key in _rr._agents:
_rr._touch(key)
return _rr._agents[key]
_built.append(key)
_rr._agents[key] = _Stub(key)
_rr._order.append(key)
_rr._evict()
return _rr._agents[key]

rr.load = load
return rr, built


check("lru/default is two", Router().max_loaded, 2)
_en = {"body": "I was charged twice for invoice 4411, please refund."}
_ml = {"body": "Der Kunde wurde zweimal belastet und moechte eine Rueckerstattung"}
for cap, want_built in ((1, 20), (2, 2)):
cr, built = counting_router(cap)
for _ in range(10): # the reported alternating workload
cr.predict(_en, Q_GENERIC)
cr.predict(_ml, Q_GENERIC)
check("lru/alternating traffic, cap=%d builds" % cap, len(built), want_built)
# and the default costs a single-language deployment nothing at all
cr, built = counting_router()
for _ in range(5):
cr.predict(_en, Q_GENERIC)
check("lru/single-language traffic builds one checkpoint", built, ["english"])


# --------------------------------------------------------------------- bundle vs standalone
check("bundle/english is repo root", DEFAULT_MODELS["english"], (BUNDLE_REPO, None))
check("bundle/multilingual subfolder", DEFAULT_MODELS["multilingual"], (BUNDLE_REPO, "multilingual"))
Expand Down