Repository navigation
Conversation
|
Good page. One rebase note: |
Implements guide 1 from [NandhaKishorM#390](NandhaKishorM#390), which lists "Routing: how the `Router` picks a checkpoint, `lang` overrides, bringing your own language detection, preload and memory, mixed batches, and what `RouteDecision` holds" as one of the topics still to write and asks contributors to claim them. Nobody had claimed it — @emiliano-go said in that thread he was not working on any of the six guides. Every example and every printed value in the page was run against laya 0.3.20 before being written down, so the reasons, the `RouteDecision` keys and the memory behaviour are transcripts rather than reconstructions. ## What the page covers - **What it decides on.** Script first, language second, and the four outcomes that route differently — including the one that is easy to get wrong: short undecided Latin text goes to the **English** checkpoint, not the multilingual one, and says so in `reason`. - **Reading the decision.** `RouteDecision` subclasses `dict` and carries `model`, `reason`, `repo`, `detection` and `workflow`; the four reasons a caller will see most often are tabulated so a misroute is diagnosable from the result alone. - **Overriding the route**, in four increasing levels of caller control: `model=`, `lang=`, `lang_guess=` (router-wide or per call), and an `on_route` hook that may replace the decision. The page notes that `lang` reaches the checkpoint as well as the router, so it changes both which model answers and how its probabilities are scaled. - **Memory and load cost**: `max_loaded`, `unload`, `preload`, and `attach`. - **Batching**: `predict_batch` groups by chosen checkpoint so a mixed batch costs one forward pass per checkpoint, and `route_batch` returns decisions without loading anything. - **Custom checkpoint sets**: `models`, `standalone_repos`, `default`. - **What routing cannot do**, so the limits are stated where a reader looks for capability. ## Two things the measurements corrected **`attach` does not respect `max_loaded`.** I had written that `max_loaded` caps how many checkpoints stay resident, full stop. It does not: `Router.attach` raises the cap to fit every attached agent (`router.py:324`), so `Router(max_loaded=1)` with two attached agents ends up at `max_loaded == 2` and evicts nothing. Verified with fakes, no model load: ``` after 1 attach: max_loaded=1 loaded=['english'] after 2 attach: max_loaded=2 loaded=['english', 'multilingual'] ``` The page now says so explicitly and explains why it is deliberate rather than a bug — `attach` is for weights the caller already owns and is holding, so evicting one would free nothing while silently detaching an agent the caller still references. Documenting the old sentence would have told a reader their cap was being enforced when it was not. **The undecided-Latin default is a policy, not a fallback.** `Ik wil mijn account opzeggen` detects as `language=None, is_english=True, language_undecided=True` and routes to `english`. That is only obvious once you know the English checkpoint degrades on other Latin languages and the multilingual one degrades on English-heavy text, so the page states the tradeoff instead of just the outcome. Deliberate choices tested: - **No new page structure or nav.** `docs/.nav.yml` does not exist in this tree, and NandhaKishorM#390 says a page that is not listed still appears under Guides and can be placed later, so the page is added as `docs/routing.md` with no navigation edit. That also means it cannot conflict with the `.nav.yml` another PR is introducing. - **Links follow the file's own convention.** `hooks/index.md` and `../BENCHMARKS.md` are the two spellings already used in `docs/`, and both resolve from `docs/routing.md` (checked). - **It defers rather than duplicates.** The hooks escape hatch links to `docs/hooks/index.md`, the checkpoint limits link to `BENCHMARKS.md`, and nothing is copied out of the README — NandhaKishorM#390 says the README is not being trimmed yet, so a guide that restated it would only create a second copy to keep in sync. - **`detect_language` and `detect_script` are documented as public**, since they are in `laya.__all__` and let a caller run the same test the router runs without building a router. ``` $ python tests/test_packaging.py all packaging tests passed $ python tests/test_doc_tables.py 54 passed, 0 failed $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! $ python -m compileall -q laya/ tests/ ``` Known limitation: this is the Routing page only. The five other guides NandhaKishorM#390 lists — questions and answers, HTTP API, command line and MCP, fine-tuning, and benchmarks — are not covered here, and the HTTP API one should wait for NandhaKishorM#304 and NandhaKishorM#281 to settle as that issue says.
04c9b64 to
57ad596
Compare
maxfield-allison
left a comment
There was a problem hiding this comment.
Checked the page at 57ad596 against main at 970dc8c. These statements need correcting:
lang_guessruns before built-in detection. Withlang_guess=lambda s: "de", clear English text routes tomultilingual. An explicitmodelortask, an opted-in workflow match, or alangvalue that identifies a language takes precedence.- Falsy values don't all abstain.
None, blank strings and neutral codes such asC.UTF-8,und,zxxandmulabstain.Falseand0are converted to non-English codes and selectmultilingual. “Return a language code, orNoneto abstain” would be accurate guidance. An abstaining per-call hint still leaves the Router-wide hint to try before built-in detection. - An attached agent can be evicted.
attachraisesmax_loadedto fit the agents resident at that moment, but later loads still use the same eviction list. WithRouter(max_loaded=1), attachingenglishand then loadingmultilingualleaves onlymultilingualinrouter.loaded. Any separate reference held by the caller survives that removal. - The decision fields depend on how the route was chosen.
workflowcan be populated withauto_task_detection=False: a matching question-id set supplies the metadata without selecting typed-decisions. Explicitmodelortaskoverrides bypass that check and leave itNone.detectionis alsoNonewhen an override or automatic workflow match decides the route, rather than always containing the detector payload. - An
on_routehook replaces the decision throughctx.decision. Its return value is ignored. The wording “may return a different one” would lead a reader to write a hook that has no effect. - A batch can need more than one forward pass per checkpoint. The later paragraph describes the schema, token-budget and calibration-language splits, but the opening promises one pass per checkpoint.
batch_sizecan split each compatible group further. “Shares forward passes within compatible groups” would cover both cases. langselects language-specific scaling only when a matchinglang_temperaturesentry exists. The defaultRouter.load()path doesn't supply that configuration. Passinglang="nl"therefore doesn't guarantee a change in probability scaling.lang_guessis a routing hint and isn't forwarded as the calibration language.- A single accented loanword can leave the English route.
Router().route("Please send the résumé to the hiring team by Friday.").modelreturnsmultilingualat this revision. That's #337, and #350 proposes the fix, so the final limits bullet describes behaviormaindoesn't have yet. Unidentified text also uses the configureddefault, which needn't beenglish.
It would help to mention the 4,000-character sample limit and the bounded line-and-field check from #207. That check can recover a non-English message alongside an English log, but it doesn't guarantee detection of every foreign field, particularly outside the sample.
|
Thank you — corrected in the description, and I checked the other claims while I was at it rather than only the one you flagged. On Two more that were stale on the same reading. Both were my own errors rather than things that changed under the PR:
Re-run on the current head: |
|
Thank you for such a thorough routing page, and for measuring things rather than guessing. #461 covers the same guide with an accurate precedence order, so we will go with that one and close this to avoid two routing pages. Several points from the earlier review still apply here (lang_guess runs before detection, attached agents can be evicted, the on_route return value is ignored). Your reason table, the RouteDecision fields, and the attach and custom-checkpoint sections are genuinely useful, and a follow-up PR adding them to docs/routing.md once #461 lands would be very welcome. |
docs: add the Routing guide
Implements guide 1 from #390, which lists
"Routing: how the
Routerpicks a checkpoint,langoverrides, bringing your own languagedetection, preload and memory, mixed batches, and what
RouteDecisionholds" as one of thetopics still to write and asks contributors to claim them. Nobody had claimed it —
@emiliano-go said in that thread he was not working on any of the six guides.
Every example and every printed value in the page was run against laya 0.3.20 before being
written down, so the reasons, the
RouteDecisionkeys and the memory behaviour are transcriptsrather than reconstructions.
What the page covers
differently — including the one that is easy to get wrong: short undecided Latin text goes to
the English checkpoint, not the multilingual one, and says so in
reason.RouteDecisionsubclassesdictand carriesmodel,reason,repo,detectionandworkflow; the four reasons a caller will see most often are tabulatedso a misroute is diagnosable from the result alone.
model=,lang=,lang_guess=(router-wide or per call), and anon_routehook that may replace the decision.The page notes that
langreaches the checkpoint as well as the router, so it changes bothwhich model answers and how its probabilities are scaled.
max_loaded,unload,preload, andattach.predict_batchgroups by chosen checkpoint so a mixed batch costs one forwardpass per checkpoint, and
route_batchreturns decisions without loading anything.models,standalone_repos,default.Two things the measurements corrected
attachdoes not respectmax_loaded. I had written thatmax_loadedcaps how manycheckpoints stay resident, full stop. It does not:
Router.attachraises the cap to fit everyattached agent (
router.py:324), soRouter(max_loaded=1)with two attached agents ends up atmax_loaded == 2and evicts nothing. Verified with fakes, no model load:The page now says so explicitly and explains why it is deliberate rather than a bug —
attachis for weights the caller already owns and is holding, so evicting one would free nothing while
silently detaching an agent the caller still references. Documenting the old sentence would have
told a reader their cap was being enforced when it was not.
The undecided-Latin default is a policy, not a fallback.
Ik wil mijn account opzeggendetects as
language=None, is_english=True, language_undecided=Trueand routes toenglish.That is only obvious once you know the English checkpoint degrades on other Latin languages and
the multilingual one degrades on English-heavy text, so the page states the tradeoff instead of
just the outcome.
Deliberate choices tested:
docs/.nav.ymlnow exists (#391), and its own comment is the reason none is needed: "A page or folder that is not listed appears under Guides, so a new guide shows up without an edit here." Theglob: "*"under Guides is what implements that, sodocs/routing.mdlands there and can be moved to a section once Docs site: structure and the guides still to write #390 settles where it belongs.[BENCHMARKS.md](../BENCHMARKS.md), which breakszensical build --strictbecause the file is outsidedocs/— the gate caught it, and the page now refers toBENCHMARKS.mdat the repository root as plain text rather than as a link the builder resolves.hooks/index.mdis the one relative link, and it is the spellingdocs/already uses.docs/hooks/index.md, the checkpoint limits point atBENCHMARKS.md, and nothing is copied out of the README — Docs site: structure and the guides still to write #390 says the README is not being trimmed yet, so a guide that restated it would only create a second copy to keep in sync.detect_languageanddetect_scriptare documented as public, since they are inlaya.__all__and let a caller run the same test the router runs without building a router.Known limitation: this is the Routing page only. The five other guides #390 lists — questions
and answers, HTTP API, command line and MCP, fine-tuning, and benchmarks — are not covered here,
and the HTTP API one should wait for #304 and #281 to settle as that issue says. The questions
guide is in flight as its own PR.