Skip to content

docs: add the Routing guide (#390) - #416

Closed
PerryLink wants to merge 1 commit into
NandhaKishorM:mainfrom
PerryLink:docs-routing-guide
Closed

PerryLink wants to merge 1 commit into
NandhaKishorM:mainfrom
PerryLink:docs-routing-guide

Conversation

@PerryLink

@PerryLink PerryLink commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

docs: add the Routing guide

Implements guide 1 from #390, which lists
"Routing: how the Router picks a checkpoint, lang overrides, bringing your own language
detection, preload and memory, mixed batches, and what RouteDecision holds" as one of the
topics still to write and asks contributors to claim them. Nobody had claimed it —
@emiliano-go said in that thread he was not working on any of the six guides.

Every example and every printed value in the page was run against laya 0.3.20 before being
written down, so the reasons, the RouteDecision keys and the memory behaviour are transcripts
rather than reconstructions.

What the page covers

  • What it decides on. Script first, language second, and the four outcomes that route
    differently — including the one that is easy to get wrong: short undecided Latin text goes to
    the English checkpoint, not the multilingual one, and says so in reason.
  • Reading the decision. RouteDecision subclasses dict and carries model, reason,
    repo, detection and workflow; the four reasons a caller will see most often are tabulated
    so a misroute is diagnosable from the result alone.
  • Overriding the route, in four increasing levels of caller control: model=, lang=,
    lang_guess= (router-wide or per call), and an on_route hook that may replace the decision.
    The page notes that lang reaches the checkpoint as well as the router, so it changes both
    which model answers and how its probabilities are scaled.
  • Memory and load cost: max_loaded, unload, preload, and attach.
  • Batching: predict_batch groups by chosen checkpoint so a mixed batch costs one forward
    pass per checkpoint, and route_batch returns decisions without loading anything.
  • Custom checkpoint sets: models, standalone_repos, default.
  • What routing cannot do, so the limits are stated where a reader looks for capability.

Two things the measurements corrected

attach does not respect max_loaded. I had written that max_loaded caps how many
checkpoints stay resident, full stop. It does not: Router.attach raises the cap to fit every
attached agent (router.py:324), so Router(max_loaded=1) with two attached agents ends up at
max_loaded == 2 and evicts nothing. Verified with fakes, no model load:

after 1 attach: max_loaded=1 loaded=['english']
after 2 attach: max_loaded=2 loaded=['english', 'multilingual']

The page now says so explicitly and explains why it is deliberate rather than a bug — attach
is for weights the caller already owns and is holding, so evicting one would free nothing while
silently detaching an agent the caller still references. Documenting the old sentence would have
told a reader their cap was being enforced when it was not.

The undecided-Latin default is a policy, not a fallback. Ik wil mijn account opzeggen
detects as language=None, is_english=True, language_undecided=True and routes to english.
That is only obvious once you know the English checkpoint degrades on other Latin languages and
the multilingual one degrades on English-heavy text, so the page states the tradeoff instead of
just the outcome.

Deliberate choices tested:

  • No navigation edit. docs/.nav.yml now exists (#391), and its own comment is the reason none is needed: "A page or folder that is not listed appears under Guides, so a new guide shows up without an edit here." The glob: "*" under Guides is what implements that, so docs/routing.md lands there and can be moved to a section once Docs site: structure and the guides still to write #390 settles where it belongs.
  • No link out of the docs tree. An earlier version ended with [BENCHMARKS.md](../BENCHMARKS.md), which breaks zensical build --strict because the file is outside docs/ — the gate caught it, and the page now refers to BENCHMARKS.md at the repository root as plain text rather than as a link the builder resolves. hooks/index.md is the one relative link, and it is the spelling docs/ already uses.
  • It defers rather than duplicates. The hooks escape hatch links to docs/hooks/index.md, the checkpoint limits point at BENCHMARKS.md, and nothing is copied out of the README — Docs site: structure and the guides still to write #390 says the README is not being trimmed yet, so a guide that restated it would only create a second copy to keep in sync.
  • detect_language and detect_script are documented as public, since they are in laya.__all__ and let a caller run the same test the router runs without building a router.
$ zensical build --strict --clean        exit 0, no griffe lines
$ python tests/test_packaging.py         all packaging tests passed
$ python tests/test_doc_tables.py        54 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/

Known limitation: this is the Routing page only. The five other guides #390 lists — questions
and answers, HTTP API, command line and MCP, fine-tuning, and benchmarks — are not covered here,
and the HTTP API one should wait for #304 and #281 to settle as that issue says. The questions
guide is in flight as its own PR.

@emiliano-go

Copy link
Copy Markdown
Contributor

Good page. One rebase note: docs/.nav.yml exists now (#391), so that line is stale; the no-nav-edit call still holds via the glob: "*" under Guides. Claims spot-checked, all good.

Implements guide 1 from [NandhaKishorM#390](NandhaKishorM#390), which lists
"Routing: how the `Router` picks a checkpoint, `lang` overrides, bringing your own language
detection, preload and memory, mixed batches, and what `RouteDecision` holds" as one of the
topics still to write and asks contributors to claim them. Nobody had claimed it —
@emiliano-go said in that thread he was not working on any of the six guides.

Every example and every printed value in the page was run against laya 0.3.20 before being
written down, so the reasons, the `RouteDecision` keys and the memory behaviour are transcripts
rather than reconstructions.

## What the page covers

- **What it decides on.** Script first, language second, and the four outcomes that route
  differently — including the one that is easy to get wrong: short undecided Latin text goes to
  the **English** checkpoint, not the multilingual one, and says so in `reason`.
- **Reading the decision.** `RouteDecision` subclasses `dict` and carries `model`, `reason`,
  `repo`, `detection` and `workflow`; the four reasons a caller will see most often are tabulated
  so a misroute is diagnosable from the result alone.
- **Overriding the route**, in four increasing levels of caller control: `model=`, `lang=`,
  `lang_guess=` (router-wide or per call), and an `on_route` hook that may replace the decision.
  The page notes that `lang` reaches the checkpoint as well as the router, so it changes both
  which model answers and how its probabilities are scaled.
- **Memory and load cost**: `max_loaded`, `unload`, `preload`, and `attach`.
- **Batching**: `predict_batch` groups by chosen checkpoint so a mixed batch costs one forward
  pass per checkpoint, and `route_batch` returns decisions without loading anything.
- **Custom checkpoint sets**: `models`, `standalone_repos`, `default`.
- **What routing cannot do**, so the limits are stated where a reader looks for capability.

## Two things the measurements corrected

**`attach` does not respect `max_loaded`.** I had written that `max_loaded` caps how many
checkpoints stay resident, full stop. It does not: `Router.attach` raises the cap to fit every
attached agent (`router.py:324`), so `Router(max_loaded=1)` with two attached agents ends up at
`max_loaded == 2` and evicts nothing. Verified with fakes, no model load:

```
after 1 attach: max_loaded=1 loaded=['english']
after 2 attach: max_loaded=2 loaded=['english', 'multilingual']
```

The page now says so explicitly and explains why it is deliberate rather than a bug — `attach`
is for weights the caller already owns and is holding, so evicting one would free nothing while
silently detaching an agent the caller still references. Documenting the old sentence would have
told a reader their cap was being enforced when it was not.

**The undecided-Latin default is a policy, not a fallback.** `Ik wil mijn account opzeggen`
detects as `language=None, is_english=True, language_undecided=True` and routes to `english`.
That is only obvious once you know the English checkpoint degrades on other Latin languages and
the multilingual one degrades on English-heavy text, so the page states the tradeoff instead of
just the outcome.

Deliberate choices tested:

- **No new page structure or nav.** `docs/.nav.yml` does not exist in this tree, and NandhaKishorM#390 says a page that is not listed still appears under Guides and can be placed later, so the page is added as `docs/routing.md` with no navigation edit. That also means it cannot conflict with the `.nav.yml` another PR is introducing.
- **Links follow the file's own convention.** `hooks/index.md` and `../BENCHMARKS.md` are the two spellings already used in `docs/`, and both resolve from `docs/routing.md` (checked).
- **It defers rather than duplicates.** The hooks escape hatch links to `docs/hooks/index.md`, the checkpoint limits link to `BENCHMARKS.md`, and nothing is copied out of the README — NandhaKishorM#390 says the README is not being trimmed yet, so a guide that restated it would only create a second copy to keep in sync.
- **`detect_language` and `detect_script` are documented as public**, since they are in `laya.__all__` and let a caller run the same test the router runs without building a router.

```
$ python tests/test_packaging.py       all packaging tests passed
$ python tests/test_doc_tables.py      54 passed, 0 failed
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ python -m compileall -q laya/ tests/
```

Known limitation: this is the Routing page only. The five other guides NandhaKishorM#390 lists —
questions and answers, HTTP API, command line and MCP, fine-tuning, and benchmarks — are not
covered here, and the HTTP API one should wait for NandhaKishorM#304 and NandhaKishorM#281 to settle as that issue says.

@maxfield-allison maxfield-allison left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked the page at 57ad596 against main at 970dc8c. These statements need correcting:

  • lang_guess runs before built-in detection. With lang_guess=lambda s: "de", clear English text routes to multilingual. An explicit model or task, an opted-in workflow match, or a lang value that identifies a language takes precedence.
  • Falsy values don't all abstain. None, blank strings and neutral codes such as C.UTF-8, und, zxx and mul abstain. False and 0 are converted to non-English codes and select multilingual. “Return a language code, or None to abstain” would be accurate guidance. An abstaining per-call hint still leaves the Router-wide hint to try before built-in detection.
  • An attached agent can be evicted. attach raises max_loaded to fit the agents resident at that moment, but later loads still use the same eviction list. With Router(max_loaded=1), attaching english and then loading multilingual leaves only multilingual in router.loaded. Any separate reference held by the caller survives that removal.
  • The decision fields depend on how the route was chosen. workflow can be populated with auto_task_detection=False: a matching question-id set supplies the metadata without selecting typed-decisions. Explicit model or task overrides bypass that check and leave it None. detection is also None when an override or automatic workflow match decides the route, rather than always containing the detector payload.
  • An on_route hook replaces the decision through ctx.decision. Its return value is ignored. The wording “may return a different one” would lead a reader to write a hook that has no effect.
  • A batch can need more than one forward pass per checkpoint. The later paragraph describes the schema, token-budget and calibration-language splits, but the opening promises one pass per checkpoint. batch_size can split each compatible group further. “Shares forward passes within compatible groups” would cover both cases.
  • lang selects language-specific scaling only when a matching lang_temperatures entry exists. The default Router.load() path doesn't supply that configuration. Passing lang="nl" therefore doesn't guarantee a change in probability scaling. lang_guess is a routing hint and isn't forwarded as the calibration language.
  • A single accented loanword can leave the English route. Router().route("Please send the résumé to the hiring team by Friday.").model returns multilingual at this revision. That's #337, and #350 proposes the fix, so the final limits bullet describes behavior main doesn't have yet. Unidentified text also uses the configured default, which needn't be english.

It would help to mention the 4,000-character sample limit and the bounded line-and-field check from #207. That check can recover a non-English message alongside an English log, but it doesn't guarantee detection of every foreign field, particularly outside the sample.

@PerryLink

Copy link
Copy Markdown
Contributor Author

Thank you — corrected in the description, and I checked the other claims while I was at it rather than only the one you flagged.

On docs/.nav.yml: you are right that it exists now (#391). The no-nav-edit call still holds, and reading the file made the reason better than the one I had written: its own comment says "A page or folder that is not listed appears under Guides, so a new guide shows up without an edit here", implemented by the glob: "*" under Guides. So the description now cites that instead of saying the file is absent, and credits #391.

Two more that were stale on the same reading. Both were my own errors rather than things that changed under the PR:

  • I quoted tests/test_doc_tables.py at 59 passed. That count is on the 51-language-refresh branch, which adds the cell checks; on this branch and on main it is 54. Corrected — it is the kind of number that is easy to carry across from another branch, which is exactly why it is worth quoting the command rather than the figure.
  • The description listed ../BENCHMARKS.md as one of the page's links. It is not one any more: it breaks zensical build --strict, because the file is outside docs/ and the builder resolves links against the site root. The gate caught it when I first ran it, and the page now refers to BENCHMARKS.md at the repository root as plain text. The description now records that, since it is the one thing about this page another contributor is likely to hit. hooks/index.md is the only relative link left.

Re-run on the current head:

$ zensical build --strict --clean        exit 0, no griffe lines
$ python tests/test_doc_tables.py        54 passed, 0 failed
$ python tests/test_packaging.py         all packaging tests passed

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for such a thorough routing page, and for measuring things rather than guessing. #461 covers the same guide with an accurate precedence order, so we will go with that one and close this to avoid two routing pages. Several points from the earlier review still apply here (lang_guess runs before detection, attached agents can be evicted, the on_route return value is ignored). Your reason table, the RouteDecision fields, and the attach and custom-checkpoint sections are genuinely useful, and a follow-up PR adding them to docs/routing.md once #461 lands would be very welcome.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants