Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 14 additions & 5 deletions docs/cli-mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,11 +146,11 @@ Laya does not open a network port.
| Tool | What it does | Main inputs |
|---|---|---|
| `laya_status` | Reports the configured or actual device, CUDA availability, loaded checkpoints, preload state, readiness, and package versions. | none |
| `laya_route` | Selects a checkpoint and returns its model, repository, and reason without running a forward pass. | `state`, `questions` |
| `laya_predict` | Runs typed questions and returns answers, routing metadata, latency, and the answering device when readable. | `state`, `questions`, optional `model` (`auto`, `english`, `multilingual`, or `typed-decisions`) |
| `laya_shortlist` | Shortlists a many-option choice question, then answers it and returns the shortlist metadata. | `state`, `questions`, optional `model`, optional `k` (default `20`) |
| `laya_preset` | Runs a built-in workflow using its built-in question set. | `preset`, `state` |
| `laya_predict_batch` | Answers many requests in one call. Requests are routed first and grouped by checkpoint, so matching question schemas share forward passes; answers come back in input order. | `requests`, each `{state, questions, model?, task?, lang?}`, optional `batch_size` |
| `laya_route` | Selects a checkpoint and returns its model, repository, and reason without running a forward pass. | `state`, `questions`, optional `model`, `task`, `lang`, `lang_guess` |
| `laya_predict` | Runs typed questions and returns answers, routing metadata, latency, and the answering device when readable. | `state`, `questions`, optional `model` (`auto`, `english`, `multilingual`, or `typed-decisions`), `task`, `lang`, `lang_guess`, `max_len`, `head_max_len`, `min_confidence` |
| `laya_shortlist` | Shortlists a many-option choice question, then answers it and returns the shortlist metadata. | `state`, `questions`, optional `model`, `k` (default `20`), `task`, `lang`, `lang_guess`, `max_len`, `head_max_len`, `min_confidence` |
| `laya_preset` | Runs a built-in workflow using its built-in question set. | `preset`, `state`, optional `task`, `lang`, `lang_guess`, `max_len`, `head_max_len`, `min_confidence` |
| `laya_predict_batch` | Answers many requests in one call. Requests are routed first and grouped by checkpoint, so matching question schemas share forward passes; answers come back in input order. | `requests`, each `{state, questions, model?, task?, lang?, lang_guess?, max_len?, head_max_len?}`, optional `batch_size` |
| `laya_route_batch` | Decides which checkpoint would answer each request, with no forward pass and no checkpoint load. | `requests`, same shape as `laya_predict_batch` |
| `laya_decide` | Answers a JSON-schema-shaped decision in one forward pass and returns the decided values with per-field confidence, instead of an answer map to parse. Schema properties may be enum choices, booleans, or integers with a minimum and maximum; free strings, arrays, and nested objects are rejected by path. | `state`, `schema`, optional `model` |

Expand All @@ -174,6 +174,15 @@ exactly one string, `laya_preset` places it under the field that preset's questi
same placement the CLI does, so a caller does not have to guess the key. Anything richer than one
string is the caller's own shape and is passed through untouched.

Every single-request tool takes the same per-call routing controls the batch requests do. Alongside
`model`, a request may set `task` (name a checkpoint by the work), `lang` (force a language code),
and `lang_guess` (a soft language hint that sits below `lang` and above the built-in detector, so a
probable-but-uncertain code can nudge which checkpoint is chosen without forcing it the way `lang`
does). `lang_guess` only participates in routing, so like `task` it is refused on a call that pins
`model` -- a pinned checkpoint has nothing left to route. `laya_predict` and `laya_shortlist` also
take `max_len`/`head_max_len` for the answering token budget and `min_confidence` for the abstention
gate.

A prediction call has the same shape as the SDK's typed call:

```json
Expand Down
13 changes: 12 additions & 1 deletion laya/mcp/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,9 @@
"refused on a call that also pins model, since the pin would win and the task would be ignored. "
"lang: a language code ('de', 'en-US') -- routes non-English text to the multilingual "
"checkpoint and selects that checkpoint's per-language calibration. "
"lang_guess: a soft language code -- a probable-but-uncertain language that participates in "
"routing (checked after lang, before built-in detection) rather than forcing the checkpoint the "
"way lang does; like task it only routes, so it is refused on a call that pins model. "
"max_len / head_max_len: positive integers overriding the answering token budget for this call "
"only -- head_max_len is the option-and-instructions budget, so raise it when a choice question "
"has many options and the answers look like the labels blur together. "
Expand Down Expand Up @@ -204,7 +207,7 @@ def laya_status_tool() -> str:
description=(
"Decide which Laya checkpoint would answer, without running a forward pass. "
"Use this to explain routing (english vs multilingual vs typed-decisions) to the user. "
"Passing a laya_predict call's model/task/lang here reproduces the routing block it "
"Passing a laya_predict call's model/task/lang/lang_guess here reproduces the routing block it "
"reported, without paying for the forward pass; leaving model unset (or 'auto') routes as "
"normal. "
+ _GUARDRAILS
Expand All @@ -217,6 +220,7 @@ def laya_route_tool(
model: str | None = None,
task: str | None = None,
lang: str | None = None,
lang_guess: str | None = None,
) -> str:
"""Decide which Laya checkpoint would answer, without running a forward pass."""
router = _router_or_error()
Expand All @@ -227,6 +231,7 @@ def laya_route_tool(
model=model,
task=task,
lang=lang,
lang_guess=lang_guess,
router=router,
)

Expand All @@ -250,6 +255,7 @@ def laya_predict_tool(
model: str = "auto",
task: str | None = None,
lang: str | None = None,
lang_guess: str | None = None,
max_len: int | None = None,
head_max_len: int | None = None,
min_confidence: float | None = None,
Expand All @@ -263,6 +269,7 @@ def laya_predict_tool(
model=model,
task=task,
lang=lang,
lang_guess=lang_guess,
max_len=max_len,
head_max_len=head_max_len,
min_confidence=min_confidence,
Expand Down Expand Up @@ -339,6 +346,7 @@ def laya_shortlist_tool(
k: int = 20,
task: str | None = None,
lang: str | None = None,
lang_guess: str | None = None,
max_len: int | None = None,
head_max_len: int | None = None,
min_confidence: float | None = None,
Expand All @@ -357,6 +365,7 @@ def laya_shortlist_tool(
k=k,
task=task,
lang=lang,
lang_guess=lang_guess,
max_len=max_len,
head_max_len=head_max_len,
min_confidence=min_confidence,
Expand Down Expand Up @@ -396,6 +405,7 @@ def laya_preset_tool(
state: dict,
task: str | None = None,
lang: str | None = None,
lang_guess: str | None = None,
max_len: int | None = None,
head_max_len: int | None = None,
min_confidence: float | None = None,
Expand All @@ -408,6 +418,7 @@ def laya_preset_tool(
state=state,
task=task,
lang=lang,
lang_guess=lang_guess,
max_len=max_len,
head_max_len=head_max_len,
min_confidence=min_confidence,
Expand Down
90 changes: 75 additions & 15 deletions laya/mcp/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -280,6 +280,25 @@ def validate_lang(lang: Any) -> str | None:
return lang


def validate_lang_guess(lang_guess: Any) -> str | None:
"""A soft routing hint for core's ``lang_guess`` override, or ``None`` for "not set".

``lang_guess`` sits between an explicit ``lang`` and the built-in detector: it names a probable
language so routing can prefer the checkpoint that reads it, without forcing the answer the way
``lang`` does. Like ``lang`` only the type is checked here -- a code's meaning is core's business,
and a blank or unknown code falls through to detection rather than failing. Core also accepts a
callable for ``lang_guess``, but an MCP client carries JSON, not a function, so a non-string is
refused rather than silently dropped. The batch surface already advertises this key
(:data:`BATCH_ITEM_OVERRIDES`); the single-request tools forward it through here.
"""
if lang_guess is None:
return None
if not isinstance(lang_guess, str):
raise ToolError("invalid_lang_guess",
"lang_guess must be a language code like 'en' or 'de', got %r" % (lang_guess,))
return lang_guess


def validate_budget(value: Any, name: str) -> int | None:
"""One of the two per-call token budgets (``max_len``, ``head_max_len``), or ``None``.

Expand Down Expand Up @@ -360,6 +379,7 @@ def laya_predict(
*,
task: Any = None,
lang: Any = None,
lang_guess: Any = None,
max_len: Any = None,
head_max_len: Any = None,
min_confidence: Any = None,
Expand All @@ -371,10 +391,12 @@ def laya_predict(
``router`` is used when model == "auto"; ``agent`` for a direct checkpoint.

``task``/``lang`` are the router's own overrides (an explicit ``model`` outranks an explicit
``task``, which outranks an explicit ``lang``); ``max_len``/``head_max_len`` override the token
budget the answering checkpoint was configured with; ``min_confidence`` flags any answer whose
calibrated confidence falls below it with ``low_confidence: true``. None of them is forwarded
unless set, so a router or agent that predates those keywords keeps working.
``task``, which outranks an explicit ``lang``); ``lang_guess`` is the router's soft hint that sits
below ``lang`` and above built-in detection -- it only participates in routing, so like ``task``
it is refused on a pinned model and never reaches a direct agent; ``max_len``/``head_max_len``
override the token budget the answering checkpoint was configured with; ``min_confidence`` flags
any answer whose calibrated confidence falls below it with ``low_confidence: true``. None of them
is forwarded unless set, so a router or agent that predates those keywords keeps working.
"""
state_d = validate_state(state)
questions_d = validate_questions(questions)
Expand All @@ -386,6 +408,11 @@ def laya_predict(
validate_budget(max_len, "max_len"),
validate_budget(head_max_len, "head_max_len"),
)
lang_guess_code = validate_lang_guess(lang_guess)
if lang_guess_code is not None:
# Only when set, so a call that passes no hint stays byte-identical to the one that made
# no ``lang_guess`` keyword exist, and a deployment's ``Router(lang_guess=...)`` still answers.
budget["lang_guess"] = lang_guess_code
min_conf = validate_min_confidence(min_confidence)
if min_conf is not None:
# Only when set, so a call that abstains over nothing stays byte-identical to the one
Expand All @@ -400,19 +427,27 @@ def laya_predict(
"invalid_task",
"task routes between checkpoints; with a pinned model there is nothing to route",
)
# `lang_guess` is routing-only in the same way, and `Agent.predict` does not accept it, so a
# pinned call that set it would either be ignored (router) or crash (direct agent).
if model_name != AUTO and "lang_guess" in budget:
raise ToolError(
"invalid_lang_guess",
"lang_guess routes between checkpoints; with a pinned model there is nothing to route",
)

def _run() -> Any:
if model_name == "auto":
if router is None:
if agent is None:
raise ToolError("models_not_ready", "Router is not loaded (auto mode)")
# An Agent answers but does not route: run it directly and report
# a null model below, instead of echoing 'auto' (#444). `task` only
# chooses between checkpoints, so it has nothing to do here.
# a null model below, instead of echoing 'auto' (#444). `task` and `lang_guess` only
# choose between checkpoints, so they have nothing to do here.
nonlocal auto_without_router
auto_without_router = True
return agent.predict(state_d, questions_d,
**{k: v for k, v in budget.items() if k != "task"})
**{k: v for k, v in budget.items()
if k not in ("task", "lang_guess")})
return router.predict(state_d, questions_d, **budget)
if agent is not None:
return agent.predict(state_d, questions_d, **budget)
Expand Down Expand Up @@ -470,12 +505,14 @@ def laya_route(
model: Any = None,
task: Any = None,
lang: Any = None,
lang_guess: Any = None,
router: Any = None,
) -> dict:
"""Routing decision only: no forward pass.

Takes the three routing overrides :meth:`laya_predict` takes -- ``model``, ``task``, ``lang`` --
so "which checkpoint would this go to?" can be asked under a pin without running anything.
Takes the routing overrides :meth:`laya_predict` takes -- ``model``, ``task``, ``lang``,
``lang_guess`` -- so "which checkpoint would this go to?" can be asked under a pin without
running anything.
Passing a predict call's controls here reproduces the ``routing`` block it returned, which is
what makes a route a cheap explanation of a decision rather than a different decision.
"""
Expand All @@ -485,13 +522,20 @@ def laya_route(
# override rather than a ValueError from normalise_name.
model_name = validate_model(model)
task_name = validate_task(task)
lang_guess_code = validate_lang_guess(lang_guess)
# Same rule as the decision tools: `_route` checks an explicit model first and never reaches
# the task, so a call that sets both is asking a question with two answers.
if model_name != AUTO and task_name is not None:
raise ToolError(
"invalid_task",
"task routes between checkpoints; with a pinned model there is nothing to route",
)
# `lang_guess` routes in the same place, so it is equally meaningless once a checkpoint is pinned.
if model_name != AUTO and lang_guess_code is not None:
raise ToolError(
"invalid_lang_guess",
"lang_guess routes between checkpoints; with a pinned model there is nothing to route",
)
overrides: dict[str, Any] = {}
if model_name != AUTO:
overrides["model"] = model_name
Expand All @@ -500,6 +544,8 @@ def laya_route(
lang_code = validate_lang(lang)
if lang_code is not None:
overrides["lang"] = lang_code
if lang_guess_code is not None:
overrides["lang_guess"] = lang_guess_code
# Everything the caller typed is checked before the server is asked for a checkpoint.
if router is None:
raise ToolError("models_not_ready", "Router is not loaded")
Expand Down Expand Up @@ -583,6 +629,7 @@ def laya_shortlist(
*,
task: Any = None,
lang: Any = None,
lang_guess: Any = None,
max_len: Any = None,
head_max_len: Any = None,
min_confidence: Any = None,
Expand All @@ -605,8 +652,9 @@ def laya_shortlist(
``head_max_len`` matters more here than anywhere else: shortlisting exists
because a large label set shares that budget, and narrowing to ``k`` is only
half of the fix. The budget override reaches the answering forward pass;
``task``/``lang`` reach the route that chose the checkpoint; ``min_confidence``
flags a kept-label answer the checkpoint is unsure of.
``task``/``lang``/``lang_guess`` reach the route that chose the checkpoint
(``lang_guess`` only routes, so like ``task`` it is stripped before the answering pass);
``min_confidence`` flags a kept-label answer the checkpoint is unsure of.
"""
# Lazy: keeps numpy/shortlist out of module import for laya.mcp.tools.
from laya.shortlist import (
Expand All @@ -620,6 +668,9 @@ def laya_shortlist(
questions_d = validate_questions(questions)
model_name = validate_model(model)
routing_overrides = _overrides(validate_task(task), validate_lang(lang), None, None)
lang_guess_code = validate_lang_guess(lang_guess)
if lang_guess_code is not None:
routing_overrides["lang_guess"] = lang_guess_code
budget = _overrides(None, None, validate_budget(max_len, "max_len"),
validate_budget(head_max_len, "head_max_len"))
min_conf = validate_min_confidence(min_confidence)
Expand All @@ -634,9 +685,16 @@ def laya_shortlist(
"invalid_task",
"task routes between checkpoints; with a pinned model there is nothing to route",
)
# `lang_guess` is routing-only in the same way, so a pinned call has nothing for it to choose.
if model_name != AUTO and "lang_guess" in routing_overrides:
raise ToolError(
"invalid_lang_guess",
"lang_guess routes between checkpoints; with a pinned model there is nothing to route",
)
# `lang` survives pinning because it means two things: it can route, and on the answering
# checkpoint it selects the per-language temperature table.
forward_overrides = {name: value for name, value in routing_overrides.items() if name != "task"}
forward_overrides = {name: value for name, value in routing_overrides.items()
if name not in ("task", "lang_guess")}
if k is None:
k = DEFAULT_SHORTLIST_K
if isinstance(k, bool) or not isinstance(k, int) or k < 1:
Expand Down Expand Up @@ -717,6 +775,7 @@ def laya_preset(
*,
task: Any = None,
lang: Any = None,
lang_guess: Any = None,
max_len: Any = None,
head_max_len: Any = None,
min_confidence: Any = None,
Expand All @@ -733,9 +792,9 @@ def laya_preset(
shape and is passed through untouched.

The preset fixes the questions, not the route or the budget, so the per-call controls a
hand-written :func:`laya_predict` takes -- ``task``/``lang``/``max_len``/``head_max_len`` and
``min_confidence`` -- are available here too; most usefully ``lang``, since a preset's
instructions are English text whatever state they read.
hand-written :func:`laya_predict` takes -- ``task``/``lang``/``lang_guess``/``max_len``/
``head_max_len`` and ``min_confidence`` -- are available here too; most usefully ``lang``, since a
preset's instructions are English text whatever state they read.
"""
preset_name = validate_preset(preset)
state_d = validate_state(state)
Expand All @@ -753,6 +812,7 @@ def laya_preset(
model="auto",
task=task,
lang=lang,
lang_guess=lang_guess,
max_len=max_len,
head_max_len=head_max_len,
min_confidence=min_confidence,
Expand Down
Loading
Loading