Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/cli-mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ the server is ready.

| Variable | Default | Meaning |
|---|---|---|
| `LAYA_DEVICE` | automatic | The device checkpoints are asked to build on, such as `cpu` or `cuda`. Not the value verbatim: `laya.mcp.device.env_device` strips and lower-cases it, so `CUDA:0` works, and a string PyTorch cannot parse is refused with an error naming this variable. Because there is no startup to refuse, that comes back as a `router construction failed` tool error. It is a preference, not a guarantee: a GPU that cannot be had falls back to CPU silently, and `laya_status` reports the loaded checkpoint's real device. |
| `LAYA_DEVICE` | automatic | The device checkpoints are asked to build on, such as `cpu` or `cuda`. Not the value verbatim: `laya.mcp.device.env_device` strips and lower-cases it, so `CUDA:0` works, and a string PyTorch cannot parse is refused with an error naming this variable. So is a string PyTorch parses into a *different* device: an index it cannot keep is read back as another one (`cuda:258` builds GPU 2), and the same refusal names both the value and what PyTorch would have used. Because there is no startup to refuse, that comes back as a `router construction failed` tool error. It is a preference, not a guarantee: a GPU that cannot be had falls back to CPU silently, and `laya_status` reports the loaded checkpoint's real device. |
| `LAYA_PRELOAD` | `1` | Build the configured checkpoints at startup. Set to `0` for lazy loading. |
| `LAYA_MODELS` | `english,multilingual` | Comma-separated checkpoints to preload. An empty value keeps the MCP default rather than preloading every checkpoint. |
| `LAYA_THREADS` | PyTorch default | Caps Torch intra-op threads for CPU inference; keep it at or below the physical core count. |
Expand Down
31 changes: 26 additions & 5 deletions laya/mcp/device.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,12 @@
device in use. `env_device` therefore parses the value it is about to return with
``torch.device`` and raises a named ``ValueError`` for one torch cannot read, so the typo is
attributed to the variable at the point the process reads it.

Parsing is only half of what "the same string" has to mean. ``cuda:258`` is perfectly
readable, and torch builds ``cuda:2`` from it, because an index torch cannot keep is read back
as a different one -- so the string the client was told and the device asked of torch would
name two GPUs. `env_device` therefore also refuses a value whose parsed device is not the
device the string names.
`resolve_device` is a best-effort label of the *configured preference*
(LAYA_DEVICE or torch auto-detection), not of the device a loaded checkpoint
actually runs on. For the real device read ``Agent.device`` via
Expand Down Expand Up @@ -45,7 +51,9 @@ def env_device() -> str | None:
cannot read raises ``ValueError`` naming ``LAYA_DEVICE``, before the string can reach
``Router(device=...)`` or be reported as the device in use. ``cuda:`` (trailing colon),
``gpu`` and ``cuda: 0`` (space after the colon) are the common typos; ``CUDA:0`` is not
one, because it is normalised first.
one, because it is normalised first. ``cuda:258`` is a second shape of the same failure:
torch reads it, and reads it as ``cuda:2``, so a value whose parsed device is not the
device it names raises the same way.
"""
value = os.environ.get(_ENV_KEY)
if value is None:
Expand All @@ -61,24 +69,36 @@ def env_device() -> str | None:


def _check_torch_device(value: str) -> None:
"""Raise a named ValueError for a device string ``torch.device`` refuses.
"""Raise a named ValueError for a device string ``torch.device`` refuses, or rewrites.

``torch.device`` is used as the parser itself rather than a lookalike of it: devices
this module has never heard of (``meta``, ``privateuseone:1``) keep working, and a
torch that cannot be imported leaves the value alone, because nothing downstream
could have used it anyway.

That torch accepts the string is not the whole contract. torch reads an index it cannot
keep back as a different one -- on torch 2.6.0 the index survives only in its low 8 bits,
so ``cuda:258`` parses happily and asks for GPU 2, and ``cuda:128`` asks for ``cuda:-128``.
The value is therefore refused unless the device torch parses is the device the string
names. Every ordinary spelling satisfies that: measured over all twenty device types torch
names, with and without an index, ``str(torch.device(v)) == v``.
"""
try:
import torch
except Exception: # pragma: no cover - torch is a hard dependency of the package
return
try:
torch.device(value)
parsed = torch.device(value)
except Exception as exc:
raise ValueError(
"%s=%r is not a torch device: %s -- use a device such as 'cpu', 'cuda', "
"'cuda:0' or 'mps'" % (_ENV_KEY, value, exc)
) from exc
if str(parsed) != value:
raise ValueError(
"%s=%r is not the device torch asks for: torch reads it as %r -- use a device such "
"as 'cpu', 'cuda', 'cuda:0' or 'mps'" % (_ENV_KEY, value, str(parsed))
)


def resolve_device(force: str | None = None) -> str:
Expand All @@ -94,10 +114,11 @@ def resolve_device(force: str | None = None) -> str:
computes on: use ``agent_device`` for the real device (a GPU -> CPU
fallback happens silently at agent build time).

A ``LAYA_DEVICE`` value torch cannot parse raises here too (see
A ``LAYA_DEVICE`` value torch cannot parse -- or can parse into some other device, the
way ``cuda:258`` builds ``cuda:2`` -- raises here too (see
``env_device``), and the auto-detected labels are device types torch names,
so neither the environment nor the availability chain can label a device
torch would refuse. ``force`` is the exception, on purpose: it is returned
torch would refuse or rewrite. ``force`` is the exception, on purpose: it is returned
verbatim, exactly like the CLI ``--device`` flag, so a test can ask what a
given preference would be reported as -- including a preference torch
refuses, which then fails where it is used rather than here.
Expand Down
5 changes: 3 additions & 2 deletions laya/mcp/server.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,9 @@
Environment (same meaning as laya.serve where it exists):
LAYA_DEVICE the device the checkpoints are asked to build on. Not the value verbatim:
laya.mcp.device.env_device strips and lower-cases it (`CUDA:0` -> `cuda:0`)
and refuses a string torch cannot parse with a ValueError naming this variable
(`gpu`, `cuda:`, `cuda: 0`), so the failure reads as a bad environment value
and refuses with a ValueError naming this variable a string torch cannot parse
(`gpu`, `cuda:`, `cuda: 0`) or parses into some other device (`cuda:258` asks for
GPU 2), so the failure reads as a bad environment value
rather than a torch internal one -- here as a `router construction failed` tool
error, because a stdio server has no startup to refuse. A preference, not a
guarantee: as in laya.serve an unavailable GPU falls back to CPU silently, and
Expand Down
90 changes: 90 additions & 0 deletions tests/test_mcp_device.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@
`agent_device`, the answer for a *loaded* checkpoint, reports a device type without its index while
the plain-string form it accepts for fakes keeps whatever it was given.

"torch did not raise" is not yet "torch got what the operator asked for": an index it cannot keep is
read back as a different one (2.6.0 keeps the low 8 bits, so ``cuda:258`` builds GPU 2), so
``env_device`` refuses a value whose parsed device is not the device the string names. The same
``force=`` asymmetry applies -- that hook is checked by neither judge.

No model weights, no network, no GPU: torch is imported for its parser only.
Run: python tests/test_mcp_device.py
"""
Expand Down Expand Up @@ -132,6 +137,73 @@ def parses(value):
return True


def names_the_same_device(value):
"""Whether torch builds the device `value` names, not merely a device it will accept.

The judge `parses` cannot be: torch reads an index it cannot keep back as a different one
(2.6.0 keeps only the low 8 bits), so ``cuda:258`` parses, and asks for GPU 2.
"""
try:
parsed = torch.device(value)
except Exception:
return False
return str(parsed) == value


def named(raw):
"""The string `env_device` hands to torch for a raw env value: stripped, type lower-cased."""
kind, sep, index = raw.strip().partition(":")
return kind.lower() + sep + index


REWRITTEN = ("cuda:128", "cuda:256", "cuda:258", "cuda:300", "cuda:511", "mps:70000")
ACCEPTED = ("cpu", "CUDA:0", "cuda:0", "cuda:7", "cuda:127", "mps", "meta", "privateuseone:1")
REFUSED_BY_TORCH = ("cuda:", "gpu", "cuda: 0")


def test_env_device_refuses_exactly_what_torch_rewrites():
# `parses()` above cannot see this failure: an accepted label can still name the wrong
# device, and then the status endpoint reports a GPU the process will never use. The guard
# is therefore "what torch builds is what the string says", and this pins both directions:
# a value this torch rewrites must be refused, and a value it round-trips must not be.
# Written as a biconditional so it stays true on a torch that stops truncating, and fails
# the moment the guard is dropped or over-broad.
for raw in ACCEPTED + REFUSED_BY_TORCH + REWRITTEN:
message = rejection(raw)
try:
value = env_device_with(raw)
except ValueError:
value = None
if value is None:
ok("%r refused only because torch would not build it verbatim" % raw,
not names_the_same_device(named(raw)), message or "refused, and it round-trips")
else:
ok("%r accepted because torch builds exactly it" % raw, names_the_same_device(value),
repr(value) + " -> torch builds " + repr(str(torch.device(value))))


def test_the_rewrite_refusal_is_attributed_to_the_variable():
# Which of the candidates this torch actually rewrites is its business; the message shape
# is ours. Derived at runtime so a torch that truncates nothing leaves no vacuous pass.
rewritten = [v for v in REWRITTEN if parses(v) and not names_the_same_device(v)]
if not rewritten:
print(" NOTE no torch-rewritten index on this version, so the message arms below do not run")
for raw in rewritten:
message = rejection(raw)
ok("%r is refused with a ValueError naming LAYA_DEVICE" % raw,
"LAYA_DEVICE" in message, message or "no error")
ok("%r's refusal quotes the value the operator wrote" % raw, raw in message, message)
ok("%r's refusal quotes torch's own answer (%s)" % (raw, str(torch.device(raw))),
str(torch.device(raw)) in message, message)
ok("%r's refusal is a different sentence from the parse refusal" % raw,
"is not a torch device" not in message and "is not the device torch asks for" in message,
message)
# The two refusals have to stay tellable apart, the same way the two `hooks_timeout` ones do.
ok("an unparseable value keeps the parse refusal's wording",
"is not a torch device" in rejection("gpu") and "is not the device torch asks for" not in
rejection("gpu"), rejection("gpu"))


def test_force_is_the_only_path_that_can_label_a_refused_device():
# The two paths the docstring claims for never hand back a label torch would refuse:
# the environment either parses or raises, and the auto-detection chain answers with a
Expand Down Expand Up @@ -181,6 +253,21 @@ def test_force_is_the_only_path_that_can_label_a_refused_device():
ok("the parse claim is scoped, not universal", "no path" not in claim, claim)


def test_force_still_passes_a_rewritten_index_through_verbatim():
# The guard covers the environment, not the tests hook: `force=` (and the CLI `--device`
# flag it mirrors) is returned exactly as given even when torch would build something else
# from it. Pinned so widening the guard to `force` fails here rather than in a CLI.
try:
given = resolve_device("cuda:258")
except Exception as exc:
ok("force='cuda:258' stays unchecked", False,
"raised %s: %s" % (type(exc).__name__, exc))
else:
ok("force='cuda:258' stays unchecked", given == "cuda:258", repr(given))
ok("and it is a label torch rewrites, which is the point of the asymmetry",
parses(given) and not names_the_same_device(given), str(torch.device(given)))


def test_agent_device_answers_by_input_shape():
# A loaded agent's `device` is a torch.device, whose `type` carries no index; the plain
# string the suite accepts for fakes has no such split, so the two shapes do not answer
Expand Down Expand Up @@ -219,6 +306,9 @@ def main():
test_unparseable_values_are_rejected_by_name,
test_the_env_path_raises_and_force_passes_through,
test_force_is_the_only_path_that_can_label_a_refused_device,
test_env_device_refuses_exactly_what_torch_rewrites,
test_the_rewrite_refusal_is_attributed_to_the_variable,
test_force_still_passes_a_rewritten_index_through_verbatim,
test_agent_device_answers_by_input_shape,
test_reported_equals_handed_to_torch):
fn()
Expand Down
Loading