Skip to content

fix(serve): /health reports where inference actually runs, not the requested device - #574

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-serve-health
Sep 27, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
aashish254:feat-serve-health

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

The problem

compose.cuda.yaml sets LAYA_DEVICE=cuda for laya-serve, and /health answered that value back verbatim:

"device": os.environ.get("LAYA_DEVICE") or "auto",

So on a host that cannot actually give the container a GPU, the Agent falls back to CPU silently, keeps answering correctly, and /health says "device": "cuda" — permanently. The healthcheck in compose.http.yaml only asserts that the route answers 200, so the container is healthy and lies about where it computes. docs/docker.md even points at this failure mode ("Laya can still fall back to CPU after a memory or inference error, so inspect its warnings") while giving no way to see it from outside.

The repo already has the fix for this, in a place /health does not use: laya/mcp/device.py exists to separate the configured preference from the device a checkpoint really runs on, and says so in its own docstring. MCP's laya_status reads Agent.device through it. Only the HTTP surface — the one a container healthcheck and a load balancer look at — never did.

Same root cause, second symptom: build_router() passed os.environ.get("LAYA_DEVICE") to Router without stripping it, even though it strips LAYA_MODELS two lines below. A value copied out of a Dockerfile or a .env file carries a trailing newline, and torch refuses it.

What changes

/health now reads Agent.device for each resident checkpoint via agent_device() / router_agent() and returns the same three keys laya_status already returns over MCP — device, device_is_preference, checkpoint_devices — and build_router() uses env_device(). No core change: laya.mcp.device is documented (and tests/test_lazy_import.py-guarded) as importable without the mcp extra and touches torch lazily, so the serve-only install path is unaffected.

One wire-visible difference worth stating: with LAYA_DEVICE unset and nothing loaded, device is now the resolved preference (cpu/cuda) rather than the string "auto", and device_is_preference is true. Nothing in-tree parses the field — the healthcheck ignores the body, and BENCHMARKS.md uses it only as a network-cost baseline.

Measured

CPU, real english checkpoint, the real FastAPI app through TestClient. Full probe in laya-bench/probe_health_device.py, logs in health_device_before.log / health_device_after.log.

case before after
LAYA_DEVICE=cuda, english resident and actually on CPU /health says cuda — /health says the truth? False, while the same router answers play_music correctly says cpu, device_is_preference: false, checkpoint_devices: {"english": "cpu"}
LAYA_DEVICE= cpu\n (an env file's trailing newline) serve build_router() → RuntimeError: Invalid device string: ' cpu; env_device() already returned 'cpu' checkpoint on cpu, no raise
LAYA_DEVICE unset, nothing resident auto cpu + device_is_preference: true
/health vs laya_status on the same router disagree (auto vs cpu / False / {'english': 'cpu'}) all three keys identical

No cost: /health with two checkpoints resident, 400 probes — median 0.609 → 0.602 ms, p95 1.079 → 0.667 ms. BENCHMARKS.md records 0.6 ms for this route, so both sides sit on the published number.

No answer changes: eight states through POST /v1/systemone via build_router() (laya-bench/probe_health_parity.py) give the identical decision digest b42662d726a7… before and after, and the same digest again when LAYA_DEVICE carries the newline — a route that previously could not start at all.

Tests

tests/test_serve.py 30 → 36 passing, all weight-free:

  • the four new payload checks use a stub router whose agents carry a plain-string device (which agent_device accepts), so no torch and no weights;
  • one test asserts /health and laya_status agree on all three keys for the same router, so the two surfaces cannot drift apart again;
  • one subprocess test blocks mcp in sys.meta_path the way tests/test_lazy_import.py does and still serves /health, which is the CI job that installs .[serve] without .[mcp];
  • one table test drives build_router() over " cpu\n", "cuda ", " ", "cpu" and unset with a recording Router stub.

Six mutations of the new code were each caught by named tests: dropping the resident-device read (3 failures), restoring the env read for device (3), removing device_is_preference (3), removing checkpoint_devices (5), restoring the unstripped env read in build_router() (1), and reporting the preference per checkpoint instead of Agent.device (4). ruff check laya/, compileall, tests/test_lazy_import.py (7 passed) and zensical build --strict --clean are clean.

Overlap

Nine open PRs touch laya/serve.py, so here is the honest read on the three nearest:

Also flagged: #408's /ready and #487's cpu_fallbacks both reach for router._agents, which is the same private handle laya/mcp/device.py:router_agent was written to wrap. Worth converging on that one reader regardless of merge order.

…evice

/health echoed LAYA_DEVICE back as `"device"`, so a container started from
compose.cuda.yaml on a host that cannot give it a GPU reported `"device":
"cuda"` forever while every checkpoint silently computed on CPU -- and stayed
healthy, because the healthcheck only checks that the route answers. The same
environment value also reached Router(device=...) unstripped, so a value pasted
out of a Dockerfile or .env with a trailing newline died as torch's
`Invalid device string`.

Both come from not using laya.mcp.device, which exists precisely to separate the
preference from the fact and is documented as importable without the mcp extra.
/health now reads Agent.device through agent_device()/router_agent() and reports
the same device / device_is_preference / checkpoint_devices triple that
laya_status already returns over MCP. An unset LAYA_DEVICE with nothing resident
now reads as the resolved preference rather than "auto"; nothing in-tree parses
the field, and the healthcheck only asserts a 200.
@rahul05ranjan

Copy link
Copy Markdown
Contributor

Thanks for the PR. Reporting the actual device a resident checkpoint computes on (rather than echoing LAYA_DEVICE) is a genuinely useful observability fix — a container that silently lost its GPU should say so.

A couple of points:

  1. next(iter(checkpoint_devices.values()), None) picks an arbitrary checkpoint's device when multiple are resident. If english is on CPU and multilingual is on GPU, /health reports whichever happens to be first in router.loaded order. The checkpoint_devices map already carries the full picture, so the top-level device is a lossy summary — worth a doc note that it's "the first resident checkpoint's device" rather than implying a single global device.

  2. device_is_preference semantics: it's true only when nothing is resident. That's a slightly surprising name — "preference" here means "no checkpoint loaded, so we're reporting the configured preference." The docstring covers it, but a reader skimming the JSON key alone might misread it. Consider device_is_configured or similar, or keep it and lean on the doc.

  3. env_device() vs raw read: good catch on the trailing-newline strip — that's a real footgun with .env/Dockerfile values.

Otherwise the change is well-scoped and the tests look reasonable.

umran666 added a commit to umran666/laya that referenced this pull request Sep 26, 2026
…_agent

Reuse the side-effect-free accessor the MCP status tool already uses instead of reading the private _agents mapping by hand: fewer lines here, normalised/aliased checkpoint keys handled by the helper, and it is the same read NandhaKishorM#574 adopts, so the two branches overlap only in the /health dict tail.

Suggested by the NandhaKishorM#574 discussion on this PR.
@NandhaKishorM
NandhaKishorM merged commit f964afd into NandhaKishorM:main Sep 27, 2026
22 checks passed
NandhaKishorM added a commit that referenced this pull request Sep 27, 2026
…and CI guards

- nix/laya-serve.nix: maxLoaded, maxTokenBudget and revision options for LAYA_MAX_LOADED
  (#603), LAYA_MAX_TOKEN_BUDGET (#566) and LAYA_REVISION (#605); test_packaging counts
  mcp/device.py's _ENV_KEY as the LAYA_DEVICE read serve now goes through (#574).
- docs/docker.md documents LAYA_REVISION.
- test_audit_scope knows the llamaindex/crewai extras and onnxscript.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thanks @aashish254. Merged, combined with #487's cpu_fallbacks in the same /health payload. It ships in 0.3.21.

NandhaKishorM pushed a commit that referenced this pull request Oct 1, 2026
The page showed four keys while the handler returns seven. device_is_preference,
checkpoint_devices and cpu_fallbacks arrived with the device-fact reporting and #574's
fallback counters, and both were written up in docs/docker.md only -- the page a reader
opens before pointing a probe at a server was the one page nobody updated.

The sample also printed "device": "auto", which is what LAYA_DEVICE defaults to rather
than a value resolve_device can return, and it keyed revisions by one checkpoint while
listing two in loaded. Rewritten as one server's answer with a bullet per field, so
`tests/test_serve.py` can hold each key to the handler's own return statement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants