Repository navigation
fix(serve): /health reports where inference actually runs, not the requested device - #574
Conversation
…evice /health echoed LAYA_DEVICE back as `"device"`, so a container started from compose.cuda.yaml on a host that cannot give it a GPU reported `"device": "cuda"` forever while every checkpoint silently computed on CPU -- and stayed healthy, because the healthcheck only checks that the route answers. The same environment value also reached Router(device=...) unstripped, so a value pasted out of a Dockerfile or .env with a trailing newline died as torch's `Invalid device string`. Both come from not using laya.mcp.device, which exists precisely to separate the preference from the fact and is documented as importable without the mcp extra. /health now reads Agent.device through agent_device()/router_agent() and reports the same device / device_is_preference / checkpoint_devices triple that laya_status already returns over MCP. An unset LAYA_DEVICE with nothing resident now reads as the resolved preference rather than "auto"; nothing in-tree parses the field, and the healthcheck only asserts a 200.
|
Thanks for the PR. Reporting the actual device a resident checkpoint computes on (rather than echoing A couple of points:
Otherwise the change is well-scoped and the tests look reasonable. |
…_agent Reuse the side-effect-free accessor the MCP status tool already uses instead of reading the private _agents mapping by hand: fewer lines here, normalised/aliased checkpoint keys handled by the helper, and it is the same read NandhaKishorM#574 adopts, so the two branches overlap only in the /health dict tail. Suggested by the NandhaKishorM#574 discussion on this PR.
…and CI guards - nix/laya-serve.nix: maxLoaded, maxTokenBudget and revision options for LAYA_MAX_LOADED (#603), LAYA_MAX_TOKEN_BUDGET (#566) and LAYA_REVISION (#605); test_packaging counts mcp/device.py's _ENV_KEY as the LAYA_DEVICE read serve now goes through (#574). - docs/docker.md documents LAYA_REVISION. - test_audit_scope knows the llamaindex/crewai extras and onnxscript. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Thanks @aashish254. Merged, combined with #487's |
The page showed four keys while the handler returns seven. device_is_preference, checkpoint_devices and cpu_fallbacks arrived with the device-fact reporting and #574's fallback counters, and both were written up in docs/docker.md only -- the page a reader opens before pointing a probe at a server was the one page nobody updated. The sample also printed "device": "auto", which is what LAYA_DEVICE defaults to rather than a value resolve_device can return, and it keyed revisions by one checkpoint while listing two in loaded. Rewritten as one server's answer with a bullet per field, so `tests/test_serve.py` can hold each key to the handler's own return statement.
The problem
compose.cuda.yamlsetsLAYA_DEVICE=cudaforlaya-serve, and/healthanswered that value back verbatim:So on a host that cannot actually give the container a GPU, the Agent falls back to CPU silently, keeps answering correctly, and
/healthsays"device": "cuda"— permanently. The healthcheck incompose.http.yamlonly asserts that the route answers 200, so the container is healthy and lies about where it computes.docs/docker.mdeven points at this failure mode ("Laya can still fall back to CPU after a memory or inference error, so inspect its warnings") while giving no way to see it from outside.The repo already has the fix for this, in a place
/healthdoes not use:laya/mcp/device.pyexists to separate the configured preference from the device a checkpoint really runs on, and says so in its own docstring. MCP'slaya_statusreadsAgent.devicethrough it. Only the HTTP surface — the one a container healthcheck and a load balancer look at — never did.Same root cause, second symptom:
build_router()passedos.environ.get("LAYA_DEVICE")toRouterwithout stripping it, even though it stripsLAYA_MODELStwo lines below. A value copied out of a Dockerfile or a.envfile carries a trailing newline, and torch refuses it.What changes
/healthnow readsAgent.devicefor each resident checkpoint viaagent_device()/router_agent()and returns the same three keyslaya_statusalready returns over MCP —device,device_is_preference,checkpoint_devices— andbuild_router()usesenv_device(). No core change:laya.mcp.deviceis documented (andtests/test_lazy_import.py-guarded) as importable without themcpextra and touches torch lazily, so theserve-only install path is unaffected.One wire-visible difference worth stating: with
LAYA_DEVICEunset and nothing loaded,deviceis now the resolved preference (cpu/cuda) rather than the string"auto", anddevice_is_preferenceistrue. Nothing in-tree parses the field — the healthcheck ignores the body, andBENCHMARKS.mduses it only as a network-cost baseline.Measured
CPU, real
englishcheckpoint, the real FastAPI app throughTestClient. Full probe inlaya-bench/probe_health_device.py, logs inhealth_device_before.log/health_device_after.log.LAYA_DEVICE=cuda,englishresident and actually on CPU/healthsayscuda—/health says the truth? False, while the same router answersplay_musiccorrectlycpu,device_is_preference: false,checkpoint_devices: {"english": "cpu"}LAYA_DEVICE= cpu\n(an env file's trailing newline)serve build_router()→RuntimeError: Invalid device string: ' cpu;env_device()already returned'cpu'checkpoint on cpu, no raiseLAYA_DEVICEunset, nothing residentautocpu+device_is_preference: true/healthvslaya_statuson the same routerautovscpu/False/{'english': 'cpu'})No cost:
/healthwith two checkpoints resident, 400 probes — median 0.609 → 0.602 ms, p95 1.079 → 0.667 ms.BENCHMARKS.mdrecords 0.6 ms for this route, so both sides sit on the published number.No answer changes: eight states through
POST /v1/systemoneviabuild_router()(laya-bench/probe_health_parity.py) give the identical decision digestb42662d726a7…before and after, and the same digest again whenLAYA_DEVICEcarries the newline — a route that previously could not start at all.Tests
tests/test_serve.py30 → 36 passing, all weight-free:device(whichagent_deviceaccepts), so no torch and no weights;/healthandlaya_statusagree on all three keys for the same router, so the two surfaces cannot drift apart again;mcpinsys.meta_paththe waytests/test_lazy_import.pydoes and still serves/health, which is the CI job that installs.[serve]without.[mcp];build_router()over" cpu\n","cuda "," ","cpu"and unset with a recordingRouterstub.Six mutations of the new code were each caught by named tests: dropping the resident-device read (3 failures), restoring the env read for
device(3), removingdevice_is_preference(3), removingcheckpoint_devices(5), restoring the unstripped env read inbuild_router()(1), and reporting the preference per checkpoint instead ofAgent.device(4).ruff check laya/,compileall,tests/test_lazy_import.py(7 passed) andzensical build --strict --cleanare clean.Overlap
Nine open PRs touch
laya/serve.py, so here is the honest read on the three nearest:cpu_fallback_countin/health, which is §1 of Follow-ups to the scoped OOM fallback: fallback observability + scoped autocast drop #351. Different event, and neither sees the other: a build-time GPU→CPU fallback never enters_infer's OOM branch, so feat(serve): report per-agent CPU fallback counters in /health #487's counters stay at0in exactly the case above; and feat(serve): report per-agent CPU fallback counters in /health #487 leaves"device"as the env echo, so a request-level demote (which restoresself.device) leaves/health'sdeviceunchanged. The two compose — mine says where it computes, theirs says how often it had to step off. Both edit the same dict, so whichever lands second is a few lines of conflict./readywith the sameos.environ.get("LAYA_DEVICE") or "auto"line, which carries the preference-over-fact issue to a second endpoint; commented there.laya/serve.pyand never touch device handling;docs/docker.mdedits in both are in the variable tables, my paragraph is under the healthcheck.Also flagged: #408's
/readyand #487'scpu_fallbacksboth reach forrouter._agents, which is the same private handlelaya/mcp/device.py:router_agentwas written to wrap. Worth converging on that one reader regardless of merge order.