Repository navigation
Add self-hostable HTTP server (Jev-compatible /v1/systemone), Nix flake, and NixOS module - #31
Conversation
|
I like this one. The observation that CI has not run on this branch, and Note there is an overlapping server PR in #3 that takes an OpenAI compatible approach. I prefer this one since it is smaller and the Jev wire compatibility is the more useful migration story, but I wanted to flag it so you are not surprised. |
21ed042 to
2e55b84
Compare
|
@NandhaKishorM - that's done now :-) |
2e55b84 to
688e805
Compare
Laya's predict() output is already schema-compatible with TypeSafe Jev's decision API, so this adds only the HTTP surface a Jev client needs to talk to a self-hosted Laya: - laya/serve.py: FastAPI app exposing POST /v1/systemone (+ /health), calling Router.predict, with optional LAYA_API_KEY bearer auth. Config via env (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK). Heavy imports are deferred so `import laya.serve` stays GPU- and web-dep-free. - laya-serve console script + laya[serve] extra (fastapi, uvicorn). - flake.nix: overlay building laya from torch-bin (prebuilt CUDA), a laya-serve runner, a dev shell, and nixosModules.default. - nix/laya-serve.nix: hardened DynamicUser systemd service (services.laya-serve) with CUDA device access and a HF weight cache. - tests/test_serve.py: shim tests (injected router, real FastAPI TestClient). Verified end-to-end on an RTX 3090: all three checkpoints preload on CUDA, English/Hindi requests route correctly and return Jev-shaped payloads, 6/6 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
Hosts that inject `pkgs` via specialArgs (read-only nixpkgs) ignore
module-level `nixpkgs.overlays`, so `pkgs.laya-serve` was missing and the
NixOS toplevel failed to evaluate. Factor the package into nix/package.nix and
have services.laya-serve.package default to `(pkgs.callPackage ./package.nix
{}).laya-serve`, built from the host's own pkgs. The flake overlay/packages now
reuse the same file.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
…tuning Cap torch intra-op threads for CPU inference. serve.py reads LAYA_THREADS and calls torch.set_num_threads (torch imported only when a limit is set); the NixOS module exposes `threads` and wires it to LAYA_THREADS + OMP_NUM_THREADS. Motivated by a thread sweep on a 32-core/64-thread EPYC: oversubscribing the logical core count is a ~4x latency regression, and single-request latency is often best below the physical core count (e.g. 8-16), while batched throughput peaks at the physical count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
688e805 to
aa571af
Compare
|
Thank you, and thanks for waiting. This is the server direction I said I preferred back on #32, and it held up well in review. What made it the right one to merge: it passes the request body straight to One note for a follow-up: it binds |
`POST /v1/systemone` is an `async def` that called `router.predict(...)` directly, so the whole synchronous torch forward pass ran on the event loop. One request stalled every other client, `GET /health` included, which defeats the health probe for exactly the load it exists to report on. The issue asked for this by name: > **`run_in_executor` and concurrency protection.** #31 currently calls the router > synchronously inside the async endpoint, so your executor approach would be a real > improvement as a small PR against `laya/serve.py`. So the call now goes through `loop.run_in_executor` on a `ThreadPoolExecutor(max_workers=1)`, behind an `asyncio.Lock`. Measured, same app and same stub router with a 0.30 s "forward pass", driving the app directly on an event loop: | | before | after | |---|---|---| | thread `predict` ran on | `MainThread` (the loop) | `laya-infer_0` | | `GET /health` worst case while calls are in flight | 44.4 ms | **1.9 ms** | | wall for 4 concurrent calls | 1248.9 ms | 1204.0 ms | | answers returned | 4 | 4 | The wall time staying near-serial is intended, see below. The `/health` figure is the point: with the call inline it waited for the running forward pass, because the loop could not start the handler until that returned. Deliberate choices tested: - **`run_in_executor` on an explicit pool, not `starlette.concurrency.run_in_threadpool`.** `run_in_threadpool` is shorter, but it uses anyio's shared limiter (40 threads by default), so a burst of requests would queue 40 blocking torch calls rather than one. The issue named `run_in_executor`, and one worker is the right number: a single CPU or GPU `Agent` wants one forward pass at a time. `Router` already guards checkpoint lifecycle itself and deliberately leaves `Agent.system_one` unguarded so concurrent predictions can share a checkpoint — a GPU-shaped choice this endpoint does not have to rely on. - **The `asyncio.Lock` is created on first request, not in `create_app`.** An `asyncio.Lock` binds to the loop it is first awaited on, and `create_app` can be called before any loop exists (module scope, a preload script, `TestClient` startup). Creating it eagerly is a real hang, not a style question. - **The gate is wrapped so the lock is not held across the `except`.** Ordering is unchanged: `_check_auth` and body validation still run before the lock, so a 401 or a 400 never waits behind an in-flight forward pass. - **Throughput is deliberately unchanged.** This is a responsiveness fix, not a speedup: wall time for concurrent calls stays near-serial because there is one worker. Batching is a separate question and belongs with #138. - **Answers are byte-identical.** The same `router.predict(state, questions, model=model)` is called with the same arguments; only the thread changes. The new tests assert the argument record is unchanged. Known limitation: the pool is per-app and is never shut down. `uvicorn` exits the process on shutdown, so this is invisible in the shipped server, but an embedder that creates many apps in one process would accumulate one idle thread each. Worth a `lifespan` hook if that becomes real; out of scope for this change. Verified: ``` $ python -m pytest tests/test_serve.py -q 9 passed (7 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ python tests/test_lang_guess.py 52 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python tests/test_empty_questions.py OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` Both new tests fail against the unchanged module, so they catch this specific defect: ``` FAILED tests/test_serve.py::test_inference_runs_off_the_event_loop AssertionError: predict ran on the event loop thread: ['MainThread'] -- one request would stall every other client, including GET /health FAILED tests/test_serve.py::test_health_stays_available_during_inference AssertionError: ['MainThread', 'MainThread'] ``` A note on how the tests observe this, because the obvious version does not work: `fastapi.testclient.TestClient` runs the loop in a portal thread and gives each call its own, so with a blocking endpoint `/health` still answers in single-digit milliseconds and a `TestClient`-based test passes either way. I wrote it that way first and it was a false green. The tests therefore drive the app on a real loop with `httpx.ASGITransport`, where inline work runs on `MainThread` and offloaded work runs on a worker. `tests/test_serve.py` is currently run by no workflow — neither `ci.yml` nor `release.yml` names it — so this fix and its tests are not yet gated in CI. It needs `pytest` and the `serve` extra, neither of which the `test` job installs. Happy to wire up `pip install -e ".[serve]"` plus a `python -m pytest tests/test_serve.py` step in both lanes as a follow-up if you want it in the same PR, but that decision is yours and #184 is already touching that wiring. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`, so the `serve` extra — and therefore the `laya-serve` console script — was not in the image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1` pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml` is empty). `NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose service that runs `laya-serve`, so the container speaks the Jev-compatible `/v1/systemone` out of the box." ``` docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve curl -s localhost:8000/health curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \ --data @examples/docker/request.json ``` End-to-end verified against a real container, not just a config parse: ``` $ docker compose -f compose.yaml -f compose.http.yaml ps repo-laya-serve-1 Up 0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp $ curl -s localhost:18000/health {"status":"ok","loaded":[],"device":"cpu"} $ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json 422 # no weights cached and the container is --network none no header HTTP 401 wrong token HTTP 401 correct token HTTP 422 # auth passed, then the offline model load fails as designed ``` Deliberate choices tested: - **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation. - **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service. - **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment. - **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default. - **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment. - **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back. Known limitations: - `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls. - The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB. - I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this. Also included, both from the same gap: - `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table. - `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist. Verified: ``` $ python tests/test_packaging.py 34 passed, 0 failed (15 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ docker compose config --quiet OK for all six combinations, including the new -f compose.http.yaml $ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \ python tests/test_docker_entrypoint.py Ran 5 tests ... OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` `tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml` run it and where the run above was made.
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`, so the `serve` extra — and therefore the `laya-serve` console script — was not in the image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1` pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml` is empty). `#45` named this as the follow-up: "A natural follow-up now that #31 has landed: a Compose service that runs `laya-serve`, so the container speaks the Jev-compatible `/v1/systemone` out of the box." ``` docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve curl -s localhost:8000/health curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \ --data @examples/docker/request.json ``` End-to-end verified against a real container, not just a config parse: ``` $ docker compose -f compose.yaml -f compose.http.yaml ps repo-laya-serve-1 Up 0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp $ curl -s localhost:18000/health {"status":"ok","loaded":[],"device":"cpu"} $ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json 422 # no weights cached and the container is --network none no header HTTP 401 wrong token HTTP 401 correct token HTTP 422 # auth passed, then the offline model load fails as designed ``` Deliberate choices tested: - **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation. - **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service. - **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment. - **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default. - **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment. - **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back. Known limitations: - `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls. - The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what #45 asked for. It is worth ~15 MB. - I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this. Also included, both from the same gap: - `docs/docker.md`'s "HTTP serving" section said serving was "pending #3 or #31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table. - `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist. Verified: ``` $ python tests/test_packaging.py 34 passed, 0 failed (15 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ docker compose config --quiet OK for all six combinations, including the new -f compose.http.yaml $ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \ python tests/test_docker_entrypoint.py Ran 5 tests ... OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` `tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds the command line without the quoting `subprocess` applied, which is why #212 skips that suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml` run it and where the run above was made. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
…ke, and NixOS module (NandhaKishorM#31) * feat: self-hostable server + Nix flake (Jev-compatible /v1/systemone) Laya's predict() output is already schema-compatible with TypeSafe Jev's decision API, so this adds only the HTTP surface a Jev client needs to talk to a self-hosted Laya: - laya/serve.py: FastAPI app exposing POST /v1/systemone (+ /health), calling Router.predict, with optional LAYA_API_KEY bearer auth. Config via env (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK). Heavy imports are deferred so `import laya.serve` stays GPU- and web-dep-free. - laya-serve console script + laya[serve] extra (fastapi, uvicorn). - flake.nix: overlay building laya from torch-bin (prebuilt CUDA), a laya-serve runner, a dev shell, and nixosModules.default. - nix/laya-serve.nix: hardened DynamicUser systemd service (services.laya-serve) with CUDA device access and a HF weight cache. - tests/test_serve.py: shim tests (injected router, real FastAPI TestClient). Verified end-to-end on an RTX 3090: all three checkpoints preload on CUDA, English/Hindi requests route correctly and return Jev-shaped payloads, 6/6 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * fix(nix): build the service package from the host pkgs, not an overlay Hosts that inject `pkgs` via specialArgs (read-only nixpkgs) ignore module-level `nixpkgs.overlays`, so `pkgs.laya-serve` was missing and the NixOS toplevel failed to evaluate. Factor the package into nix/package.nix and have services.laya-serve.package default to `(pkgs.callPackage ./package.nix {}).laya-serve`, built from the host's own pkgs. The flake overlay/packages now reuse the same file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * feat(serve): LAYA_THREADS knob + services.laya-serve.threads for CPU tuning Cap torch intra-op threads for CPU inference. serve.py reads LAYA_THREADS and calls torch.set_num_threads (torch imported only when a limit is set); the NixOS module exposes `threads` and wires it to LAYA_THREADS + OMP_NUM_THREADS. Motivated by a thread sweep on a 32-core/64-thread EPYC: oversubscribing the logical core count is a ~4x latency regression, and single-request latency is often best below the physical core count (e.g. 8-16), while batched throughput peaks at the physical count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe * chore(nix): commit flake.lock pinning nixpkgs to ccad53c Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`POST /v1/systemone` is an `async def` that called `router.predict(...)` directly, so the whole synchronous torch forward pass ran on the event loop. One request stalled every other client, `GET /health` included, which defeats the health probe for exactly the load it exists to report on. The issue asked for this by name: > **`run_in_executor` and concurrency protection.** NandhaKishorM#31 currently calls the router > synchronously inside the async endpoint, so your executor approach would be a real > improvement as a small PR against `laya/serve.py`. So the call now goes through `loop.run_in_executor` on a `ThreadPoolExecutor(max_workers=1)`, behind an `asyncio.Lock`. Measured, same app and same stub router with a 0.30 s "forward pass", driving the app directly on an event loop: | | before | after | |---|---|---| | thread `predict` ran on | `MainThread` (the loop) | `laya-infer_0` | | `GET /health` worst case while calls are in flight | 44.4 ms | **1.9 ms** | | wall for 4 concurrent calls | 1248.9 ms | 1204.0 ms | | answers returned | 4 | 4 | The wall time staying near-serial is intended, see below. The `/health` figure is the point: with the call inline it waited for the running forward pass, because the loop could not start the handler until that returned. Deliberate choices tested: - **`run_in_executor` on an explicit pool, not `starlette.concurrency.run_in_threadpool`.** `run_in_threadpool` is shorter, but it uses anyio's shared limiter (40 threads by default), so a burst of requests would queue 40 blocking torch calls rather than one. The issue named `run_in_executor`, and one worker is the right number: a single CPU or GPU `Agent` wants one forward pass at a time. `Router` already guards checkpoint lifecycle itself and deliberately leaves `Agent.system_one` unguarded so concurrent predictions can share a checkpoint — a GPU-shaped choice this endpoint does not have to rely on. - **The `asyncio.Lock` is created on first request, not in `create_app`.** An `asyncio.Lock` binds to the loop it is first awaited on, and `create_app` can be called before any loop exists (module scope, a preload script, `TestClient` startup). Creating it eagerly is a real hang, not a style question. - **The gate is wrapped so the lock is not held across the `except`.** Ordering is unchanged: `_check_auth` and body validation still run before the lock, so a 401 or a 400 never waits behind an in-flight forward pass. - **Throughput is deliberately unchanged.** This is a responsiveness fix, not a speedup: wall time for concurrent calls stays near-serial because there is one worker. Batching is a separate question and belongs with NandhaKishorM#138. - **Answers are byte-identical.** The same `router.predict(state, questions, model=model)` is called with the same arguments; only the thread changes. The new tests assert the argument record is unchanged. Known limitation: the pool is per-app and is never shut down. `uvicorn` exits the process on shutdown, so this is invisible in the shipped server, but an embedder that creates many apps in one process would accumulate one idle thread each. Worth a `lifespan` hook if that becomes real; out of scope for this change. Verified: ``` $ python -m pytest tests/test_serve.py -q 9 passed (7 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ python tests/test_lang_guess.py 52 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python tests/test_empty_questions.py OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` Both new tests fail against the unchanged module, so they catch this specific defect: ``` FAILED tests/test_serve.py::test_inference_runs_off_the_event_loop AssertionError: predict ran on the event loop thread: ['MainThread'] -- one request would stall every other client, including GET /health FAILED tests/test_serve.py::test_health_stays_available_during_inference AssertionError: ['MainThread', 'MainThread'] ``` A note on how the tests observe this, because the obvious version does not work: `fastapi.testclient.TestClient` runs the loop in a portal thread and gives each call its own, so with a blocking endpoint `/health` still answers in single-digit milliseconds and a `TestClient`-based test passes either way. I wrote it that way first and it was a false green. The tests therefore drive the app on a real loop with `httpx.ASGITransport`, where inline work runs on `MainThread` and offloaded work runs on a worker. `tests/test_serve.py` is currently run by no workflow — neither `ci.yml` nor `release.yml` names it — so this fix and its tests are not yet gated in CI. It needs `pytest` and the `serve` extra, neither of which the `test` job installs. Happy to wire up `pip install -e ".[serve]"` plus a `python -m pytest tests/test_serve.py` step in both lanes as a follow-up if you want it in the same PR, but that decision is yours and NandhaKishorM#184 is already touching that wiring. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
…#234) The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`, so the `serve` extra — and therefore the `laya-serve` console script — was not in the image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1` pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml` is empty). `NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose service that runs `laya-serve`, so the container speaks the Jev-compatible `/v1/systemone` out of the box." ``` docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve curl -s localhost:8000/health curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \ --data @examples/docker/request.json ``` End-to-end verified against a real container, not just a config parse: ``` $ docker compose -f compose.yaml -f compose.http.yaml ps repo-laya-serve-1 Up 0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp $ curl -s localhost:18000/health {"status":"ok","loaded":[],"device":"cpu"} $ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json 422 # no weights cached and the container is --network none no header HTTP 401 wrong token HTTP 401 correct token HTTP 422 # auth passed, then the offline model load fails as designed ``` Deliberate choices tested: - **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation. - **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service. - **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment. - **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default. - **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment. - **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back. Known limitations: - `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls. - The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB. - I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this. Also included, both from the same gap: - `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table. - `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist. Verified: ``` $ python tests/test_packaging.py 34 passed, 0 failed (15 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ docker compose config --quiet OK for all six combinations, including the new -f compose.http.yaml $ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \ python tests/test_docker_entrypoint.py Ran 5 tests ... OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` `tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml` run it and where the run above was made. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
Summary
Adds an optional, fully backward-compatible way to self-host Laya over an HTTP API that speaks TypeSafe Jev's
/v1/systemonewire protocol, plus first-class Nix packaging.The key observation: Laya's
predict()output is already schema-identical to the Jev decision API —choice/score/noulanswers and a{input_tokens, output_tokens}usage block. So a client written against Jev can point itsbaseUrlat a self-hosted Laya and work unchanged; the only missing piece was the HTTP surface itself. This PR adds it (and nothing more — the SDK and model code are untouched).What's added (all additive)
laya/serve.py— a FastAPI app exposingPOST /v1/systemone(+/health) backed byRouter.predict, with optionalAuthorization: Bearerauth. Configured entirely by env vars (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK/API_KEY, andLAYA_THREADSto cap torch intra-op threads for CPU). Heavy imports (fastapi/uvicorn/torch) are deferred, soimport laya.servestays cheap and GPU-free.laya-serveconsole script + alaya[serve]extra (fastapi,uvicorn) — no new hard dependencies for the core package.flake.nix,nix/package.nix) — builds Laya from the prebuilt CUDAtorch-binwheel (no source compile), plus a dev shell and packages.nix/laya-serve.nix) — a hardenedDynamicUsersystemd service with CUDA device access, a Hugging Face weight cache, optional bearer token viaLoadCredential, and athreadsoption.tests/test_serve.py— server-shim tests (injected router, real FastAPI TestClient).Verification
Built and run end-to-end on an RTX 3090 and a 64-core CPU:
Latency (warm, HTTP round-trip):
Peak ~705 q/s (English) / ~1270 q/s (multilingual) on one 3090. A CPU thread-count sweep found that oversubscribing logical (hyperthread) cores is a ~4× regression — best to cap
LAYA_THREADSat the physical core count (hence the knob).Notes
laya.load(...)/Routerusage is unchanged, and the server deps only install viapip install laya[serve].🤖 Generated with Claude Code