Skip to content

Add self-hostable HTTP server (Jev-compatible /v1/systemone), Nix flake, and NixOS module - #31

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
getmissionctrl:feat/self-hosting-server
Sep 23, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
getmissionctrl:feat/self-hosting-server

Conversation

@commandodev

Copy link
Copy Markdown
Contributor

Summary

Adds an optional, fully backward-compatible way to self-host Laya over an HTTP API that speaks TypeSafe Jev's /v1/systemone wire protocol, plus first-class Nix packaging.

The key observation: Laya's predict() output is already schema-identical to the Jev decision API — choice/score/noul answers and a {input_tokens, output_tokens} usage block. So a client written against Jev can point its baseUrl at a self-hosted Laya and work unchanged; the only missing piece was the HTTP surface itself. This PR adds it (and nothing more — the SDK and model code are untouched).

What's added (all additive)

  • laya/serve.py — a FastAPI app exposing POST /v1/systemone (+ /health) backed by Router.predict, with optional Authorization: Bearer auth. Configured entirely by env vars (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK/API_KEY, and LAYA_THREADS to cap torch intra-op threads for CPU). Heavy imports (fastapi/uvicorn/torch) are deferred, so import laya.serve stays cheap and GPU-free.
  • laya-serve console script + a laya[serve] extra (fastapi, uvicorn) — no new hard dependencies for the core package.
  • Nix flake (flake.nix, nix/package.nix) — builds Laya from the prebuilt CUDA torch-bin wheel (no source compile), plus a dev shell and packages.
  • NixOS module (nix/laya-serve.nix) — a hardened DynamicUser systemd service with CUDA device access, a Hugging Face weight cache, optional bearer token via LoadCredential, and a threads option.
  • tests/test_serve.py — server-shim tests (injected router, real FastAPI TestClient).
  • README — a "Self-Hosting: HTTP Server" section.

Verification

Built and run end-to-end on an RTX 3090 and a 64-core CPU:

  • All three checkpoints preload; English/Hindi requests route correctly (script/lang detection) and return byte-for-byte Jev-shaped payloads.
  • 7/7 shim tests pass.

Latency (warm, HTTP round-trip):

workload RTX 3090 CPU (32-core, tuned)
English, 1 q 23 ms ~142 ms
English, 10 q batched 2.6 ms/q ~48 ms/q
Multilingual, 50 q batched 0.8 ms/q ~17 ms/q

Peak ~705 q/s (English) / ~1270 q/s (multilingual) on one 3090. A CPU thread-count sweep found that oversubscribing logical (hyperthread) cores is a ~4× regression — best to cap LAYA_THREADS at the physical core count (hence the knob).

Notes

  • Entirely opt-in: existing laya.load(...) / Router usage is unchanged, and the server deps only install via pip install laya[serve].
  • Apache-2.0, consistent with the project.

🤖 Generated with Claude Code

@NandhaKishorM

Copy link
Copy Markdown
Owner

I like this one. The observation that predict() is already schema identical to the Jev decision API, so a client can just repoint baseUrl, is the right framing, and I appreciate that it is purely additive and does not touch the SDK or model code.

CI has not run on this branch, and main requires tests (py3.9), tests (py3.11), tests (py3.13), lint and package before merge. Could you rebase on latest main and push so the workflows run?

Note there is an overlapping server PR in #3 that takes an OpenAI compatible approach. I prefer this one since it is smaller and the Jev wire compatibility is the more useful migration story, but I wanted to flag it so you are not surprised.

@commandodev
commandodev force-pushed the feat/self-hosting-server branch from 21ed042 to 2e55b84 Compare September 21, 2026 04:12
@commandodev

Copy link
Copy Markdown
Contributor Author

@NandhaKishorM - that's done now :-)

@commandodev
commandodev force-pushed the feat/self-hosting-server branch from 2e55b84 to 688e805 Compare September 22, 2026 09:48
commandodev and others added 4 commits September 22, 2026 09:50
Laya's predict() output is already schema-compatible with TypeSafe Jev's
decision API, so this adds only the HTTP surface a Jev client needs to talk
to a self-hosted Laya:

- laya/serve.py: FastAPI app exposing POST /v1/systemone (+ /health), calling
  Router.predict, with optional LAYA_API_KEY bearer auth. Config via env
  (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK). Heavy imports are deferred
  so `import laya.serve` stays GPU- and web-dep-free.
- laya-serve console script + laya[serve] extra (fastapi, uvicorn).
- flake.nix: overlay building laya from torch-bin (prebuilt CUDA), a
  laya-serve runner, a dev shell, and nixosModules.default.
- nix/laya-serve.nix: hardened DynamicUser systemd service (services.laya-serve)
  with CUDA device access and a HF weight cache.
- tests/test_serve.py: shim tests (injected router, real FastAPI TestClient).

Verified end-to-end on an RTX 3090: all three checkpoints preload on CUDA,
English/Hindi requests route correctly and return Jev-shaped payloads, 6/6
tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
Hosts that inject `pkgs` via specialArgs (read-only nixpkgs) ignore
module-level `nixpkgs.overlays`, so `pkgs.laya-serve` was missing and the
NixOS toplevel failed to evaluate. Factor the package into nix/package.nix and
have services.laya-serve.package default to `(pkgs.callPackage ./package.nix
{}).laya-serve`, built from the host's own pkgs. The flake overlay/packages now
reuse the same file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
…tuning

Cap torch intra-op threads for CPU inference. serve.py reads LAYA_THREADS and
calls torch.set_num_threads (torch imported only when a limit is set); the NixOS
module exposes `threads` and wires it to LAYA_THREADS + OMP_NUM_THREADS.

Motivated by a thread sweep on a 32-core/64-thread EPYC: oversubscribing the
logical core count is a ~4x latency regression, and single-request latency is
often best below the physical core count (e.g. 8-16), while batched throughput
peaks at the physical count.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, and thanks for waiting. This is the server direction I said I preferred back on #32, and it held up well in review.

What made it the right one to merge: it passes the request body straight to Router.predict, so it accepts every shape the real Jev API does (choice criteria as a list, noul criteria as a dict, unknown fields ignored), which is exactly where the stricter schema in #3 broke. Malformed questions now come back as clear 422s thanks to #183. Your 7 tests pass with the serve extra, and import laya without the extra is unchanged.

One note for a follow-up: it binds 0.0.0.0 and has no auth unless LAYA_API_KEY is set, so a line in the README saying so would help anyone deploying it. Merging.

@NandhaKishorM
NandhaKishorM merged commit d531dc7 into NandhaKishorM:main Sep 23, 2026
NandhaKishorM pushed a commit that referenced this pull request Sep 23, 2026
`POST /v1/systemone` is an `async def` that called `router.predict(...)` directly, so the
whole synchronous torch forward pass ran on the event loop. One request stalled every
other client, `GET /health` included, which defeats the health probe for exactly the
load it exists to report on.

The issue asked for this by name:

> **`run_in_executor` and concurrency protection.** #31 currently calls the router
> synchronously inside the async endpoint, so your executor approach would be a real
> improvement as a small PR against `laya/serve.py`.

So the call now goes through `loop.run_in_executor` on a `ThreadPoolExecutor(max_workers=1)`,
behind an `asyncio.Lock`.

Measured, same app and same stub router with a 0.30 s "forward pass", driving the app
directly on an event loop:

| | before | after |
|---|---|---|
| thread `predict` ran on | `MainThread` (the loop) | `laya-infer_0` |
| `GET /health` worst case while calls are in flight | 44.4 ms | **1.9 ms** |
| wall for 4 concurrent calls | 1248.9 ms | 1204.0 ms |
| answers returned | 4 | 4 |

The wall time staying near-serial is intended, see below. The `/health` figure is the
point: with the call inline it waited for the running forward pass, because the loop
could not start the handler until that returned.

Deliberate choices tested:

- **`run_in_executor` on an explicit pool, not `starlette.concurrency.run_in_threadpool`.** `run_in_threadpool` is shorter, but it uses anyio's shared limiter (40 threads by default), so a burst of requests would queue 40 blocking torch calls rather than one. The issue named `run_in_executor`, and one worker is the right number: a single CPU or GPU `Agent` wants one forward pass at a time. `Router` already guards checkpoint lifecycle itself and deliberately leaves `Agent.system_one` unguarded so concurrent predictions can share a checkpoint — a GPU-shaped choice this endpoint does not have to rely on.
- **The `asyncio.Lock` is created on first request, not in `create_app`.** An `asyncio.Lock` binds to the loop it is first awaited on, and `create_app` can be called before any loop exists (module scope, a preload script, `TestClient` startup). Creating it eagerly is a real hang, not a style question.
- **The gate is wrapped so the lock is not held across the `except`.** Ordering is unchanged: `_check_auth` and body validation still run before the lock, so a 401 or a 400 never waits behind an in-flight forward pass.
- **Throughput is deliberately unchanged.** This is a responsiveness fix, not a speedup: wall time for concurrent calls stays near-serial because there is one worker. Batching is a separate question and belongs with #138.
- **Answers are byte-identical.** The same `router.predict(state, questions, model=model)` is called with the same arguments; only the thread changes. The new tests assert the argument record is unchanged.

Known limitation: the pool is per-app and is never shut down. `uvicorn` exits the process on shutdown, so this is invisible in the shipped server, but an embedder that creates many apps in one process would accumulate one idle thread each. Worth a `lifespan` hook if that becomes real; out of scope for this change.

Verified:

```
$ python -m pytest tests/test_serve.py -q      9 passed      (7 before)
$ python tests/test_criteria.py               72 passed, 0 failed
$ python tests/test_router.py                 all routing tests passed
$ python tests/test_email.py                  46 passed, 0 failed
$ python tests/test_lang_guess.py             52 passed, 0 failed
$ python tests/test_packaging.py              all packaging tests passed
$ python tests/test_empty_questions.py        OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

Both new tests fail against the unchanged module, so they catch this specific defect:

```
FAILED tests/test_serve.py::test_inference_runs_off_the_event_loop
  AssertionError: predict ran on the event loop thread: ['MainThread'] -- one request
  would stall every other client, including GET /health
FAILED tests/test_serve.py::test_health_stays_available_during_inference
  AssertionError: ['MainThread', 'MainThread']
```

A note on how the tests observe this, because the obvious version does not work:
`fastapi.testclient.TestClient` runs the loop in a portal thread and gives each call its
own, so with a blocking endpoint `/health` still answers in single-digit milliseconds and
a `TestClient`-based test passes either way. I wrote it that way first and it was a false
green. The tests therefore drive the app on a real loop with `httpx.ASGITransport`, where
inline work runs on `MainThread` and offloaded work runs on a worker.

`tests/test_serve.py` is currently run by no workflow — neither `ci.yml` nor `release.yml`
names it — so this fix and its tests are not yet gated in CI. It needs `pytest` and the
`serve` extra, neither of which the `test` job installs. Happy to wire up
`pip install -e ".[serve]"` plus a `python -m pytest tests/test_serve.py` step in both
lanes as a follow-up if you want it in the same PR, but that decision is yours and #184 is
already touching that wiring.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 23, 2026
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`,
so the `serve` extra — and therefore the `laya-serve` console script — was not in the
image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1`
pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview
branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml`
is empty).

`NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose
service that runs `laya-serve`, so the container speaks the Jev-compatible
`/v1/systemone` out of the box."

```
docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json
```

End-to-end verified against a real container, not just a config parse:

```
$ docker compose -f compose.yaml -f compose.http.yaml ps
repo-laya-serve-1  Up  0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp

$ curl -s localhost:18000/health
{"status":"ok","loaded":[],"device":"cpu"}

$ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json
422                       # no weights cached and the container is --network none

no header      HTTP 401
wrong token    HTTP 401
correct token  HTTP 422   # auth passed, then the offline model load fails as designed
```

Deliberate choices tested:

- **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation.
- **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service.
- **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment.
- **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default.
- **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment.
- **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back.

Known limitations:

- `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls.
- The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB.
- I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this.

Also included, both from the same gap:

- `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table.
- `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist.

Verified:

```
$ python tests/test_packaging.py            34 passed, 0 failed    (15 before)
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ docker compose config --quiet             OK for all six combinations,
                                            including the new -f compose.http.yaml
$ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \
    python tests/test_docker_entrypoint.py  Ran 5 tests ... OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

`tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds
the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that
suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml`
run it and where the run above was made.
NandhaKishorM pushed a commit that referenced this pull request Sep 23, 2026
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`,
so the `serve` extra — and therefore the `laya-serve` console script — was not in the
image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1`
pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview
branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml`
is empty).

`#45` named this as the follow-up: "A natural follow-up now that #31 has landed: a Compose
service that runs `laya-serve`, so the container speaks the Jev-compatible
`/v1/systemone` out of the box."

```
docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json
```

End-to-end verified against a real container, not just a config parse:

```
$ docker compose -f compose.yaml -f compose.http.yaml ps
repo-laya-serve-1  Up  0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp

$ curl -s localhost:18000/health
{"status":"ok","loaded":[],"device":"cpu"}

$ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json
422                       # no weights cached and the container is --network none

no header      HTTP 401
wrong token    HTTP 401
correct token  HTTP 422   # auth passed, then the offline model load fails as designed
```

Deliberate choices tested:

- **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation.
- **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service.
- **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment.
- **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default.
- **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment.
- **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back.

Known limitations:

- `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls.
- The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what #45 asked for. It is worth ~15 MB.
- I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this.

Also included, both from the same gap:

- `docs/docker.md`'s "HTTP serving" section said serving was "pending #3 or #31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table.
- `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist.

Verified:

```
$ python tests/test_packaging.py            34 passed, 0 failed    (15 before)
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ docker compose config --quiet             OK for all six combinations,
                                            including the new -f compose.http.yaml
$ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \
    python tests/test_docker_entrypoint.py  Ran 5 tests ... OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

`tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds
the command line without the quoting `subprocess` applied, which is why #212 skips that
suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml`
run it and where the run above was made.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…ke, and NixOS module (NandhaKishorM#31)

* feat: self-hostable server + Nix flake (Jev-compatible /v1/systemone)

Laya's predict() output is already schema-compatible with TypeSafe Jev's
decision API, so this adds only the HTTP surface a Jev client needs to talk
to a self-hosted Laya:

- laya/serve.py: FastAPI app exposing POST /v1/systemone (+ /health), calling
  Router.predict, with optional LAYA_API_KEY bearer auth. Config via env
  (LAYA_HOST/PORT/DEVICE/PRELOAD/MODELS/AUTO_TASK). Heavy imports are deferred
  so `import laya.serve` stays GPU- and web-dep-free.
- laya-serve console script + laya[serve] extra (fastapi, uvicorn).
- flake.nix: overlay building laya from torch-bin (prebuilt CUDA), a
  laya-serve runner, a dev shell, and nixosModules.default.
- nix/laya-serve.nix: hardened DynamicUser systemd service (services.laya-serve)
  with CUDA device access and a HF weight cache.
- tests/test_serve.py: shim tests (injected router, real FastAPI TestClient).

Verified end-to-end on an RTX 3090: all three checkpoints preload on CUDA,
English/Hindi requests route correctly and return Jev-shaped payloads, 6/6
tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe

* fix(nix): build the service package from the host pkgs, not an overlay

Hosts that inject `pkgs` via specialArgs (read-only nixpkgs) ignore
module-level `nixpkgs.overlays`, so `pkgs.laya-serve` was missing and the
NixOS toplevel failed to evaluate. Factor the package into nix/package.nix and
have services.laya-serve.package default to `(pkgs.callPackage ./package.nix
{}).laya-serve`, built from the host's own pkgs. The flake overlay/packages now
reuse the same file.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe

* feat(serve): LAYA_THREADS knob + services.laya-serve.threads for CPU tuning

Cap torch intra-op threads for CPU inference. serve.py reads LAYA_THREADS and
calls torch.set_num_threads (torch imported only when a limit is set); the NixOS
module exposes `threads` and wires it to LAYA_THREADS + OMP_NUM_THREADS.

Motivated by a thread sweep on a 32-core/64-thread EPYC: oversubscribing the
logical core count is a ~4x latency regression, and single-request latency is
often best below the physical core count (e.g. 8-16), while batched throughput
peaks at the physical count.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe

* chore(nix): commit flake.lock pinning nixpkgs to ccad53c

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NnoUnPhBMoS4S7eYWHYfe

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
`POST /v1/systemone` is an `async def` that called `router.predict(...)` directly, so the
whole synchronous torch forward pass ran on the event loop. One request stalled every
other client, `GET /health` included, which defeats the health probe for exactly the
load it exists to report on.

The issue asked for this by name:

> **`run_in_executor` and concurrency protection.** NandhaKishorM#31 currently calls the router
> synchronously inside the async endpoint, so your executor approach would be a real
> improvement as a small PR against `laya/serve.py`.

So the call now goes through `loop.run_in_executor` on a `ThreadPoolExecutor(max_workers=1)`,
behind an `asyncio.Lock`.

Measured, same app and same stub router with a 0.30 s "forward pass", driving the app
directly on an event loop:

| | before | after |
|---|---|---|
| thread `predict` ran on | `MainThread` (the loop) | `laya-infer_0` |
| `GET /health` worst case while calls are in flight | 44.4 ms | **1.9 ms** |
| wall for 4 concurrent calls | 1248.9 ms | 1204.0 ms |
| answers returned | 4 | 4 |

The wall time staying near-serial is intended, see below. The `/health` figure is the
point: with the call inline it waited for the running forward pass, because the loop
could not start the handler until that returned.

Deliberate choices tested:

- **`run_in_executor` on an explicit pool, not `starlette.concurrency.run_in_threadpool`.** `run_in_threadpool` is shorter, but it uses anyio's shared limiter (40 threads by default), so a burst of requests would queue 40 blocking torch calls rather than one. The issue named `run_in_executor`, and one worker is the right number: a single CPU or GPU `Agent` wants one forward pass at a time. `Router` already guards checkpoint lifecycle itself and deliberately leaves `Agent.system_one` unguarded so concurrent predictions can share a checkpoint — a GPU-shaped choice this endpoint does not have to rely on.
- **The `asyncio.Lock` is created on first request, not in `create_app`.** An `asyncio.Lock` binds to the loop it is first awaited on, and `create_app` can be called before any loop exists (module scope, a preload script, `TestClient` startup). Creating it eagerly is a real hang, not a style question.
- **The gate is wrapped so the lock is not held across the `except`.** Ordering is unchanged: `_check_auth` and body validation still run before the lock, so a 401 or a 400 never waits behind an in-flight forward pass.
- **Throughput is deliberately unchanged.** This is a responsiveness fix, not a speedup: wall time for concurrent calls stays near-serial because there is one worker. Batching is a separate question and belongs with NandhaKishorM#138.
- **Answers are byte-identical.** The same `router.predict(state, questions, model=model)` is called with the same arguments; only the thread changes. The new tests assert the argument record is unchanged.

Known limitation: the pool is per-app and is never shut down. `uvicorn` exits the process on shutdown, so this is invisible in the shipped server, but an embedder that creates many apps in one process would accumulate one idle thread each. Worth a `lifespan` hook if that becomes real; out of scope for this change.

Verified:

```
$ python -m pytest tests/test_serve.py -q      9 passed      (7 before)
$ python tests/test_criteria.py               72 passed, 0 failed
$ python tests/test_router.py                 all routing tests passed
$ python tests/test_email.py                  46 passed, 0 failed
$ python tests/test_lang_guess.py             52 passed, 0 failed
$ python tests/test_packaging.py              all packaging tests passed
$ python tests/test_empty_questions.py        OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

Both new tests fail against the unchanged module, so they catch this specific defect:

```
FAILED tests/test_serve.py::test_inference_runs_off_the_event_loop
  AssertionError: predict ran on the event loop thread: ['MainThread'] -- one request
  would stall every other client, including GET /health
FAILED tests/test_serve.py::test_health_stays_available_during_inference
  AssertionError: ['MainThread', 'MainThread']
```

A note on how the tests observe this, because the obvious version does not work:
`fastapi.testclient.TestClient` runs the loop in a portal thread and gives each call its
own, so with a blocking endpoint `/health` still answers in single-digit milliseconds and
a `TestClient`-based test passes either way. I wrote it that way first and it was a false
green. The tests therefore drive the app on a real loop with `httpx.ASGITransport`, where
inline work runs on `MainThread` and offloaded work runs on a worker.

`tests/test_serve.py` is currently run by no workflow — neither `ci.yml` nor `release.yml`
names it — so this fix and its tests are not yet gated in CI. It needs `pytest` and the
`serve` extra, neither of which the `test` job installs. Happy to wire up
`pip install -e ".[serve]"` plus a `python -m pytest tests/test_serve.py` step in both
lanes as a follow-up if you want it in the same PR, but that decision is yours and NandhaKishorM#184 is
already touching that wiring.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…#234)

The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`,
so the `serve` extra — and therefore the `laya-serve` console script — was not in the
image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1`
pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview
branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml`
is empty).

`NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose
service that runs `laya-serve`, so the container speaks the Jev-compatible
`/v1/systemone` out of the box."

```
docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json
```

End-to-end verified against a real container, not just a config parse:

```
$ docker compose -f compose.yaml -f compose.http.yaml ps
repo-laya-serve-1  Up  0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp

$ curl -s localhost:18000/health
{"status":"ok","loaded":[],"device":"cpu"}

$ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json
422                       # no weights cached and the container is --network none

no header      HTTP 401
wrong token    HTTP 401
correct token  HTTP 422   # auth passed, then the offline model load fails as designed
```

Deliberate choices tested:

- **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation.
- **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service.
- **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment.
- **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default.
- **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment.
- **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back.

Known limitations:

- `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls.
- The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB.
- I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this.

Also included, both from the same gap:

- `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table.
- `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist.

Verified:

```
$ python tests/test_packaging.py            34 passed, 0 failed    (15 before)
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ docker compose config --quiet             OK for all six combinations,
                                            including the new -f compose.http.yaml
$ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \
    python tests/test_docker_entrypoint.py  Ran 5 tests ... OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

`tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds
the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that
suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml`
run it and where the run above was made.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants