Repository navigation
feat(ci): run the test suite on Windows as well as Linux - #212
Conversation
|
Thank you. Windows is where #123 and #140 both lived unseen, and a separate job that leaves the required |
Every workflow in this repository runs on `ubuntu-latest` and nowhere else. A test that only fails off Linux is therefore invisible until a user hits it, and that has already happened twice: tests/test_download.py (NandhaKishorM#140, since fixed and merged) and the Windows + Python 3.14 segfault tracked in NandhaKishorM#123. The change: * A new `test-windows` job on `windows-latest`, Python 3.11. * Deliberately a separate job, not an extra matrix axis on `test`. That leaves `test`'s name -- and therefore the `tests (pyX.Y)` required checks -- byte for byte as they are, so branch protection needs no edit. It runs one interpreter because the variable that was untested is the OS, not the Python version. * It runs the same suites as the Linux job. The list is guarded with `-f` so a suite added by another open branch cannot make this job red on a tree that does not have it yet. * torch installs plainly on Windows. The `+cpu` index used on Linux is Linux/macOS only; on Windows the default PyPI wheel is already CPU-only, so the plain install is both correct and what a Windows user gets. * Every `run` step declares `shell: bash`. The default shell on a Windows runner is PowerShell, where `python -c "...; ..."` and a multi-line script are not the same command. GitHub's Windows runners provide Git Bash. `test` is untouched, so Linux behaviour does not move. One suite is skipped, with the reason recorded in the workflow: test_docker_entrypoint It drives `docker/entrypoint.py`, which only ever runs inside a Linux container, through `os.execvp`. On Windows `exec*` rebuilds the command line from argv without re-applying the quoting `subprocess` used to build it, so its `-c "<script>"` argument arrives split at the first space and the child exits with "SyntaxError: invalid syntax" before printing. I confirmed the environment passthrough the suite exists to check is unaffected, and that replacing `execvp` with `subprocess.run` makes it pass -- but that is a change to a container script and belongs in its own PR rather than here. Verified on Windows 11, Python 3.12, torch 2.14.0, transformers 5.17.0, against today's main (which now carries 16 suites). Every suite this job runs passes: test_router, test_criteria, test_download, test_shortlist, test_decision_model, test_packaging, test_lang_guess, test_calibration_persistence, test_context_manager, test_criteria_normalization, test_empty_questions, test_router_memory, test_shortlist_cosine, test_temperature_loading, test_email 14 in the loop plus test_email as its own step; none skipped except the one above. `python -m build`, `twine check` and the import check also pass on Windows. This branch is rebased onto current main. It was opened when NandhaKishorM#211 had not landed, so it previously carried the NandhaKishorM#140 fix to tests/test_download.py; NandhaKishorM#140 merged, so that change is now identical to main and drops out. The diff is `.github/workflows/ci.yml` only.
b145e83 to
2f73a2b
Compare
|
Rebased onto current Two things I found while rebasing, because today's The Linux job now runs 16 suites, not 7. The Windows job now runs all of them, minus one. The list is guarded with One suite cannot pass on Windows, and it is worth knowing which. It drives I have skipped it in the Windows job with the reason in the workflow, rather than change Verified on Windows 11, Python 3.12, torch 2.14.0, transformers 5.17.0, against today's I also checked that |
|
Rebase confirmed against the current All 14 checks pass on that head, including the job this PR exists to add:
Two things I noticed while waiting that you may want, both opt-in and neither in this diff: The Windows job runs Python 3.11 only, so it does not cover #123. That issue is specific to Windows + 3.14, and 3.11-on-Windows would not have reproduced the
|
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`, so the `serve` extra — and therefore the `laya-serve` console script — was not in the image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1` pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml` is empty). `NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose service that runs `laya-serve`, so the container speaks the Jev-compatible `/v1/systemone` out of the box." ``` docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve curl -s localhost:8000/health curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \ --data @examples/docker/request.json ``` End-to-end verified against a real container, not just a config parse: ``` $ docker compose -f compose.yaml -f compose.http.yaml ps repo-laya-serve-1 Up 0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp $ curl -s localhost:18000/health {"status":"ok","loaded":[],"device":"cpu"} $ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json 422 # no weights cached and the container is --network none no header HTTP 401 wrong token HTTP 401 correct token HTTP 422 # auth passed, then the offline model load fails as designed ``` Deliberate choices tested: - **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation. - **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service. - **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment. - **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default. - **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment. - **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back. Known limitations: - `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls. - The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB. - I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this. Also included, both from the same gap: - `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table. - `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist. Verified: ``` $ python tests/test_packaging.py 34 passed, 0 failed (15 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ docker compose config --quiet OK for all six combinations, including the new -f compose.http.yaml $ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \ python tests/test_docker_entrypoint.py Ran 5 tests ... OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` `tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml` run it and where the run above was made.
NandhaKishorM#208 ask) The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so the before/after is visible. `research/results/cpu_51_language_sweep_clamped.json` holds the committed columns, an `--unclamped` re-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases. committed unclamped re-run clamped re-run 0.2269 0.2269 0.2269 macro accuracy 0.7331 0.7331 0.5709 macro ECE 0.2053 0.2053 0.2053 macro F1 Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in 51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and this one 2.14.0. The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower. `choice:11+` is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at all: a temperature-scaled softmax has the same argmax at every positive temperature. Worth recording because it cuts against the reading that the clamp only distorts the published number: `acc_at_50_coverage` is the one rank-quality column that consumes the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98. The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the committed columns stopped matching the unclamped re-run or if the clamped re-run started matching them. Deliberate choices tested: - `bench_local.py` was not modified. It produced the committed file, and rewriting it to reproduce its own pre-clamp output would make the old and new runs come from different scripts. `laya_eval.py` already has `--unclamped` for this, and the three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are recorded in the new file rather than quietly dropped. - The clamped re-run is committed alongside rather than replacing anything, so `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite. - `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is wired into both lanes, and this suite is currently in neither; adding it here would collide with that PR and with the verbatim `test` job in NandhaKishorM#212. Known limitation: only the `english` checkpoint was re-run. The committed file's `part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks are in `part_a`, so the multilingual half is untouched and unmeasured here. Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.
#208 ask) (#222) * research: re-run the 51-language sweep in both temperature regimes (the #208 ask) The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from before #42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer reproduce. This adds the re-run #208 asked for, with the old file left in place so the before/after is visible. `research/results/cpu_51_language_sweep_clamped.json` holds the committed columns, an `--unclamped` re-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases. committed unclamped re-run clamped re-run 0.2269 0.2269 0.2269 macro accuracy 0.7331 0.7331 0.5709 macro ECE 0.2053 0.2053 0.2053 macro F1 Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in 51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and this one 2.14.0. The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower. `choice:11+` is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at all: a temperature-scaled softmax has the same argmax at every positive temperature. Worth recording because it cuts against the reading that the clamp only distorts the published number: `acc_at_50_coverage` is the one rank-quality column that consumes the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98. The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the committed columns stopped matching the unclamped re-run or if the clamped re-run started matching them. Deliberate choices tested: - `bench_local.py` was not modified. It produced the committed file, and rewriting it to reproduce its own pre-clamp output would make the old and new runs come from different scripts. `laya_eval.py` already has `--unclamped` for this, and the three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are recorded in the new file rather than quietly dropped. - The clamped re-run is committed alongside rather than replacing anything, so `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite. - `ci.yml` is untouched. #184 is adding the check that every model-free suite is wired into both lanes, and this suite is currently in neither; adding it here would collide with that PR and with the verbatim `test` job in #212. Known limitation: only the `english` checkpoint was re-run. The committed file's `part_b` covers the English checkpoint only, and the calibration columns #208 tracks are in `part_a`, so the multilingual half is untouched and unmeasured here. Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed. * research: record the multilingual eliminations for the committed-row gap Three more candidates for the unreconciled multilingual row are now ruled out on re-measurement rather than by inspection: the shipped head_max_len (256, read from the checkpoint's own config and recorded in the run), which of the two multilingual copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51), and a checkpoint change since the committed sweep (weights and config are identical by size and hash at every revision in that window). Also corrects the test count in the same file, which the new #208 guard changes. --------- Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`, so the `serve` extra — and therefore the `laya-serve` console script — was not in the image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1` pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml` is empty). `#45` named this as the follow-up: "A natural follow-up now that #31 has landed: a Compose service that runs `laya-serve`, so the container speaks the Jev-compatible `/v1/systemone` out of the box." ``` docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve curl -s localhost:8000/health curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \ --data @examples/docker/request.json ``` End-to-end verified against a real container, not just a config parse: ``` $ docker compose -f compose.yaml -f compose.http.yaml ps repo-laya-serve-1 Up 0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp $ curl -s localhost:18000/health {"status":"ok","loaded":[],"device":"cpu"} $ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json 422 # no weights cached and the container is --network none no header HTTP 401 wrong token HTTP 401 correct token HTTP 422 # auth passed, then the offline model load fails as designed ``` Deliberate choices tested: - **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation. - **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service. - **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment. - **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default. - **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment. - **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back. Known limitations: - `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls. - The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what #45 asked for. It is worth ~15 MB. - I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this. Also included, both from the same gap: - `docs/docker.md`'s "HTTP serving" section said serving was "pending #3 or #31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table. - `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist. Verified: ``` $ python tests/test_packaging.py 34 passed, 0 failed (15 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ docker compose config --quiet OK for all six combinations, including the new -f compose.http.yaml $ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \ python tests/test_docker_entrypoint.py Ran 5 tests ... OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` `tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds the command line without the quoting `subprocess` applied, which is why #212 skips that suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml` run it and where the run above was made. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
) `BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells did not match: | row | column | table said | JSON says | |---|---|---|---| | `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) | | `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) | Both round down, so this is not a formatting difference. The other 16 numeric cells and all twelve argmax agreement counts do match. The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and `README.md:488` both said the fast path *is at least as close to the fp32 reference as the stock bf16 path is*, and on `laya-multilingual` / `noul` it is not: ``` laya-multilingual noul fast_vs_fp32 = 0.045488 stock_vs_fp32 = 0.044593 ``` which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports — within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and name the row that is the exception, rather than asserting something a reader can falsify from the table in front of them. Deliberate choices tested: - **A check, not just two corrections.** `#170`, `#134` and `#208` were all a published number that no longer traced to a run, and `#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `#285`'s proposal that can run in CI today. - **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite: ``` 52 passed, 2 failed FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043' FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045' ``` With them in place, `54 passed, 0 failed`. - **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero. - **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted. - **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence. - **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `#212` added the Windows lane. - Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model. ``` $ python tests/test_doc_tables.py 54 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_load_errors.py 17 passed, 0 failed $ python tests/test_email.py 90 passed, 0 failed ``` Known limitation: this verifies the tables that cite a committed JSON, and it cannot check prose — it would have caught the two cells but not the sentence above them, which is why both were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that is `#285`'s half and needs the hardware. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
…rM#212) Every workflow in this repository runs on `ubuntu-latest` and nowhere else. A test that only fails off Linux is therefore invisible until a user hits it, and that has already happened twice: tests/test_download.py (NandhaKishorM#140, since fixed and merged) and the Windows + Python 3.14 segfault tracked in NandhaKishorM#123. The change: * A new `test-windows` job on `windows-latest`, Python 3.11. * Deliberately a separate job, not an extra matrix axis on `test`. That leaves `test`'s name -- and therefore the `tests (pyX.Y)` required checks -- byte for byte as they are, so branch protection needs no edit. It runs one interpreter because the variable that was untested is the OS, not the Python version. * It runs the same suites as the Linux job. The list is guarded with `-f` so a suite added by another open branch cannot make this job red on a tree that does not have it yet. * torch installs plainly on Windows. The `+cpu` index used on Linux is Linux/macOS only; on Windows the default PyPI wheel is already CPU-only, so the plain install is both correct and what a Windows user gets. * Every `run` step declares `shell: bash`. The default shell on a Windows runner is PowerShell, where `python -c "...; ..."` and a multi-line script are not the same command. GitHub's Windows runners provide Git Bash. `test` is untouched, so Linux behaviour does not move. One suite is skipped, with the reason recorded in the workflow: test_docker_entrypoint It drives `docker/entrypoint.py`, which only ever runs inside a Linux container, through `os.execvp`. On Windows `exec*` rebuilds the command line from argv without re-applying the quoting `subprocess` used to build it, so its `-c "<script>"` argument arrives split at the first space and the child exits with "SyntaxError: invalid syntax" before printing. I confirmed the environment passthrough the suite exists to check is unaffected, and that replacing `execvp` with `subprocess.run` makes it pass -- but that is a change to a container script and belongs in its own PR rather than here. Verified on Windows 11, Python 3.12, torch 2.14.0, transformers 5.17.0, against today's main (which now carries 16 suites). Every suite this job runs passes: test_router, test_criteria, test_download, test_shortlist, test_decision_model, test_packaging, test_lang_guess, test_calibration_persistence, test_context_manager, test_criteria_normalization, test_empty_questions, test_router_memory, test_shortlist_cosine, test_temperature_loading, test_email 14 in the loop plus test_email as its own step; none skipped except the one above. `python -m build`, `twine check` and the import check also pass on Windows. This branch is rebased onto current main. It was opened when NandhaKishorM#211 had not landed, so it previously carried the NandhaKishorM#140 fix to tests/test_download.py; NandhaKishorM#140 merged, so that change is now identical to main and drops out. The diff is `.github/workflows/ci.yml` only. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
NandhaKishorM#208 ask) (NandhaKishorM#222) * research: re-run the 51-language sweep in both temperature regimes (the NandhaKishorM#208 ask) The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so the before/after is visible. `research/results/cpu_51_language_sweep_clamped.json` holds the committed columns, an `--unclamped` re-run and a default re-run side by side, per language, from the same 51 languages and 5,100 cases. committed unclamped re-run clamped re-run 0.2269 0.2269 0.2269 macro accuracy 0.7331 0.7331 0.5709 macro ECE 0.2053 0.2053 0.2053 macro F1 Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the committed macro ECE at 0.7331, so the clamp is the only variable left. Per language the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in 51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and this one 2.14.0. The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower. `choice:11+` is the only bucket the clamp moves and every case here is a 20-option question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at all: a temperature-scaled softmax has the same argmax at every positive temperature. Worth recording because it cuts against the reading that the clamp only distorts the published number: `acc_at_50_coverage` is the one rank-quality column that consumes the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98. The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the committed columns stopped matching the unclamped re-run or if the clamped re-run started matching them. Deliberate choices tested: - `bench_local.py` was not modified. It produced the committed file, and rewriting it to reproduce its own pre-clamp output would make the old and new runs come from different scripts. `laya_eval.py` already has `--unclamped` for this, and the three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are recorded in the new file rather than quietly dropped. - The clamped re-run is committed alongside rather than replacing anything, so `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite. - `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is wired into both lanes, and this suite is currently in neither; adding it here would collide with that PR and with the verbatim `test` job in NandhaKishorM#212. Known limitation: only the `english` checkpoint was re-run. The committed file's `part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks are in `part_a`, so the multilingual half is untouched and unmeasured here. Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed. * research: record the multilingual eliminations for the committed-row gap Three more candidates for the unreconciled multilingual row are now ruled out on re-measurement rather than by inspection: the shipped head_max_len (256, read from the checkpoint's own config and recorded in the run), which of the two multilingual copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51), and a checkpoint change since the committed sweep (weights and config are identical by size and hash at every revision in that window). Also corrects the test count in the same file, which the new NandhaKishorM#208 guard changes. --------- Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
…#234) The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`, so the `serve` extra — and therefore the `laya-serve` console script — was not in the image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1` pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml` is empty). `NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose service that runs `laya-serve`, so the container speaks the Jev-compatible `/v1/systemone` out of the box." ``` docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve curl -s localhost:8000/health curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \ --data @examples/docker/request.json ``` End-to-end verified against a real container, not just a config parse: ``` $ docker compose -f compose.yaml -f compose.http.yaml ps repo-laya-serve-1 Up 0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp $ curl -s localhost:18000/health {"status":"ok","loaded":[],"device":"cpu"} $ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json 422 # no weights cached and the container is --network none no header HTTP 401 wrong token HTTP 401 correct token HTTP 422 # auth passed, then the offline model load fails as designed ``` Deliberate choices tested: - **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation. - **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service. - **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment. - **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default. - **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment. - **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back. Known limitations: - `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls. - The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB. - I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this. Also included, both from the same gap: - `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table. - `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist. Verified: ``` $ python tests/test_packaging.py 34 passed, 0 failed (15 before) $ python tests/test_criteria.py 72 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_email.py 46 passed, 0 failed $ docker compose config --quiet OK for all six combinations, including the new -f compose.http.yaml $ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \ python tests/test_docker_entrypoint.py Ran 5 tests ... OK $ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 All checks passed! ``` `tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml` run it and where the run above was made. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com> Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
…andhaKishorM#299) `BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells did not match: | row | column | table said | JSON says | |---|---|---|---| | `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) | | `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) | Both round down, so this is not a formatting difference. The other 16 numeric cells and all twelve argmax agreement counts do match. The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and `README.md:488` both said the fast path *is at least as close to the fp32 reference as the stock bf16 path is*, and on `laya-multilingual` / `noul` it is not: ``` laya-multilingual noul fast_vs_fp32 = 0.045488 stock_vs_fp32 = 0.044593 ``` which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports — within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and name the row that is the exception, rather than asserting something a reader can falsify from the table in front of them. Deliberate choices tested: - **A check, not just two corrections.** `NandhaKishorM#170`, `NandhaKishorM#134` and `NandhaKishorM#208` were all a published number that no longer traced to a run, and `NandhaKishorM#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `NandhaKishorM#285`'s proposal that can run in CI today. - **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite: ``` 52 passed, 2 failed FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043' FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045' ``` With them in place, `54 passed, 0 failed`. - **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero. - **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted. - **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence. - **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `NandhaKishorM#212` added the Windows lane. - Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model. ``` $ python tests/test_doc_tables.py 54 passed, 0 failed $ python tests/test_packaging.py all packaging tests passed $ python tests/test_criteria.py 128 passed, 0 failed $ python tests/test_router.py all routing tests passed $ python tests/test_load_errors.py 17 passed, 0 failed $ python tests/test_email.py 90 passed, 0 failed ``` Known limitation: this verifies the tables that cite a committed JSON, and it cannot check prose — it would have caught the two cells but not the sentence above them, which is why both were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that is `NandhaKishorM#285`'s half and needs the hardware. Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Every workflow in this repository runs on
ubuntu-latestand nowhere else — all 10 jobs acrossci.yml,release.ymlandsecurity.yml. A test that only fails off Linux is therefore invisible until a user hits it, and that has already happened:tests/test_download.pyfails on Windows on currentmain.I went looking for whether that failure hides a real bug. It does not — but nothing in CI could have told anyone either way, which is the actual problem.
The Windows failure is test-only
I ran every suite CI gates on, on Windows (Python 3.12, torch 2.14.0, transformers 5.17.0). Only one fails:
To check whether the checkpoint is actually loaded correctly on Windows or only the assertion is wrong, I instrumented the test's own fixture and recorded what diverges:
Agentlocates the subfolder, appliesallow_patterns, walks the snapshot and runs inference correctly on Windows; only the set comparison differs, becausefilter_repo_objectsand the mocked listing emit POSIX paths whilestr(Path.relative_to)yields backslashes. An assertion that cannot pass on that platform, versus a regression a user would feel.What this changes
test-windowsjob onwindows-latest, Python 3.11.test. That leavestest's name — and therefore thetests (pyX.Y)required checks — exactly as they are, so branch protection needs no edit. It runs one interpreter because the variable that was untested is the OS, not the Python version.--index-url https://download.pytorch.org/whl/cpuis a Linux/macOS index; on Windows the default PyPI wheel is already CPU-only, so the plain install is both correct and what a Windows user actually gets.runstep declaresshell: bash. The default shell on a Windows runner is PowerShell, wherepython -c "...; ..."and a multi-line script are not the same command. GitHub's Windows runners ship Git Bash.testis untouched, so Linux behaviour does not move at all.if [ -f "tests/$t.py" ], rather than a fixed list, so a suite added by another open branch cannot make this job red on a tree that does not have it yet. On this tree it skipstest_lang_guess, which lands with feat(router): accept a caller-supplied language hint (the #35 ask) #211; once feat(router): accept a caller-supplied language hint (the #35 ask) #211 merges the same loop picks it up with no further change.tests/test_download.pycompares POSIX repository paths.The
build,lintandsecurityjobs stay Ubuntu-only. I left them alone rather than guess at whether packaging or the scanners need to be cross-platform — the test job is what would have caught this class of bug.Verification
On Windows 11, Python 3.12, after the change — every suite present on this tree passes:
I also checked the Linux job is genuinely untouched rather than assuming it: same name, same
ubuntu-latest, same Python matrix, same four run steps, noshell:keys added, and its test list is still the explicit six-line list rather than a loop.Overlap with #140, and what I could not test locally
alpha912's #140 makes the identical three-line fix totests/test_download.pyand was opened first. I included it only because a red Windows job is not worth adding. If #140 lands first I will drop it here and rebase; if this lands first, #140 is subsumed. Flagging it rather than quietly duplicating the work.I could not execute bash locally to dry-run the loop —
bashon this machine is a WSL launcher that cannot run here — so I verified the loop by running its exact body as a list of files and checking each result, and by confirming the guard has something to guard on this tree. The first CI run on this PR is the real test of the job itself, on both platforms at once.