Skip to content

feat(ci): run the test suite on Windows as well as Linux - #212

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:ci/windows-test-job
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
PerryLink:ci/windows-test-job

Conversation

@PerryLink

Copy link
Copy Markdown
Contributor

Every workflow in this repository runs on ubuntu-latest and nowhere else — all 10 jobs across ci.yml, release.yml and security.yml. A test that only fails off Linux is therefore invisible until a user hits it, and that has already happened: tests/test_download.py fails on Windows on current main.

I went looking for whether that failure hides a real bug. It does not — but nothing in CI could have told anyone either way, which is the actual problem.

The Windows failure is test-only

I ran every suite CI gates on, on Windows (Python 3.12, torch 2.14.0, transformers 5.17.0). Only one fails:

FAIL tests/test_download.py       4 assertions, 5 failures
OK   tests/test_router.py         all routing tests passed
OK   tests/test_criteria.py       34 passed, 0 failed
OK   tests/test_shortlist.py      81 passed, 0 failed
OK   tests/test_decision_model.py all decision model tests passed
OK   tests/test_packaging.py      all packaging tests passed
OK   tests/test_email.py          11 passed, 0 failed

To check whether the checkpoint is actually loaded correctly on Windows or only the assertion is wrong, I instrumented the test's own fixture and recorded what diverges:

subfolder=None               inference matches direct load: True
subfolder='multilingual'     inference matches direct load: True
subfolder='variants/english' inference matches direct load: True

  want - got: 'encoder\\config.json', 'tokenizer\\tokenizer.json', ...
  got - want: 'encoder/config.json',   'tokenizer/tokenizer.json', ...

Agent locates the subfolder, applies allow_patterns, walks the snapshot and runs inference correctly on Windows; only the set comparison differs, because filter_repo_objects and the mocked listing emit POSIX paths while str(Path.relative_to) yields backslashes. An assertion that cannot pass on that platform, versus a regression a user would feel.

What this changes

  • A new test-windows job on windows-latest, Python 3.11.
  • Deliberately a separate job, not an extra matrix axis on test. That leaves test's name — and therefore the tests (pyX.Y) required checks — exactly as they are, so branch protection needs no edit. It runs one interpreter because the variable that was untested is the OS, not the Python version.
  • torch installs plainly on Windows. --index-url https://download.pytorch.org/whl/cpu is a Linux/macOS index; on Windows the default PyPI wheel is already CPU-only, so the plain install is both correct and what a Windows user actually gets.
  • Every run step declares shell: bash. The default shell on a Windows runner is PowerShell, where python -c "...; ..." and a multi-line script are not the same command. GitHub's Windows runners ship Git Bash. test is untouched, so Linux behaviour does not move at all.
  • The suite list is a guarded loop, if [ -f "tests/$t.py" ], rather than a fixed list, so a suite added by another open branch cannot make this job red on a tree that does not have it yet. On this tree it skips test_lang_guess, which lands with feat(router): accept a caller-supplied language hint (the #35 ask) #211; once feat(router): accept a caller-supplied language hint (the #35 ask) #211 merges the same loop picks it up with no further change.
  • tests/test_download.py compares POSIX repository paths.

The build, lint and security jobs stay Ubuntu-only. I left them alone rather than guess at whether packaging or the scanners need to be cross-platform — the test job is what would have caught this class of bug.

Verification

On Windows 11, Python 3.12, after the change — every suite present on this tree passes:

tests/test_download.py           4 tests, OK
tests/test_router.py             all routing tests passed
tests/test_criteria.py           34 passed, 0 failed
tests/test_shortlist.py          81 passed, 0 failed
tests/test_decision_model.py     all decision model tests passed
tests/test_packaging.py          all packaging tests passed
tests/test_email.py              11 passed, 0 failed

python -m build                  laya-0.3.6-py3-none-any.whl
twine check                      PASSED
import laya                      0.3.6, 27 exports

I also checked the Linux job is genuinely untouched rather than assuming it: same name, same ubuntu-latest, same Python matrix, same four run steps, no shell: keys added, and its test list is still the explicit six-line list rather than a loop.

Overlap with #140, and what I could not test locally

alpha912's #140 makes the identical three-line fix to tests/test_download.py and was opened first. I included it only because a red Windows job is not worth adding. If #140 lands first I will drop it here and rebase; if this lands first, #140 is subsumed. Flagging it rather than quietly duplicating the work.

I could not execute bash locally to dry-run the loop — bash on this machine is a WSL launcher that cannot run here — so I verified the loop by running its exact body as a list of files and checking each result, and by confirming the guard has something to guard on this tree. The first CI run on this PR is the real test of the job itself, on both platforms at once.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you. Windows is where #123 and #140 both lived unseen, and a separate job that leaves the required tests (py3.x) names alone is a thoughtful design. #140 landed today, so test_download.py should now pass there. Today's main moved a lot (35 PRs merged), so this now conflicts. Could you rebase onto the latest main? Then I'll approve the workflow run and merge once the Windows job is green.

Every workflow in this repository runs on `ubuntu-latest` and nowhere else. A test
that only fails off Linux is therefore invisible until a user hits it, and that has
already happened twice: tests/test_download.py (NandhaKishorM#140, since fixed and merged) and
the Windows + Python 3.14 segfault tracked in NandhaKishorM#123.

The change:

* A new `test-windows` job on `windows-latest`, Python 3.11.
* Deliberately a separate job, not an extra matrix axis on `test`. That leaves
  `test`'s name -- and therefore the `tests (pyX.Y)` required checks -- byte for byte
  as they are, so branch protection needs no edit. It runs one interpreter because
  the variable that was untested is the OS, not the Python version.
* It runs the same suites as the Linux job. The list is guarded with `-f` so a suite
  added by another open branch cannot make this job red on a tree that does not have
  it yet.
* torch installs plainly on Windows. The `+cpu` index used on Linux is Linux/macOS
  only; on Windows the default PyPI wheel is already CPU-only, so the plain install
  is both correct and what a Windows user gets.
* Every `run` step declares `shell: bash`. The default shell on a Windows runner is
  PowerShell, where `python -c "...; ..."` and a multi-line script are not the same
  command. GitHub's Windows runners provide Git Bash. `test` is untouched, so Linux
  behaviour does not move.

One suite is skipped, with the reason recorded in the workflow:

  test_docker_entrypoint

It drives `docker/entrypoint.py`, which only ever runs inside a Linux container,
through `os.execvp`. On Windows `exec*` rebuilds the command line from argv without
re-applying the quoting `subprocess` used to build it, so its `-c "<script>"`
argument arrives split at the first space and the child exits with
"SyntaxError: invalid syntax" before printing. I confirmed the environment
passthrough the suite exists to check is unaffected, and that replacing `execvp`
with `subprocess.run` makes it pass -- but that is a change to a container script
and belongs in its own PR rather than here.

Verified on Windows 11, Python 3.12, torch 2.14.0, transformers 5.17.0, against
today's main (which now carries 16 suites). Every suite this job runs passes:

  test_router, test_criteria, test_download, test_shortlist, test_decision_model,
  test_packaging, test_lang_guess, test_calibration_persistence, test_context_manager,
  test_criteria_normalization, test_empty_questions, test_router_memory,
  test_shortlist_cosine, test_temperature_loading, test_email

14 in the loop plus test_email as its own step; none skipped except the one above.
`python -m build`, `twine check` and the import check also pass on Windows.

This branch is rebased onto current main. It was opened when NandhaKishorM#211 had not landed, so
it previously carried the NandhaKishorM#140 fix to tests/test_download.py; NandhaKishorM#140 merged, so that
change is now identical to main and drops out. The diff is `.github/workflows/ci.yml`
only.
@PerryLink

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (2f73a2b). The diff is .github/workflows/ci.yml only — tests/test_download.py dropped out entirely, since #140 landed the same fix and my copy was by then identical to main.

Two things I found while rebasing, because today's main moved a long way:

The Linux job now runs 16 suites, not 7. The Windows job now runs all of them, minus one. The list is guarded with -f so a suite added by another open branch cannot make this job red on a tree that does not have it yet.

One suite cannot pass on Windows, and it is worth knowing which. test_docker_entrypoint fails there, and not for a Windows-y reason:

File "<string>", line 1
  import
        ^
SyntaxError: invalid syntax

It drives docker/entrypoint.py, which ends in os.execvp(sys.argv[1], sys.argv[1:]). On Windows exec* rebuilds the command line from argv without re-applying the quoting subprocess used to build it, so the suite's -c "<script>" argument arrives split at the first space. I checked that the thing the suite actually exists for — the HF_TOKEN passthrough — is fine, and that subprocess.run in place of execvp makes it pass. So the entrypoint is not portable, and the suite is correct to fail.

I have skipped it in the Windows job with the reason in the workflow, rather than change entrypoint.py here. That is a container script and the fix deserves its own PR; say the word and I will open one. Flagging it rather than quietly excluding it, because a Windows job that silently skips a real portability failure is worse than no job.

Verified on Windows 11, Python 3.12, torch 2.14.0, transformers 5.17.0, against today's main:

14 of 14 in the loop pass, plus test_email as its own step
test_docker_entrypoint skipped, reason recorded
python -m build / twine check / import check all pass

I also checked that test's name and body are byte-identical to main and that feishu-benchmark is untouched, so nothing of today's work is disturbed and the tests (pyX.Y) required checks still resolve.

@PerryLink

Copy link
Copy Markdown
Contributor Author

Rebase confirmed against the current main tip, so nothing is waiting on me here.

origin/main tip           010bace   (release: 0.3.7)
PR #212 head              2f73a2b
merge-base                010bace   <- identical, no drift

All 14 checks pass on that head, including the job this PR exists to add:

tests (windows)   pass   2m35s
tests (py3.10)    pass   1m17s      lint      pass
tests (py3.11)    pass   0m55s      package   pass
tests (py3.12)    pass   1m03s      smoke     pass
tests (py3.13)    pass   1m31s      CodeQL    pass
Chinese benchmark audit  pass       secret scan / CVEs / deserialization  pass

mergeable is MERGEABLE and the state is BLOCKED on the review, so it is ready for your approval whenever you get to it.

Two things I noticed while waiting that you may want, both opt-in and neither in this diff:

The Windows job runs Python 3.11 only, so it does not cover #123. That issue is specific to Windows + 3.14, and 3.11-on-Windows would not have reproduced the initialize_weights segfault even before #195. actions/python-versions publishes python-3.14.7-win32-x64.zip, so setup-python can provision it on windows-latest. If you would like the Windows matrix to be ["3.11", "3.14"], I can add it as a second entry on top of this PR — that would make the job cover the platform-and-version combination that actually produced the report, and give #195 a regression gate on Windows.

docker/entrypoint.py is genuinely not Windows-portable, which is why test_docker_entrypoint is the one suite the job skips, with the reason recorded in the workflow. os.execvp at the end of the file rebuilds the command line from argv on Windows without re-applying the quoting subprocess used, so a -c "<script>" argument arrives split at the first space. I verified the HF_TOKEN passthrough the suite exists for is fine, and that subprocess.run in its place makes it pass. Say the word and I will open that as its own PR so the Windows job can run all 16 suites instead of 15.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, and for rebasing twice while main kept moving. Windows is where #123 and #140 both lived unseen, and a separate job that leaves the required tests (py3.x) names alone was a thoughtful design. Merging.

@NandhaKishorM
NandhaKishorM merged commit 4931d79 into NandhaKishorM:main Sep 23, 2026
14 checks passed
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 23, 2026
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`,
so the `serve` extra — and therefore the `laya-serve` console script — was not in the
image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1`
pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview
branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml`
is empty).

`NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose
service that runs `laya-serve`, so the container speaks the Jev-compatible
`/v1/systemone` out of the box."

```
docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json
```

End-to-end verified against a real container, not just a config parse:

```
$ docker compose -f compose.yaml -f compose.http.yaml ps
repo-laya-serve-1  Up  0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp

$ curl -s localhost:18000/health
{"status":"ok","loaded":[],"device":"cpu"}

$ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json
422                       # no weights cached and the container is --network none

no header      HTTP 401
wrong token    HTTP 401
correct token  HTTP 422   # auth passed, then the offline model load fails as designed
```

Deliberate choices tested:

- **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation.
- **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service.
- **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment.
- **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default.
- **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment.
- **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back.

Known limitations:

- `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls.
- The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB.
- I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this.

Also included, both from the same gap:

- `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table.
- `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist.

Verified:

```
$ python tests/test_packaging.py            34 passed, 0 failed    (15 before)
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ docker compose config --quiet             OK for all six combinations,
                                            including the new -f compose.http.yaml
$ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \
    python tests/test_docker_entrypoint.py  Ran 5 tests ... OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

`tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds
the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that
suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml`
run it and where the run above was made.
PerryLink added a commit to PerryLink/laya that referenced this pull request Sep 23, 2026
NandhaKishorM#208 ask)

The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from
before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer
reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so
the before/after is visible.

`research/results/cpu_51_language_sweep_clamped.json` holds the committed columns,
an `--unclamped` re-run and a default re-run side by side, per language, from the
same 51 languages and 5,100 cases.

    committed      unclamped re-run   clamped re-run
    0.2269         0.2269             0.2269     macro accuracy
    0.7331         0.7331             0.5709     macro ECE
    0.2053         0.2053             0.2053     macro F1

Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the
committed macro ECE at 0.7331, so the clamp is the only variable left. Per language
the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in
51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do
so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and
this one 2.14.0.

The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower.
`choice:11+` is the only bucket the clamp moves and every case here is a 20-option
question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at
all: a temperature-scaled softmax has the same argmax at every positive temperature.

Worth recording because it cuts against the reading that the clamp only distorts the
published number: `acc_at_50_coverage` is the one rank-quality column that consumes
the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98.

The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the
committed columns stopped matching the unclamped re-run or if the clamped re-run
started matching them.

Deliberate choices tested:

- `bench_local.py` was not modified. It produced the committed file, and rewriting it
  to reproduce its own pre-clamp output would make the old and new runs come from
  different scripts. `laya_eval.py` already has `--unclamped` for this, and the
  three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are
  recorded in the new file rather than quietly dropped.
- The clamped re-run is committed alongside rather than replacing anything, so
  `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite.
- `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is
  wired into both lanes, and this suite is currently in neither; adding it here
  would collide with that PR and with the verbatim `test` job in NandhaKishorM#212.

Known limitation: only the `english` checkpoint was re-run. The committed file's
`part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks
are in `part_a`, so the multilingual half is untouched and unmeasured here.

Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.
NandhaKishorM pushed a commit that referenced this pull request Sep 23, 2026
#208 ask) (#222)

* research: re-run the 51-language sweep in both temperature regimes (the #208 ask)

The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from
before #42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer
reproduce. This adds the re-run #208 asked for, with the old file left in place so
the before/after is visible.

`research/results/cpu_51_language_sweep_clamped.json` holds the committed columns,
an `--unclamped` re-run and a default re-run side by side, per language, from the
same 51 languages and 5,100 cases.

    committed      unclamped re-run   clamped re-run
    0.2269         0.2269             0.2269     macro accuracy
    0.7331         0.7331             0.5709     macro ECE
    0.2053         0.2053             0.2053     macro F1

Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the
committed macro ECE at 0.7331, so the clamp is the only variable left. Per language
the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in
51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do
so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and
this one 2.14.0.

The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower.
`choice:11+` is the only bucket the clamp moves and every case here is a 20-option
question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at
all: a temperature-scaled softmax has the same argmax at every positive temperature.

Worth recording because it cuts against the reading that the clamp only distorts the
published number: `acc_at_50_coverage` is the one rank-quality column that consumes
the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98.

The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the
committed columns stopped matching the unclamped re-run or if the clamped re-run
started matching them.

Deliberate choices tested:

- `bench_local.py` was not modified. It produced the committed file, and rewriting it
  to reproduce its own pre-clamp output would make the old and new runs come from
  different scripts. `laya_eval.py` already has `--unclamped` for this, and the
  three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are
  recorded in the new file rather than quietly dropped.
- The clamped re-run is committed alongside rather than replacing anything, so
  `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite.
- `ci.yml` is untouched. #184 is adding the check that every model-free suite is
  wired into both lanes, and this suite is currently in neither; adding it here
  would collide with that PR and with the verbatim `test` job in #212.

Known limitation: only the `english` checkpoint was re-run. The committed file's
`part_b` covers the English checkpoint only, and the calibration columns #208 tracks
are in `part_a`, so the multilingual half is untouched and unmeasured here.

Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.

* research: record the multilingual eliminations for the committed-row gap

Three more candidates for the unreconciled multilingual row are now ruled out on
re-measurement rather than by inspection: the shipped head_max_len (256, read from
the checkpoint's own config and recorded in the run), which of the two multilingual
copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51),
and a checkpoint change since the committed sweep (weights and config are identical by
size and hash at every revision in that window).

Also corrects the test count in the same file, which the new #208 guard changes.

---------

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
NandhaKishorM pushed a commit that referenced this pull request Sep 23, 2026
The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`,
so the `serve` extra — and therefore the `laya-serve` console script — was not in the
image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1`
pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview
branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml`
is empty).

`#45` named this as the follow-up: "A natural follow-up now that #31 has landed: a Compose
service that runs `laya-serve`, so the container speaks the Jev-compatible
`/v1/systemone` out of the box."

```
docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json
```

End-to-end verified against a real container, not just a config parse:

```
$ docker compose -f compose.yaml -f compose.http.yaml ps
repo-laya-serve-1  Up  0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp

$ curl -s localhost:18000/health
{"status":"ok","loaded":[],"device":"cpu"}

$ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json
422                       # no weights cached and the container is --network none

no header      HTTP 401
wrong token    HTTP 401
correct token  HTTP 422   # auth passed, then the offline model load fails as designed
```

Deliberate choices tested:

- **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation.
- **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service.
- **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment.
- **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default.
- **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment.
- **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back.

Known limitations:

- `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls.
- The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what #45 asked for. It is worth ~15 MB.
- I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this.

Also included, both from the same gap:

- `docs/docker.md`'s "HTTP serving" section said serving was "pending #3 or #31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table.
- `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist.

Verified:

```
$ python tests/test_packaging.py            34 passed, 0 failed    (15 before)
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ docker compose config --quiet             OK for all six combinations,
                                            including the new -f compose.http.yaml
$ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \
    python tests/test_docker_entrypoint.py  Ran 5 tests ... OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

`tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds
the command line without the quoting `subprocess` applied, which is why #212 skips that
suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml`
run it and where the run above was made.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
NandhaKishorM pushed a commit that referenced this pull request Sep 24, 2026
)

`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells
did not match:

| row | column | table said | JSON says |
|---|---|---|---|
| `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) |
| `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) |

Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.

The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and
`README.md:488` both said the fast path *is at least as close to the fp32 reference as the
stock bf16 path is*, and on `laya-multilingual` / `noul` it is not:

```
laya-multilingual noul   fast_vs_fp32 = 0.045488   stock_vs_fp32 = 0.044593
```

which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five
rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports
— within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.

Deliberate choices tested:

- **A check, not just two corrections.** `#170`, `#134` and `#208` were all a published number that no longer traced to a run, and `#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `#285`'s proposal that can run in CI today.
- **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite:
  ```
  52 passed, 2 failed
    FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
    FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
  ```
  With them in place, `54 passed, 0 failed`.
- **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
- **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted.
- **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence.
- **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `#212` added the Windows lane.
- Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model.

```
$ python tests/test_doc_tables.py               54 passed, 0 failed
$ python tests/test_packaging.py                all packaging tests passed
$ python tests/test_criteria.py                 128 passed, 0 failed
$ python tests/test_router.py                   all routing tests passed
$ python tests/test_load_errors.py              17 passed, 0 failed
$ python tests/test_email.py                    90 passed, 0 failed
```

Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is `#285`'s half and needs the hardware.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…rM#212)

Every workflow in this repository runs on `ubuntu-latest` and nowhere else. A test
that only fails off Linux is therefore invisible until a user hits it, and that has
already happened twice: tests/test_download.py (NandhaKishorM#140, since fixed and merged) and
the Windows + Python 3.14 segfault tracked in NandhaKishorM#123.

The change:

* A new `test-windows` job on `windows-latest`, Python 3.11.
* Deliberately a separate job, not an extra matrix axis on `test`. That leaves
  `test`'s name -- and therefore the `tests (pyX.Y)` required checks -- byte for byte
  as they are, so branch protection needs no edit. It runs one interpreter because
  the variable that was untested is the OS, not the Python version.
* It runs the same suites as the Linux job. The list is guarded with `-f` so a suite
  added by another open branch cannot make this job red on a tree that does not have
  it yet.
* torch installs plainly on Windows. The `+cpu` index used on Linux is Linux/macOS
  only; on Windows the default PyPI wheel is already CPU-only, so the plain install
  is both correct and what a Windows user gets.
* Every `run` step declares `shell: bash`. The default shell on a Windows runner is
  PowerShell, where `python -c "...; ..."` and a multi-line script are not the same
  command. GitHub's Windows runners provide Git Bash. `test` is untouched, so Linux
  behaviour does not move.

One suite is skipped, with the reason recorded in the workflow:

  test_docker_entrypoint

It drives `docker/entrypoint.py`, which only ever runs inside a Linux container,
through `os.execvp`. On Windows `exec*` rebuilds the command line from argv without
re-applying the quoting `subprocess` used to build it, so its `-c "<script>"`
argument arrives split at the first space and the child exits with
"SyntaxError: invalid syntax" before printing. I confirmed the environment
passthrough the suite exists to check is unaffected, and that replacing `execvp`
with `subprocess.run` makes it pass -- but that is a change to a container script
and belongs in its own PR rather than here.

Verified on Windows 11, Python 3.12, torch 2.14.0, transformers 5.17.0, against
today's main (which now carries 16 suites). Every suite this job runs passes:

  test_router, test_criteria, test_download, test_shortlist, test_decision_model,
  test_packaging, test_lang_guess, test_calibration_persistence, test_context_manager,
  test_criteria_normalization, test_empty_questions, test_router_memory,
  test_shortlist_cosine, test_temperature_loading, test_email

14 in the loop plus test_email as its own step; none skipped except the one above.
`python -m build`, `twine check` and the import check also pass on Windows.

This branch is rebased onto current main. It was opened when NandhaKishorM#211 had not landed, so
it previously carried the NandhaKishorM#140 fix to tests/test_download.py; NandhaKishorM#140 merged, so that
change is now identical to main and drops out. The diff is `.github/workflows/ci.yml`
only.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
NandhaKishorM#208 ask) (NandhaKishorM#222)

* research: re-run the 51-language sweep in both temperature regimes (the NandhaKishorM#208 ask)

The committed `cpu_51_language_sweep.json` stores `ece` and `mean_confidence` from
before NandhaKishorM#42 clamped temperatures to `[0.5, 5]`, so its calibration columns no longer
reproduce. This adds the re-run NandhaKishorM#208 asked for, with the old file left in place so
the before/after is visible.

`research/results/cpu_51_language_sweep_clamped.json` holds the committed columns,
an `--unclamped` re-run and a default re-run side by side, per language, from the
same 51 languages and 5,100 cases.

    committed      unclamped re-run   clamped re-run
    0.2269         0.2269             0.2269     macro accuracy
    0.7331         0.7331             0.5709     macro ECE
    0.2053         0.2053             0.2053     macro F1

Macro accuracy reproduces at 0.2269 and the raw-temperature re-run reproduces the
committed macro ECE at 0.7331, so the clamp is the only variable left. Per language
the unclamped run agrees with the committed file on `accuracy` and `macro_f1` in
51/51, on `ece` in 48/51 and on `mean_confidence` in 49/51; the three that differ do
so by 0.0001, the last stored digit, because the committed run used torch 2.8.0 and
this one 2.14.0.

The clamped run differs on `ece` and `mean_confidence` in 51/51, every one lower.
`choice:11+` is the only bucket the clamp moves and every case here is a 20-option
question, so it applies to all 5,100. `accuracy` and `macro_f1` cannot move with T at
all: a temperature-scaled softmax has the same argmax at every positive temperature.

Worth recording because it cuts against the reading that the clamp only distorts the
published number: `acc_at_50_coverage` is the one rank-quality column that consumes
the confidence values, and it goes up, macro 0.3004 -> 0.3020 and `en` 0.94 -> 0.98.

The new file is guarded by `research/eval/test_laya_eval.py`, which fails if the
committed columns stopped matching the unclamped re-run or if the clamped re-run
started matching them.

Deliberate choices tested:

- `bench_local.py` was not modified. It produced the committed file, and rewriting it
  to reproduce its own pre-clamp output would make the old and new runs come from
  different scripts. `laya_eval.py` already has `--unclamped` for this, and the
  three columns it does not compute (`brier`, `nll`, `seconds`, `dropped`) are
  recorded in the new file rather than quietly dropped.
- The clamped re-run is committed alongside rather than replacing anything, so
  `cpu_51_language_sweep.json` keeps the accuracy columns other tables cite.
- `ci.yml` is untouched. NandhaKishorM#184 is adding the check that every model-free suite is
  wired into both lanes, and this suite is currently in neither; adding it here
  would collide with that PR and with the verbatim `test` job in NandhaKishorM#212.

Known limitation: only the `english` checkpoint was re-run. The committed file's
`part_b` covers the English checkpoint only, and the calibration columns NandhaKishorM#208 tracks
are in `part_a`, so the multilingual half is untouched and unmeasured here.

Verified: `python research/eval/test_laya_eval.py` -> 62 passed, 0 failed.

* research: record the multilingual eliminations for the committed-row gap

Three more candidates for the unreconciled multilingual row are now ruled out on
re-measurement rather than by inspection: the shipped head_max_len (256, read from
the checkpoint's own config and recorded in the run), which of the two multilingual
copies was measured (bundled subfolder and standalone repo both give 0.4008 / 6-of-51),
and a checkpoint change since the committed sweep (weights and config are identical by
size and hash at every revision in that window).

Also corrects the test count in the same file, which the new NandhaKishorM#208 guard changes.

---------

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…#234)

The image could not serve HTTP at all. `Dockerfile:16` was `pip install . && pip check`,
so the `serve` extra — and therefore the `laya-serve` console script — was not in the
image, and `compose.yaml` only ran the one-shot SDK quickstart. `compose.cuda.yaml:1`
pointed at the gap: it described `compose.http.yaml` as living "on the HTTP preview
branch", and that file has never existed in any commit (`git log --all -- compose.http.yaml`
is empty).

`NandhaKishorM#45` named this as the follow-up: "A natural follow-up now that NandhaKishorM#31 has landed: a Compose
service that runs `laya-serve`, so the container speaks the Jev-compatible
`/v1/systemone` out of the box."

```
docker compose -f compose.yaml -f compose.http.yaml up --build laya-serve
curl -s localhost:8000/health
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' \
  --data @examples/docker/request.json
```

End-to-end verified against a real container, not just a config parse:

```
$ docker compose -f compose.yaml -f compose.http.yaml ps
repo-laya-serve-1  Up  0.0.0.0:18000->18000/tcp, [::]:18000->18000/tcp

$ curl -s localhost:18000/health
{"status":"ok","loaded":[],"device":"cpu"}

$ curl -s -o /dev/null -w '%{http_code}' -X POST localhost:18000/v1/systemone -d @examples/docker/request.json
422                       # no weights cached and the container is --network none

no header      HTTP 401
wrong token    HTTP 401
correct token  HTTP 422   # auth passed, then the offline model load fails as designed
```

Deliberate choices tested:

- **The CUDA override repeats its build args and device reservation for `laya-serve`.** A Compose override replaces a service's `build` block whole rather than merging it, so adding `-f compose.cuda.yaml` without repeating them leaves `laya-serve` on CPU while `laya` gets the GPU. Verified both services receive `TORCH_INDEX: cu128`, `LAYA_DEVICE: cuda` and the same `driver: nvidia` reservation.
- **`compose.http.yaml` never touches the `laya` service.** It only adds `laya-serve`, so `docker compose run --rm laya` and `docs/docker.md`'s existing examples behave exactly as before. `tests/test_packaging.py` now asserts that, so a later edit cannot quietly publish a port on the quickstart service.
- **`LAYA_PRELOAD` defaults to `0` here**, against the package default of `1`. Preloading makes the first container boot download all three checkpoints, which is a bad first impression for a quickstart; the docs say to set it to `1` for a long-running deployment.
- **One port variable, not two.** `LAYA_PORT` sets both the published host port and the port `laya-serve` binds (it already reads `LAYA_PORT`), so they cannot drift. The test asserts the two sides of the mapping are the same variable with the same default.
- **`LAYA_API_KEY_FILE` rather than only `LAYA_API_KEY`.** `docker/entrypoint.py` already had the `*_FILE` mechanism for `HF_TOKEN`, so `LAYA_API_KEY` joins `SECRET_NAMES`; the file is read once at startup and the `_FILE` variable is removed before the server execs. This is the first secret the server has, and a bearer token belongs in a file or a Compose secret, not in the environment.
- **`restart: unless-stopped` for the server, and none for the quickstart.** The quickstart is a one-shot command; a server that exits on a transient error should come back.

Known limitations:

- `/health` stays unauthenticated, so it reports `{"status":"ok","loaded":[]}` to anyone. That is the shipped behaviour of `laya.serve` and this PR does not change it; the health probe is what an orchestrator polls.
- The image grows by fastapi and uvicorn for every user of the quickstart, including those who never serve HTTP. The alternative was a second Dockerfile or a build arg, and one image that can do both is what NandhaKishorM#45 asked for. It is worth ~15 MB.
- I could not run `docker build --pull` to completion here: the daemon resolves `auth.docker.io` over IPv6 and that path times out on this machine, so the build used the cached base image. The `FROM` line and everything after it built and ran, and `docker.yml`'s `--pull` build is the check that covers this.

Also included, both from the same gap:

- `docs/docker.md`'s "HTTP serving" section said serving was "pending NandhaKishorM#3 or NandhaKishorM#31" with the server command still to come. It now documents the service, the seven `LAYA_*` server variables, the port override and the token-from-file pattern, and the two new variables are in the main configuration table.
- `tests/test_packaging.py` gains 19 checks over the Compose layout: the service and its command, the port mapping, that the quickstart service is untouched, that the Dockerfile installs `.[serve]` and still runs `pip check`, that the CUDA override covers both services, that no file references a missing Compose file, and that all four Compose files exist.

Verified:

```
$ python tests/test_packaging.py            34 passed, 0 failed    (15 before)
$ python tests/test_criteria.py             72 passed, 0 failed
$ python tests/test_router.py               all routing tests passed
$ python tests/test_email.py                46 passed, 0 failed
$ docker compose config --quiet             OK for all six combinations,
                                            including the new -f compose.http.yaml
$ docker run --rm -v "$PWD:/workspace:ro" -w /workspace repo-laya-serve \
    python tests/test_docker_entrypoint.py  Ran 5 tests ... OK
$ python -m ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
```

`tests/test_docker_entrypoint.py` fails 1 of 5 on Windows because `os.execvp` rebuilds
the command line without the quoting `subprocess` applied, which is why NandhaKishorM#212 skips that
suite on its Windows job; it passes on Linux, which is where `ci.yml` and `docker.yml`
run it and where the run above was made.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Co-authored-by: NandhaKishorM <nandakishor@convaiinnovations.com>
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…andhaKishorM#299)

`BENCHMARKS.md:274` says the fast-path comparison "can be re-checked without a GPU" because
every per-option probability is in `benchmarks/results/parity_*.json`. It can, and two cells
did not match:

| row | column | table said | JSON says |
|---|---|---|---|
| `laya` / `noul` | `max \|p_fast - p_fp32\|` | `0.044` | **0.043** (`d_fast_fp32 = 0.0434565544128418`) |
| `laya-multilingual` / `noul` | `max \|p_fast - p_fp32\|` | `0.046` | **0.045** (`d_fast_fp32 = 0.04548829793930054`) |

Both round down, so this is not a formatting difference. The other 16 numeric cells and all
twelve argmax agreement counts do match.

The sentence under that table was falsified by its own numbers. `BENCHMARKS.md:286` and
`README.md:488` both said the fast path *is at least as close to the fp32 reference as the
stock bf16 path is*, and on `laya-multilingual` / `noul` it is not:

```
laya-multilingual noul   fast_vs_fp32 = 0.045488   stock_vs_fp32 = 0.044593
```

which is the `0.045` vs `0.045` printed two lines above it. The claim held on the other five
rows (e.g. `laya`/`noul` 0.0435 vs 0.0578). Both places now state the bound the data supports
— within `0.046` of fp32, within `0.076` of stock, argmax agreement at least `47/48` — and
name the row that is the exception, rather than asserting something a reader can falsify from
the table in front of them.

Deliberate choices tested:

- **A check, not just two corrections.** `NandhaKishorM#170`, `NandhaKishorM#134` and `NandhaKishorM#208` were all a published number that no longer traced to a run, and `NandhaKishorM#285` proposes re-benchmarking everything each release. I did not want to add to that checklist, because re-measuring would not have caught either cell: the run is not stale, the transcription is. `tests/test_doc_tables.py` reads the published table out of `BENCHMARKS.md` and compares it cell by cell to the committed JSON, so this half is deterministic, needs no weights and no GPU, and is the part of `NandhaKishorM#285`'s proposal that can run in CI today.
- **It fails loudly on drift instead of skipping.** Removing the two corrections and running the suite:
  ```
  52 passed, 2 failed
    FAIL parity table/laya/noul d_fast_fp32: got '0.044', want '0.043'
    FAIL parity table/laya-multilingual/noul d_fast_fp32: got '0.046', want '0.045'
  ```
  With them in place, `54 passed, 0 failed`.
- **It reports what it cannot check rather than implying coverage.** Four published tables have no committed artifact behind them (the fine-tuned checkpoint's row, the `103-332 q/s` throughput range, the fast-path latency table whose `fast_*.json` carry no `eval` key, and the reload latencies whose results file is absent). The suite prints those with the reason, and asserts that the list is non-empty, so the gap cannot quietly become zero.
- **Rounding matches how the tables were written.** `round(0.009501, 3)` is `0.010` under banker's rounding but the table's own `0.010` from `0.009501` shows the ordinary convention was used, so the comparison quantizes with `ROUND_HALF_UP` rather than `round()`. Using `round()` would report that row as drifted.
- **The old claim is pinned as false, not just deleted.** The suite asserts that the set of rows where the stock path is closer to fp32 is exactly `laya-multilingual`/`noul`, so a future edit cannot quietly restore the stronger sentence.
- **Line endings.** The suite opens markdown with `newline=""` and normalises CRLF, so it passes on a Windows checkout as well as Linux — which matters because `NandhaKishorM#212` added the Windows lane.
- Wired into both lanes beside `test_packaging.py`, which is the other suite that reads repository files rather than running the model.

```
$ python tests/test_doc_tables.py               54 passed, 0 failed
$ python tests/test_packaging.py                all packaging tests passed
$ python tests/test_criteria.py                 128 passed, 0 failed
$ python tests/test_router.py                   all routing tests passed
$ python tests/test_load_errors.py              17 passed, 0 failed
$ python tests/test_email.py                    90 passed, 0 failed
```

Known limitation: this verifies the tables that cite a committed JSON, and it cannot check
prose — it would have caught the two cells but not the sentence above them, which is why both
were fixed by hand. It also cannot tell whether a committed JSON is still reproducible; that
is `NandhaKishorM#285`'s half and needs the hardware.

Co-authored-by: PerryLink <255665900+PerryLink@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants