Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,20 @@ node_modules/
.DS_Store
.env
.pypirc
.ruff_cache/
.idea/

# local setup artifacts from setup_laya.sh: ~2.3 GB of checkpoint weights and the download caches
models/
.hf-cache/
.pip-cache/

# outputs written by the research/scripts harnesses; curated runs live in research/results/
research/*_benchmark_results.json

# output of a local notebook run (Kaggle uses /kaggle/working)
notebooks/finetune_run/


# session transcripts (may contain pasted credentials)
session-*.md
Expand Down
262 changes: 262 additions & 0 deletions LOCAL_SETUP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,262 @@
# Laya on this machine — local setup

Laya (the checkout in `laya/`) now runs here on real weights: all three checkpoints, real forward
passes, on CPU and on the AMD GPU through Metal (MPS). This file records what was installed, the
two code changes that were needed, how to re-run everything, and the numbers that were measured.

## Host

| | |
|---|---|
| machine | `MacPro7,1` (2019 Mac Pro), Intel Xeon, 56 logical cores, 256 GB RAM |
| OS | macOS (Darwin 25.6.0), `x86_64` |
| GPU | AMD Radeon Pro Vega II (32 GB) + Radeon PRO W6800X Duo, Metal 3 |
| accelerator APIs | **no CUDA** — CPU, or MPS via Metal |

## What was installed (all inside the repository; everything below is gitignored)

| path | contents |
|---|---|
| `.venv/` | Python 3.11.5 virtualenv — torch 2.2.2, transformers 4.57.6, numpy 1.26.4, safetensors 0.8.0, huggingface_hub 0.36.2 |
| `models/` | the three checkpoints (~2.3 GB): `laya/` (English), plus `laya-multilingual` and `laya-typed-decisions` symlinked into the bundle |
| `.pip-cache/`, `.hf-cache/` | download caches, kept local so nothing is written outside the project |
| `setup_laya.sh` | idempotent setup: venv → pinned deps → editable install → checkpoints → verify |
| `verify/laya_smoke_test.py` | end-to-end check on real weights: routing, all three checkpoints, all three primitives, presets, latency |
| `verify/numerics_check.py` | RoPE base actually used vs. trained, run-to-run determinism, SDPA vs. eager |
| `verify/bench_devices.py` | CPU vs. MPS latency, plus CPU thread scaling |
| `verify/edge_sweep.py` | inference paths the other suites miss: 12 questions in one pass, 20/77/120-option choice, odd input shapes, truncation, router lifecycle, CPU-vs-MPS agreement |
| `verify/checkpoints.py` + `verify/checkpoints.json` | checks the weights against recorded sha256 hashes (a truncated download is otherwise silently wrong) |
| `verify/soak_check.py` | 200 repeated calls for drift and RSS growth, reload/eviction cycles, concurrent calls from several threads. Uses `psutil` if installed and falls back to peak RSS otherwise |
| `examples/` | 41 worked examples as an eight-stage learning path, plus `run_all.sh` and the path doc |

`laya` is installed editable, so `import laya` from anywhere uses this checkout and picks up edits.

## Code changes that were required

The first two are real portability bugs, not local hacks; both are covered by
`tests/test_portability.py` (18 checks, including the CUDA-OOM fallback), which is wired into
CI along with `tests/test_training.py`.

### 1. `laya/common.py` — transformers 5 checkpoints on transformers 4

The checkpoints were saved by transformers 5, which records ModernBERT's RoPE bases as
`rope_parameters = {"full_attention": {...}, "sliding_attention": {...}}`. transformers 4.x does
not read that key, so it silently keeps its own defaults — global `160000`, local `10000`:

* `laya` and `laya-typed-decisions` want `160000 / 10000` → they happened to match.
* `laya-multilingual` (mmBERT) wants `160000 / 160000` → it ran with the **wrong RoPE base** on
every sliding-attention layer, with no error. This is the "quietly wrong numbers" class of bug.

`build_model()` now maps `rope_parameters` onto the `global_rope_theta` / `local_rope_theta`
attributes that 4.x reads (and that the upstream `jhu-clsp/mmBERT-base` config still uses). On
transformers 5 the function is a no-op. Effect on the multilingual checkpoint, same input:

| | before | after |
|---|---|---|
| Hindi "charged twice, refund" → `billing` | 0.8647 | **0.9328** |
| Hindi → `refund_requested` | 0.9897 | 0.9909 |
| English / typed-decisions outputs | — | **bit-identical** (their configs already matched) |

### 2. `laya/agent.py` — MPS crashed every call

`system_one` entered `torch.autocast(device_type=self.device.type, enabled=use_amp)` on every
call. Autocast is only ever *enabled* on CUDA, but torch validates the device type regardless, and
torch has no MPS autocast backend:

```
RuntimeError: User specified an unsupported autocast device_type 'mps'
```

Laya selects MPS automatically when it is available, so on this machine every `predict()` raised.
The forward pass now goes through `_amp_context(device, dtype)`, which returns
`torch.autocast(..., "cuda")` on CUDA and a `nullcontext()` otherwise.

### 3. Smaller fixes found while verifying

* **`collate_items` failed unhelpfully on malformed training data.** A target with more entries
than the item has markers died inside a tensor assignment with *"The expanded size of the tensor
(k) must match the existing size (kmax)"*, which says nothing about the mistake. It now raises a
`ValueError` naming the item and both counts, and `tests/test_training.py` asserts that.
* **`Router.load()` does not evict on a cache hit.** Asking for a checkpoint that is already
resident returns early, so `max_loaded` is enforced when a checkpoint is *loaded* rather than
when one is touched. The behaviour is unchanged — it is now stated in the docstring, because
lowering `max_loaded` after a `preload` does not free anything until the next miss.

## Verify it

```bash
cd /Users/threaded/projects/Laya/laya # repository root
./setup_laya.sh # venv + deps + checkpoints + smoke test (idempotent)

# or individually
.venv/bin/python verify/laya_smoke_test.py --models ./models # device auto → MPS
.venv/bin/python verify/laya_smoke_test.py --models ./models --device cpu
.venv/bin/python verify/numerics_check.py
.venv/bin/python verify/bench_devices.py
.venv/bin/python verify/edge_sweep.py
.venv/bin/python verify/checkpoints.py # weights vs recorded hashes
.venv/bin/python verify/soak_check.py # drift, RSS, concurrency
.venv/bin/python tests/test_local_e2e.py ./models # the repo's own e2e suite
./examples/run_all.sh # every example, one log per run
```

Results on this machine:

| check | result |
|---|---|
| `tests/test_local_e2e.py` (real weights, CPU) | **23 passed, 0 failed** — multilingual billing 8/8, all 11 routing languages correct |
| `tests/test_router.py` | 106 passed, 0 failed |
| `tests/test_criteria.py` | 34 passed, 0 failed |
| `tests/test_portability.py` | 18 passed, 0 failed (includes the CUDA-OOM fallback) |
| `verify/laya_smoke_test.py` | all checks passed on **both** CPU and MPS |
| `verify/numerics_check.py` | RoPE bases match training; answers bit-identical run-to-run and SDPA vs. eager (`0.00e+00` on every reported value) |
| `verify/edge_sweep.py` | all edge-case checks pass (25 here; the MPS checks skip where MPS is unavailable) |
| `verify/checkpoints.py` | 3/3 checkpoints match the recorded sha256; a deliberately corrupted copy is caught |
| `verify/soak_check.py` | 200/200 calls byte-identical with flat RSS; 6 reload cycles leave exactly 1 live agent and 1 live model (0 after `unload()`); 4-thread concurrency matches the single-threaded answer |
| `tests/test_training.py` | 59 passed — proper-scoring-rule reward, TD(λ) targets, ECE, entropy confidence, collation, sequence building |
| `examples/run_all.sh` | **41 passed, 0 failed** (eight stages, 23-49 s each; the whole sweep is ~20 min) |
| CI lint (`ruff`) + `compileall` | pass |

CPU and MPS produce identical answers (same presets, same confidences), so the GPU path is not a
numerical downgrade.

## Measured performance

One `predict()` call answering 4 questions on a short email, median of 10 after warm-up:

| checkpoint | CPU (28 threads) | MPS (Radeon Pro Vega II) |
|---|---|---|
| `laya` (421M) | 453 ms — 113 ms/question | **146 ms — 37 ms/question** |
| `laya-multilingual` (322M) | 192 ms — 48 ms/question | **121 ms — 30 ms/question** |
| `laya-typed-decisions` (421M) | 441 ms — 110 ms/question | **145 ms — 36 ms/question** |

Re-measured on a quiet machine for the final verification pass; individual runs move by a few
percent with machine load (an earlier pass gave 424/195/427 ms on CPU and 141/117/142 ms on MPS).

The GPU is 1.6-3× faster than this 56-core CPU (3× on the two ModernBERT-large checkpoints, 1.6×
on the smaller multilingual one). The README's T4 reference is 32.8 ms/question for
`laya-multilingual`; MPS lands at 29 ms/question, i.e. T4-class latency on this hardware.

CPU thread scaling on `laya` (medians): 4 → 501 ms, 8 → 385 ms, **16 → 366 ms**, 28 → 415 ms,
56 → 1266 ms. Around 16 threads is the sweet spot; the default 28 already over-subscribes and 56
threads is markedly worse, so `OMP_NUM_THREADS=16` (or `torch.set_num_threads(16)`) is worth
setting for CPU-only work.

The ONNX Runtime path is faster than torch eager on the same CPU: one `predict()` answering 3
questions drops from 489 ms to **331 ms (1.48×)** in fp32, bit-identical to eager, and to
**170 ms (2.9×)** with the opt-in per-channel INT8 copy, which moves probabilities by up to
0.16. Installing the `onnx` extra on this stack also wants `numpy<2`, since torch below 2.3
cannot convert tensors against NumPy 2.

Notes: the first MPS call pays ~13 s of Metal kernel compilation, so warm up before timing;
everything runs fp32 (bf16 autocast is CUDA-only in Laya).

## Independent accuracy check against public data

Setting the machine up only proved the plumbing worked. The published tables were re-checked by
running the repo's own harnesses against public datasets — this is measured here, CPU only, not
copied from `BENCHMARKS.md`.

### MASSIVE intent, 20 options — `research/scripts/bench_local.py --langs 10 --per-lang 60 --skip-b`

| | english | multilingual | published, 51 languages (english / multilingual) |
|---|---|---|---|
| macro accuracy | 0.2500 | **0.3950** | 0.2269 / 0.3661 |
| macro ECE *(lower better)* | 0.7100 | **0.4008** | 0.7331 / 0.3869 |
| languages > 3x random | 4 / 10 | **8 / 10** | 23 / 51 / 45 / 51 |

Per language, `en` scores **0.783** — the published English figure exactly — and the English
checkpoint then collapses off English *while staying confident*: Amharic 0.100 at 0.949 mean
confidence, Arabic 0.133 at 0.897, Bengali 0.117 at 0.953, Greek 0.150 at 0.970. The multilingual
checkpoint holds those up (Arabic 0.450, Bengali 0.450, Greek 0.417, German 0.467), which is the
whole reason `Router` exists.

### Applications — `research/scripts/bench_apps.py` with `BENCH_N=80`

| suite | english | multilingual | typed-decisions | README (routed) |
|---|---|---|---|---|
| AG News, 4 labels | 0.963 | **0.975** | 0.963 | 0.950 |
| DAIR Emotion, 6 labels | 0.637 | 0.600 | **0.662** | 0.595 |
| Banking77, 77 labels at once | 0.412 | 0.500 | 0.425 | 0.425 |
| phishing email | 0.975 | **0.988** | 0.925 | — |
| email spam | **0.988** | 0.963 | 0.925 | — |
| guardrails (jailbreak) | 0.838 | **0.875** | 0.863 | 0.698 (injections) |
| moderation (toxicity) | 0.500 | 0.487 | 0.500 | — |
| support triage | 0.550 | 0.550 | 0.537 | — |

Banking77 lands on **0.425**, the number the README quotes for the 77-option case, and the
AG News / emotion figures agree with the published routed results within sampling noise (80 cases
per suite here). The README's honest limits reproduce too: `score` is the weakest primitive and
50+ options in one question is where Laya trails Jev.

Both tables were re-run end to end a second time (same seed, same cached datasets) and
reproduced **identically — every accuracy, F1 and ECE digit, and both macro aggregates**. The
runs are deterministic; only the ms/case figures move with machine load.

### Two fixes the harnesses needed to run here

* they hardcoded the model root to `~/laya_models` — now `LAYA_MODELS` (default unchanged), so
`LAYA_MODELS=$PWD/models` points them at the checkpoints this repo already has;
* their `sys.path` insert pointed at a non-existent `research/laya`; it now points at the
repository root.

### What the soak check found

Repeated checkpoint loads push RSS up and it never comes back down — after 6 reload cycles the
process sat at 3.4 GB against 2.3 GB for a single resident checkpoint, and `unload()` did not
return it. That is macOS `malloc` keeping freed pages, not a leak: counting live objects shows
**exactly 1 `Agent` and 1 `DecisionModel` while cycling and 0 after `unload()`**, which is what
the check now asserts.

The same exercise threw up an API subtlety worth knowing: `Router.load()` returns early for a
checkpoint that is already resident, so it does not run eviction. `max_loaded` is enforced when a
checkpoint is *loaded*, not when one is touched — lower it and then load something new, or call
`unload()`, if residency has to drop immediately. That is now stated in `Router.load`'s docstring.

## Using it from your own code

The checkpoints are already on disk, so point the router at the local roots and nothing touches
the network:

```python
import laya
from laya import Router

root = "/Users/threaded/projects/Laya/laya/models"
router = Router(models={"english": root + "/laya",
"multilingual": root + "/laya-multilingual",
"typed-decisions": root + "/laya-typed-decisions"},
preload=True) # device=None → MPS here; pass device="cpu" to force CPU

res = router.predict({"body": "发票4411被重复扣款,请今天退款。"}, laya.triage_questions())
print(res["routing"]["model"], res["answers"]["intent"]["choice"])
```

`laya.load("<path>", device="mps")` works the same way for a single checkpoint. Leaving `device`
unset picks MPS, then CPU.

## Why these versions

* **torch 2.2.2** is the newest PyTorch with a macOS `x86_64` wheel. Both PyPI and
`download.pytorch.org` stop at 2.2.2 (2.3.0 onward are Apple-Silicon-only), which is the whole
constraint chain here.
* **transformers 4.57.6** is the last 4.x line. transformers 5.x requires torch ≥ 2.4, so it
cannot be used with the only torch this CPU can install — hence the RoPE mapping above. 4.57.x
is also the line that reads both ModernBERT and mmBERT checkpoints.
* **numpy < 2** because torch 2.2.2 was compiled against NumPy 1.x and fails to initialise under
NumPy 2 (`Failed to initialize NumPy: _ARRAY_API not found`).
* **huggingface_hub < 1.0** is pinned by transformers 4.x.

If a newer torch is ever wanted, conda-forge still builds `osx-64` PyTorch (CPU/MKL) past 2.13,
which would allow transformers 5.x and remove the need for the RoPE mapping. That is an
alternative, not what is installed and verified here.

## Caveats

* No CUDA: fp16/bf16 mixed precision is unavailable, CPU and MPS both run fp32.
* MPS is a torch 2.2-era backend on Intel AMD hardware — it works and matches CPU numerics here,
but it is not an NVIDIA-class path. `--device cpu` is the conservative choice.
* `models/` is ~2.3 GB and `.venv/` ~1 GB; both live in the repository and are gitignored, so
they can be deleted and rebuilt with `./setup_laya.sh` without touching the working tree.
* No TensorFlow is installed on purpose (`USE_TF=0`): a TF install alongside torch can deadlock
model construction, which the repo's tests already guard against.
Loading
Loading