Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 18 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Three checkpoints, and a `Router` that picks between them per request:

* **About 10x faster loading.** Checkpoints are built without the throwaway random weight initialisation, so `laya.load()` drops from about 22 s to about 2 s on CPU with bit-identical answers. This also skips the pass that crashed on Windows with Python 3.14 (#123). A second `Agent` for the same checkpoint reuses its parsed tokenizer and loads in about 0.5 s.
* **`import laya` no longer loads torch.** Routing, language detection and e-mail cleaning work in lightweight processes; torch loads on first use of a model.
* **Batch scoring.** `agent.predict_batch(states, questions)` scores many states in shared forward passes, with answers identical to calling `predict` one state at a time. See [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass).
* **Batch scoring.** `agent.predict_batch(states, questions)` scores many states in shared forward passes and returns results in input order. See [Batch Mode](#batch-mode-score-many-states-in-one-forward-pass).
* **Faster paths, all opt-in.** `laya.load(..., fast=True)` uses a TileLang GPU fast path that matches the stock bf16 forward within rounding (see [GPU Fast Path](#gpu-fast-path-tilelang)). `Agent(compile=True)` enables `torch.compile`, and `laya.onnx_agent.ONNXAgent` runs an exported model on ONNX Runtime.
* **Run it your way, locally.** A self-hosted Jev-compatible HTTP server (`pip install "laya[serve]"`, then `laya-serve`, see [Self-Hosting](#self-hosting-http-server-jev-compatible)), a `laya` [command](#command-line) for quick local tests, an optional [MCP server](#mcp-server-optional) (`pip install "laya[mcp]"`), [LangChain and LangGraph](#langchain-and-langgraph-integration) routing, guardrails, triage and evaluation (`pip install "laya[langchain]"`), a Docker quickstart under `docs/docker.md`, and `laya-ts/`, a TypeScript package for Node and the browser that gives the same answers as the Python package.
* **Better routing.** Plain-ASCII Spanish, Italian, Portuguese and French, Brazilian Portuguese support text, CJK text containing Latin brand names, romanized Bangla and Azerbaijani now reach the multilingual checkpoint. Scripts the router has no range for no longer fall through to English, and URLs, e-mail addresses and dotted names no longer count as words. Checked on 20,000 English texts: at most 5 English sentences move, all quoting long native-script names.
Expand Down Expand Up @@ -457,18 +457,29 @@ a log slice — `predict_batch` packs them into shared forward passes:
states = [{"body": t} for t in ticket_texts] # a list of states

results = agent.predict_batch(states, questions) # one forward pass for the whole list
# results[i] is exactly what agent.predict(states[i], questions) would return
# results[i] corresponds to states[i], with the same output shape as predict

# Bound peak memory when the list (or the texts) are large — chunk into passes of N:
results = agent.predict_batch(states, questions, batch_size=64)

# Reduce padding when input lengths vary; results still follow the original state order:
results = agent.predict_batch(states, questions, batch_size=64, sort_by_length=True)
```

Results are aligned with `states` by index and identical in shape to `predict`. Decisions match the
one-at-a-time path exactly (numbers are bit-identical on CPU; on GPU they can differ in the 4th
decimal because fp16 autocast reorders reductions across padding widths). Batching is a **GPU
`sort_by_length=True` groups states by their longest encoded question row, after truncation.
It looks ahead at most eight batches and reuses the encoded rows for sorting. This uses more temporary
CPU memory for tokenized inputs, and takes effect only when `1 < batch_size < len(states)`.
Benchmark it on your workload and backend: uniform lengths offer little benefit, and changed
batch shapes can cause small floating-point differences, including near decision thresholds.
Hooks still see states and final results in input order. The option is available on `Agent`.

Results are aligned with `states` by index and identical in shape to `predict`. Changing batch
shapes can introduce floating-point differences on CPU and GPU; check decision thresholds on
your workload, particularly with mixed precision. Batching is a **GPU
throughput win** — on an RTX 5060 Ti, per-decision latency drops from ~10 ms one-by-one to ~1 ms
batched (measured ~9–10×). On CPU the model is already compute-bound, so batching does not speed it
up; use it there only for API convenience.
batched (measured ~9–10×). On CPU, increasing batch size alone may not speed up inference;
length grouping can help by reducing the padded work in a mixed-length workload. See the
[CPU measurements and reproduction commands](research/README.md#length-batching).

---

Expand Down
2 changes: 1 addition & 1 deletion docs/hooks/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ load(..., hooks=None, on_predict_start=None, on_predict_end=None,

agent.predict_batch(states, questions, batch_size=None,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
max_len=None, head_max_len=None)
max_len=None, head_max_len=None, sort_by_length=False)

agent.system_one(state, questions,
hooks=None, on_predict_start=None, on_predict_end=None, hooks_raise=None,
Expand Down
58 changes: 38 additions & 20 deletions laya/agent.py
Original file line number Diff line number Diff line change
Expand Up @@ -603,7 +603,8 @@ def predict_batch(self, states: List[Union[str, dict, list]], questions: Dict[st
on_predict_start=None, on_predict_end=None,
hooks_raise: Optional[bool] = None,
max_len: Optional[int] = None,
head_max_len: Optional[int] = None) -> List[Dict[str, Any]]:
head_max_len: Optional[int] = None,
sort_by_length: bool = False) -> List[Dict[str, Any]]:
"""Evaluate the same questions over many states, packing them into shared forward passes.

This is the throughput path. `system_one`/`predict` handle one state per forward pass; on a
Expand All @@ -624,6 +625,11 @@ def predict_batch(self, states: List[Union[str, dict, list]], questions: Dict[st
hooks_raise: Override the Agent's `hooks_raise` for this call.
max_len, head_max_len: Override the agent config for this call. A start hook may also
set `ctx.max_len` / `ctx.head_max_len` to shape the token budget.
sort_by_length: Group similarly sized encoded states within windows of eight batches
to reduce padding. Requires an explicit `batch_size` greater than one and
smaller than the number of states; otherwise it has no effect. Results retain
input order. This buffers up to eight batches of tokenized states instead of
one. Changing batch shapes can slightly change floating-point predictions.

Returns:
A list of per-state result dicts, each identical in shape to `system_one`'s output and
Expand Down Expand Up @@ -669,25 +675,37 @@ def predict_batch(self, states: List[Union[str, dict, list]], questions: Dict[st
overrides["head_max_len"] = ctx.head_max_len

results: List[Dict[str, Any]] = []
for start in range(0, len(states), chunk):
part = states[start:start + chunk]
per_state_items = [self._encode_state(st, ids, internal, **overrides) for st in part]

b = collate_items(per_state_items, self.tok.pad_token_id)
logits, act = self._forward(b)
att = b["attention_mask"]

row = 0
for items in per_state_items:
nrows = len(items)
n_tokens = int(att[row:row + nrows].sum())
answers = self._decode_answers(logits, act, items, ids, internal, row)
results.append({
"model": "laya-rl-agent",
"answers": answers,
"usage": {"input_tokens": n_tokens, "output_tokens": 0},
})
row += nrows
reorder = sort_by_length and 1 < chunk < len(states)
# Bound tokenized lookahead independently of the full input size. Use
# actual post-truncation lengths, with each state's questions kept together.
window = chunk * 8 if reorder else chunk
for start in range(0, len(states), window):
part = states[start:start + window]
encoded = [self._encode_state(st, ids, internal, **overrides) for st in part]
order = list(range(len(encoded)))
if reorder:
order.sort(key=lambda i: max(len(item["ids"]) for item in encoded[i]))
window_results = [None] * len(encoded)
for offset in range(0, len(order), chunk):
indices = order[offset:offset + chunk]
per_state_items = [encoded[i] for i in indices]

b = collate_items(per_state_items, self.tok.pad_token_id)
logits, act = self._forward(b)
att = b["attention_mask"]

row = 0
for index, items in zip(indices, per_state_items):
nrows = len(items)
n_tokens = int(att[row:row + nrows].sum())
answers = self._decode_answers(logits, act, items, ids, internal, row)
window_results[index] = {
"model": "laya-rl-agent",
"answers": answers,
"usage": {"input_tokens": n_tokens, "output_tokens": 0},
}
row += nrows
results.extend(window_results)
ctx.results = results
except BaseException as exc:
ctx.error = exc
Expand Down
40 changes: 40 additions & 0 deletions research/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,51 @@ the numbers quoted in the main README — nothing here is imported by the `laya`
| `scripts/bench_local.py` | CPU sweep: MASSIVE intent across **all 51 languages**, plus typed-decisions on all three checkpoints |
| `scripts/bench_apps.py` | the six application workflows (support triage, email + phishing, guardrails, RAG relevance, moderation, model routing) plus the datasets where public Jev numbers exist |
| `scripts/bench_latency.py` | inference speed including what routing costs: detection overhead, hot path, cold-swap, mixed-language throughput at several `max_loaded` settings |
| `scripts/bench_length_batching.py` | compare upstream contiguous batches with optional length sorting on synthetic tickets, including output consistency and optional fresh-process memory profiles (`psutil` required for memory mode) |
| `scripts/make_plots.py` | renders `assets/laya_benchmark.png` from the result JSONs |

Everything runs with `USE_TF=0` — `transformers` probes for TensorFlow at import, and when TF is
installed its abseil runtime can deadlock model construction on macOS/Python 3.9.

## Length batching

[Local CPU measurements](results/length_batching_cpu_20260924.json) compare the original
`predict_batch` method at `1e28ac20c0896b1c37a744cd11f740eb98f8b178` with `sort_by_length=True`.
On 10,000 synthetic English support tickets, the multilingual checkpoint took **1775.71 s
before and 825.09 s after (2.15x)**, including tokenization, collation, inference and decoding.
There were zero choice/action decision changes, zero usage mismatches, and a maximum returned
numeric difference of **0.0001**. This is a throughput workload, not a labeled accuracy test.

The machine was a Ryzen 9 5950X with 64 GiB RAM, Windows 11, CPU float32 and 16 Torch threads.
An unrelated adaptation experiment was running on the GPU. Small tests alternate execution
order over repeated rounds; the 1,000 and 10,000 input tests each have one full timing pair.
The similar-length control showed no regression. GPU, MPS, compiled and TileLang performance
were not measured. Results depend on input lengths, checkpoint, batch size and hardware.

To reproduce, use a fresh Python 3.12 environment and the pinned model snapshot:

```sh
python -m pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cpu
python -m pip install transformers==4.57.1 huggingface_hub==0.36.2 numpy==2.3.5 psutil==7.2.2
python -c "from huggingface_hub import snapshot_download; print(snapshot_download('convaiinnovations/laya', revision='aa8c91ca088ec597df95a0d1c76b3063cb2ae5e8', allow_patterns=['multilingual/*']) + '/multilingual')"
```

Replace `MODEL_DIR` below with the printed checkpoint directory. Start with `--count 16
--rounds 2 --questions 3`, then increase the count. The full CPU comparison takes tens of minutes:

```sh
python research/scripts/bench_length_batching.py MODEL_DIR --count 10000 --batch-size 8 --threads 16 --rounds 1 --output length-10000.json
```

Memory measurements run each mode in a separate process after model warm-up. They sample
process RSS every 10 ms during 64-input inference, excluding model-load transients; short
allocation peaks can be missed. They do not measure the peak memory of the 10,000-input run:

```sh
python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode original --output memory-original.json
python research/scripts/bench_length_batching.py MODEL_DIR --count 64 --batch-size 8 --threads 16 --memory-mode grouped --output memory-grouped.json
```

## Results

| file | contents |
Expand Down
Loading
Loading