Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@ jobs:
python tests/test_batch.py
python tests/test_predict_long.py
python tests/test_attention_dynamic_shapes.py
python tests/test_state_budget.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_structured.py
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ jobs:
python tests/test_criteria.py
python tests/test_batch.py
python tests/test_predict_long.py
python tests/test_state_budget.py
python tests/test_hooks.py
python tests/test_hooks_api.py
python tests/test_onnx_lang_parity.py
Expand Down
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1118,9 +1118,21 @@ failure; it does not establish calibrated confidence.
The positive control passed on both checkpoints. These are narrow cancellation examples, not
evidence that every negated state fails. Validate the exact checkpoint and wording you serve;
using semantic keys alone does not avoid this failure.
* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget (`max_len - head_max_len`):
* `laya` (English) defaults to 512 context (`head_max_len = 192`, ~320 tokens for state).
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE).
* **High-cardinality choice questions and token budgets:** Sequences split into an option prompt budget (`head_max_len`) and the remaining document/state budget. **`head_max_len` is a cap, not the head length.** The renderer fills the option prompt budget only as far as the question and its option descriptions need, and `build_sequence` then sizes the state from the head it *actually built* (`room = max_len - head_len - 1`). The real room therefore depends on the question, not on the cap:
* `laya` (English) defaults to 512 context (`head_max_len = 192`). The shipped department question below renders a **48-token** head, leaving **463 tokens for state**, not 320.
* `laya-multilingual` and `laya-typed-decisions` default to 1,024 context (`head_max_len = 256`). The shipped question renders a **45-token** head, leaving **978 tokens for state**, not 768 (mmBERT-base encoder supports up to 8,192 with RoPE).
`max_len - head_max_len` is **not a safe figure in either direction**, so size a state to `max_len - head_len - 1` for the question you are actually asking:
* For a question that does not fill the cap it **understates** the room, as above — conservative, but it discards state the model would have read.
* For a high-cardinality question it **overstates** it. `per = max(4, (head_max_len - 16) // k)` floors at 4 tokens per option and `head_ids` keeps a floor of 8, so past roughly `head_max_len / 4` options the head grows *beyond* the cap. A state sized to the cap-based figure is then **truncated**, not merely conservative:

| | `k` options | head | real room | cap-based figure |
|---|---|---|---|---|
| `laya`, `max_len=512` | 77 | 319 | **192** | 320 |
| `laya`, `max_len=512` | 100 | 411 | **100** | 320 |
| `laya-multilingual`, `max_len=1024` | 100 | 411 | **612** | 768 |

The crossover is at `k ≈ head_max_len / 4` and the gap widens from there; `MAX_CHOICE_OPTIONS = 100` in `laya/serve.py` puts the overstating regime within reach over HTTP.

At default settings, a 77-option question like Banking77 allocates only `(256 - 16) // 77` ≈ 3–4 tokens per label, which causes accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question:
1. Raise `agent.cfg["head_max_len"] = 512` and `agent.cfg["max_len"] = 1024` (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. Both are also per-request: `predict(state, questions, head_max_len=512, max_len=1024)` widens one question without changing the agent for everyone else, and every LangChain node takes the same two arguments ([LangChain guide](docs/langchain.md)). `laya --questions` takes the same two budgets as `--max-len` / `--head-max-len`.
2. Or shortlist with embeddings and run one forward pass on the top `k` labels (`predict_shortlist`, example below). `predict` and `system_one` still score every criterion they are given.
Expand Down
237 changes: 237 additions & 0 deletions tests/test_state_budget.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,237 @@
"""Regression: the state budget is sized from the head actually built, not from `head_max_len`.

`head_max_len` is a cap. `build_sequence` fills the option prompt budget only as far as the
question and its option descriptions need, then sizes the state as `max_len - head_len - 1`
from the head it actually built.

`max_len - head_max_len` is wrong in *both* directions, which is what the README now says:

* a question that does not fill the cap -> the cap-based figure **understates** the room
(conservative, but it discards state the model would have read);
* a high-cardinality question -> it **overstates** it. `per = max(4, (head_max_len - 16) // k)`
floors at 4 tokens per option and `head_ids` keeps a floor of 8, so past roughly
`head_max_len / 4` options the head grows beyond the cap, and a state sized to the cap-based
figure is **truncated** rather than merely conservative.

This suite pins the relationship, and — when the checkpoints are present — reads the figures
back out of `README.md` and checks them against a live measurement, so the renderer, the
documentation and this file cannot drift apart silently.
"""
import os
import re
import sys

sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from laya.common import build_sequence # noqa: E402

PASS, FAIL, SKIPPED = [], [], []

# The same 4-option question the README's token-budget section walks through.
Q = {"t": "choice", "ins": "Which department should handle this request?",
"crit": {"billing": "invoices, payments, refunds",
"technical": "bugs, outages, system errors",
"sales": "pricing, new contracts, plan upgrades",
"other": "everything else"}}


class _Skip(Exception):
pass


def skip(reason):
"""A skip that is visible in both runners.

Under pytest this raises `pytest.skip`, so `-rs` reports it. As a plain script it raises
`_Skip`, which the runner prints and counts -- a bare `return` passes vacuously and no
runner flag reveals it.
"""
print(" SKIP: " + reason)
SKIPPED.append(reason)
if __name__ != "__main__":
import pytest
pytest.skip(reason)
raise _Skip(reason)


def check(name, got, want):
if got == want:
PASS.append(name)
else:
FAIL.append("%s: got %r, want %r" % (name, got, want))


class _FakeTok:
"""Deterministic stand-in: one id per word, so head length is predictable."""
cls_token_id, sep_token_id, mask_token_id, pad_token_id = 0, 1, 4, 2
mask_token = "[MASK]"

def __call__(self, text, add_special_tokens=False, truncation=False, max_length=None):
ids = [10 + (len(w) % 90) for w in text.split() if w]
if truncation and max_length:
ids = ids[:max_length]
return {"input_ids": ids}

def id_of(self, word):
return self(" " + word)["input_ids"][0]


def _head_len(tok, q, head_max_len):
"""Length of the built prompt the state is appended to, excluding the trailing [SEP]."""
seq, _ = build_sequence(tok, "", q, max_len=10 ** 6, head_max_len=head_max_len)
return len(seq) - 1


def _kept_state(tok, state, q, max_len, head_max_len):
head = _head_len(tok, q, head_max_len)
seq, _ = build_sequence(tok, state, q, max_len=max_len, head_max_len=head_max_len)
return seq[head:-1]


def _many_options(k, desc="a short description of this option"):
return {"t": "choice", "ins": "Which intent is this?",
"crit": {"intent_%d" % i: desc for i in range(k)}}


def _models_dir():
return os.environ.get("LAYA_LAB_MODELS", "/home/parshu/projects/contri/laya-lab/models")


def _real_tok(name):
"""The checkpoint's tokenizer, or None. Callers decide whether absence is a skip."""
path = os.path.join(_models_dir(), name, "tokenizer")
if not os.path.isdir(path):
return None
from transformers import AutoTokenizer
return AutoTokenizer.from_pretrained(path)


# --------------------------------------------------------------------------- invariant checks

def test_state_budget_is_sized_from_the_actual_head():
"""A state longer than the room is cut to exactly `max_len - head_len - 1`."""
tok = _FakeTok()
max_len, head_max_len = 512, 192
head = _head_len(tok, Q, head_max_len)
assert head < head_max_len, "fixture assumes a question that does not fill the cap"

room = max_len - head - 1
state = " ".join("w%d" % i for i in range(room + 200))
kept = _kept_state(tok, state, Q, max_len, head_max_len)
check("kept state length is max_len - head_len - 1", len(kept), room)


def test_dropping_the_minus_one_is_caught():
"""The `-1` is load-bearing: one token more than the room must lose its tail."""
tok = _FakeTok()
max_len, head_max_len = 512, 192
head = _head_len(tok, Q, head_max_len)
marker = "TAILMARKER"
exact = max_len - head - 1 # the true room

state = " ".join("w%d" % i for i in range(exact - 1)) + " " + marker
kept = _kept_state(tok, state, Q, max_len, head_max_len)
check("a state of exactly the room is kept whole", len(kept), exact)
assert tok.id_of(marker) in kept, "the tail marker must survive an exactly-sized state"

over = " ".join("w%d" % i for i in range(exact)) + " " + marker
kept_over = _kept_state(tok, over, Q, max_len, head_max_len)
assert tok.id_of(marker) not in kept_over, (
"a state one token over the room must lose its tail, so the room is max_len-head_len-1 "
"and not max_len-head_len")


def test_cap_based_figure_understates_a_short_question():
"""A state of exactly `max_len - head_max_len` tokens survives for a short question."""
tok = _FakeTok()
max_len, head_max_len = 512, 192
documented = max_len - head_max_len
head = _head_len(tok, Q, head_max_len)
assert max_len - head - 1 > documented, "fixture assumes the cap-based figure understates"

marker = "UNIQUETAILMARKER"
state = " ".join("w%d" % i for i in range(documented - 3)) + " " + marker
kept = _kept_state(tok, state, Q, max_len, head_max_len)
assert tok.id_of(marker) in kept, (
"a %d-token state must survive a %d budget for this question" % (documented, max_len))


def test_cap_based_figure_overstates_a_high_cardinality_question():
"""Past ~`head_max_len / 4` options the head exceeds the cap, so the cap-based figure lies.

This is the direction that matters: a caller sizing a state to `max_len - head_max_len`
is truncated, not merely conservative.
"""
tok = _FakeTok()
max_len, head_max_len = 512, 192
head = _head_len(tok, _many_options(100), head_max_len)
assert head > head_max_len, (
"a 100-option question must build a head beyond the %d cap, got %d" % (head_max_len, head))
check("cap-based figure overstates the room at k=100",
max_len - head_max_len > max_len - head - 1, True)


# --------------------------------------------------------------------- documentation agreement

def _readme_quotes():
"""The figures README.md states, parsed out of the token-budget bullet."""
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
with open(os.path.join(root, "README.md"), encoding="utf-8") as fh:
text = fh.read()
heads = {int(m) for m in re.findall(r"\*\*(\d+)-token\*\* head", text)}
rooms = {int(m) for m in re.findall(r"\*\*(\d+) tokens for state\*\*", text)}
return heads, rooms


def test_readme_quotes_the_measured_figures():
"""README's head/room figures equal a live measurement, for each shipped checkpoint.

This is what makes the suite stop the README and the renderer desynchronising; without it
the file only guards the renderer.
"""
heads, rooms = _readme_quotes()
assert heads and rooms, "could not parse the head/room figures out of README.md"

checked = 0
for name, max_len, head_max_len in (("english", 512, 192), ("multilingual", 1024, 256)):
try:
tok = _real_tok(name)
except Exception as e: # transformers absent or unloadable
skip("%s: tokenizer unavailable (%s)" % (name, e))
continue
if tok is None:
skip("%s: no tokenizer under %s" % (name, _models_dir()))
continue
head = _head_len(tok, Q, head_max_len)
room = max_len - head - 1
assert head in heads, (
"README does not quote a %d-token head for %s (quotes %s)" % (head, name, sorted(heads)))
assert room in rooms, (
"README does not quote a %d-token state room for %s (quotes %s)"
% (room, name, sorted(rooms)))
checked += 1
if not checked:
skip("no checkpoints under %s" % _models_dir())


if __name__ == "__main__":
for fn in (test_state_budget_is_sized_from_the_actual_head,
test_dropping_the_minus_one_is_caught,
test_cap_based_figure_understates_a_short_question,
test_cap_based_figure_overstates_a_high_cardinality_question,
test_readme_quotes_the_measured_figures):
try:
fn()
except AssertionError as e:
FAIL.append("%s: %s" % (fn.__name__, e))
except _Skip:
pass
else:
PASS.append(fn.__name__)

print("\n%d passed, %d failed, %d skipped" % (len(PASS), len(FAIL), len(SKIPPED)))
for f in FAIL:
print(" FAIL " + f)
for s in SKIPPED:
print(" SKIP " + s)
sys.exit(1 if FAIL else 0)