Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
174 changes: 174 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,164 @@ triage = agent.predict({"message": "My payment failed twice"}, laya.triage_quest
| **`score`** | Expected level on ordinal rubric, distribution, confidence | Frustration level, ticket urgency, harm severity |
| **`noul`** | Calibrated probability P(true) from 0.0 to 1.0 | Phishing detection, spam filtering, jailbreak detection, churn risk |

### Structured criteria

Criteria may be JSON objects as well as strings, for rubric-style options:

```python
questions = {
"category": {
"type": "choice",
"instructions": "Which team should handle this?",
"criteria": {
"billing": {"what": "invoices, refunds", "not_for": ["outages"], "examples": ["double charge"]},
"technical": {"what": "bugs and outages", "examples": ["500 on login"]},
},
},
}
```

This works for `choice` options, `score` levels and the `noul` `true`/`false` boundaries.

---

## Typed Answers

Answers are typed objects, and remain dict-compatible so existing code keeps working:

```python
result = agent.predict(state, questions)

result.answers["category"].choice # attribute access
result.answers["category"].confidence
result.answers["needs_reply"].is_true # P(true) >= 0.5
result.request_id # for correlating logs

result["answers"]["category"]["choice"] # the pre-0.2 style still works
```

If the state is longer than the context window, the dropped tokens are reported rather
than silently discarded:

```python
for t in result.truncated:
print(f"{t.question_id}: dropped {t.dropped_tokens} of {t.state_tokens} tokens")
```

Pass `truncate_left=True` to keep the *end* of a long input instead of the beginning, and
`max_len=` to widen the window (see the note under Context Window below).

## Error Handling

All errors derive from `LayaError`, and keep their original builtin bases for compatibility:

```python
from laya import LayaError, OptionsTooLongError, QuestionError

try:
result = agent.predict(state, questions)
except OptionsTooLongError: # too many options for one head -- see patterns.two_stage_choice
...
except QuestionError: # malformed question definition
...
except LayaError: # anything else from laya
...
```

## Async

```python
from laya import AsyncAgent, RetryPolicy

agent = await AsyncAgent.load("convaiinnovations/laya", retry=RetryPolicy(max_attempts=3))
result = await agent.system_one(state, questions)
results = await agent.batch([state_a, state_b, state_c], questions, max_concurrency=8)
```

Inference is local compute, so `AsyncAgent` runs the forward pass in a thread executor to
keep the event loop responsive rather than to overlap network I/O.

## Patterns

Compositions over `predict`, in `laya.patterns`:

| Pattern | What it does |
|---|---|
| `confidence_gate` | Split answers into automatic vs. escalated by calibrated confidence |
| `route` | Dispatch to a handler based on a `choice` answer |
| `composite_score` | Combine several questions into one weighted 0–1 score |
| `self_consistency` | Ask a question several ways; flag disagreement for review |
| `rerank` | Score and sort candidates best-first |
| `cascade` | Run staged questions, stopping early when a gate fails |
| `fan_out` | Evaluate one question set over many states |
| `two_stage_choice` | Coarse-then-fine selection for more options than fit one head (pass `group_of` — see below) |
| `taxonomy_choice` | Walk a nested taxonomy to a leaf |
| `chunk_state` / `map_reduce` | Cover documents longer than the context window |
| `select_tool` | Pick which tool/function should handle a request (with a decline option) |
| `check_citation` / `check_citations` | Verify a claim is actually supported by its source |

```python
from laya import patterns

gated = patterns.confidence_gate(agent, state, questions, threshold=0.85)
for qid, answer in gated["escalate"].items():
send_to_human_review(qid, answer)
```

### Choosing among many options

Measured on an L40S over a 96-case routing benchmark ([full report](docs/gpu_validation_report.md)):

| Options | Recommended approach | Accuracy |
|---|---|---|
| under ~100 | a single `predict` call | 0.750 |
| ~100+ | `two_stage_choice` with `group_of` | 0.646 |
| ~100+ | `two_stage_choice` without `group_of` | 0.229 |

A single call is both simpler and more accurate while the options still fit, so reach for
two-stage only past the ceiling — and give it meaningful groups when you do:

```python
out = patterns.two_stage_choice(
agent, state, "Which category?", options,
group_of=lambda name, criterion: name.split("_")[0], # billing_*, outage_*, ...
)
```

Without `group_of`, options are chunked arbitrarily and the first stage has no semantic signal
to choose between groups — which costs most of the accuracy in the table above.

### Checking citations

`check_citation` asks whether a source supports a claim, contradicts it, or is simply
off-topic:

```python
out = patterns.check_citation(agent, claim="The service had an outage Tuesday.", source=doc)
out["verdict"] # "supported" | "contradicted" | "unsupported"
out["supported"] # 0.937
```

On the published checkpoint the **support** signal is reliable (0.94–0.99 on genuinely
supported pairs, <0.03 otherwise), but the **contradiction** signal is not — the model scores
an unrelated source at 0.996 contradiction, indistinguishable from a real refutation. Treat a
`"supported"` verdict as dependable and anything else as "needs a human"; `check_citations`
returns exactly that list via `out["unsupported"]`.

## Context Window

The published checkpoint is trained at `max_len=512`. You can raise it:

```python
agent = laya.Agent("convaiinnovations/laya", max_len=2048)
```

Sequence building handles the longer window correctly, and ModernBERT's encoder supports
8k natively — but **accuracy and calibration beyond the trained length are unverified**,
and laya logs a warning when you exceed it. For long documents on the current checkpoint,
prefer `patterns.map_reduce`, which chunks the input and combines per-chunk answers. Raising
the trained window properly requires retraining at the longer length.

---

## Benchmark: Laya vs. TypeSafe Jev
Expand All @@ -146,6 +304,22 @@ triage = agent.predict({"message": "My payment failed twice"}, laya.triage_quest
<img src="assets/benchmark_comparison.png" alt="Laya vs TypeSafe Jev Benchmark" width="900" />
</div>

> **How to read this table.** These are *not* head-to-head measurements. Jev's column
> reproduces its published numbers; Laya's column is measured locally on our own evaluation
> set. The two differ in what they measure:
>
> - **Latency is not like-for-like.** Jev's 70–500 ms is end-to-end network latency to a
> hosted API (including TLS, queueing and egress). Laya's ~38 ms is local GPU compute on
> an already-loaded model, excluding any network. A self-hosted deployment behind a
> network hop pays that hop too.
> - **Accuracy is measured on different data.** Jev's agreement figures come from its own
> published workflows; Laya's are from its in-task evaluation set. The rows are aligned by
> theme, not by a shared benchmark, so the differences are not a controlled comparison.
>
> Treat the table as a description of each system's published characteristics, not as a
> claim that one outperforms the other under identical conditions. A true head-to-head would
> require running both against the same inputs on the same hardware.

| Metric / Dimension | TypeSafe Jev (Published) | Laya (Fine-Tuned Checkpoint) | Analysis / Advantage |
|---|---|---|---|
| **P50 Latency (1 Question)** | ~400 ms avg (70 to 500 ms, 150 ms best) | **38.4 ms** (p95: 42.1 ms) | **Laya is ~10.4x faster on avg (4x faster than Jev best-case)** |
Expand Down
164 changes: 164 additions & 0 deletions docs/feature_parity_requirements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# Feature Parity Requirements: Laya vs. TypeSafe Jev

Gap analysis against [TypeSafe Jev](https://typesafe.ai/blog/introducing-system-one-models-and-jev) (launched 2026-09-15), the hosted System One decision model Laya reimplements.

> **Status (2026-09-18, v0.2.0).** P0 #1, P1 and P2 are implemented and tested. P0 #2 and #3
> were validated on an L40S GPU — see the [GPU validation report](gpu_validation_report.md).
> P0 #3 is resolved; P0 #2 is proven feasible and now needs real training data rather than
> compute. See [Implementation status](#implementation-status) for the full picture.
> Three measurements in the original analysis were corrected against real hardware — the option
> ceiling, the notebook `max_len` range, and the value of two-stage choice — and are annotated inline.

**The concept is already covered.** Both expose the same three primitives (`choice`/`score`/`noul`), the same `{state, questions} -> {answers}` shape, and the same RLCD calibration story. Nearly every gap below is surface area and limits, not architecture.

Ordered by importance for parity.

---

## P0 — Blocking

### 1. Structured JSON criteria (one path crashes)

Jev accepts JSON objects in `instructions`, choice option values, score levels, and noul `true`/`false` boundaries ([docs](https://docs.typesafe.ai/primitives/advanced.md)). This enables rubric-style options (`{"what": ..., "not_for": ..., "examples": [...]}`) and taxonomy walking via nested child options.

`render_options` (`laya/common.py:22`) assumes strings throughout:

- **choice / score** — dict criteria stringify as Python reprs (`billing: {'what': 'invoices'}`, single-quoted, not JSON)
- **noul** — dict criteria raise `TypeError: can only concatenate str (not "dict") to str` at `laya/common.py:30`

`instructions` is fine — `_to_internal` already json-dumps non-strings (`laya/agent.py:180`). Only `criteria` is affected.

**Fix:** `json.dumps` non-string criteria values in all three branches. ~1 hour; it is currently a hard crash.

### 2. Context window: 512 vs ~32,000 tokens

`max_len` defaults to 512 (`laya/agent.py:206`); notebooks train at 512 (Colab) and 1024 (Kaggle). [^1] Jev handles ~32k (~150k chars) — a 60x gap, and the most consequential one.

Jev's flagship cookbook runs 13 questions over a long Wikipedia article; Laya cannot hold the document.

[^1]: Corrected. The original text said "192–1024"; 192 is the default `head_max_len`, not a trained `max_len`.

State is also truncated **from the right** silently: `truncate_left` defaults to `False` (`laya/common.py:44`) and is never plumbed through `predict`, so long inputs lose their tail with no warning.

**Fix:** ModernBERT (the encoder) supports 8k natively, so this is a config/training choice, not an architectural limit. Raise `max_len`, expose `truncate_left`, and warn on truncation.

### 3. Choice option ceiling: ~126 vs 255

All options must fit `head_max_len=192` tokens alongside instructions, each capped at 48 tokens (`laya/common.py:54`). Past that, `predict` raises `"options exceed head_max_len"` (`laya/agent.py:213`).

**Measured, not estimated:** with the real ModernBERT tokenizer and realistic descriptive labels, **~126 options fit**, not the 27–38 originally estimated. [^2] The gap to Jev's 255 is real but roughly half as wide as first assessed.

[^2]: Corrected. The original estimate assumed each option consumed its full token cost, but `build_sequence` shrinks the per-option budget when options do not fit (`laya/common.py:57-60`), so far more short options fit than a naive count suggests.

This rules out hierarchical classification and skill-suggestion use cases outright.

**Fix:** needs a two-stage or hierarchical scoring path. The only item here that is genuine design work.

**Measured outcome (GPU, 2026-09-18):** two-stage works, but *only with semantic grouping*. With
default chunking it scored 0.229 against a single call's 0.750 at 100 options, because stage 1
cannot discriminate between arbitrarily-named groups (stage-1 accuracy 0.042, stage-2 1.000).
With `group_of` supplying real categories it reaches 0.646. Guidance is now: single call below
~100 options, `two_stage_choice(..., group_of=...)` above.

---

## P1 — SDK ergonomics

Cheap, mechanical, and what makes the library feel production-ready.

| Feature | Jev | Laya |
|---|---|---|
| Async client | yes | missing (sync only) |
| Configurable retry policy | yes | missing |
| Typed error hierarchy | 12 classes (`RateLimitError`, `APITimeoutError`, …) | raw `ValueError` / `RuntimeError` |
| Typed answer objects | `ChoiceAnswer.confidence` | raw dicts |
| `request_id` + structured logging | yes | missing |
| JS/TS SDK | `@typesafe-ai/sdk` | Python only |

Typed answer objects and a proper exception hierarchy matter most — they are what callers write code against.

---

## P2 — Patterns library

Jev ships **18 cookbooks**; Laya has none. These are library-level compositions over the same primitives, not model features — the cheapest gap to close and the one that most changes day-to-day usability.

Highest value first: speculative fan-out, confidence-gated routing, composite scoring, self-consistency (run a question several ways, route disagreement to review), re-ranking, function calling, citation checking, multi-stage extraction cascades.

**Fix:** port as `laya.patterns` helpers on top of the existing `predict`.

---

## P3 — Tooling

Playground / console, hosted docs site, per-version known-issues page, agent skill for Claude Code.

---

## Where Laya already wins

Worth protecting — none of this exists in Jev:

- **Self-hosted, Apache-2.0 weights.** Zero egress, zero marginal cost, air-gap capable. Jev is waitlisted cloud-only.
- **Exposed calibration internals.** Per-bucket temperature scaling, `ece_score` for measuring your own calibration, `proper_reward` and `td_lambda_targets` for retraining. Jev hides all of it.
- **`act_head`** — an explicit act/escalate signal alongside the answer.
- **TD(λ) multi-turn trajectory modeling.** Jev has no documented multi-turn support at all.
- **Email-specific cleaning** (`laya/email.py`) and fine-tuning notebooks. Jev offers no fine-tuning.

---

## Benchmark caveat

The README's comparison table uses Jev's *published* numbers, not a head-to-head run. Jev's 70–500 ms is end-to-end network latency; Laya's ~38 ms is local GPU compute. Those are not the same measurement and the table should say so.

---

## Suggested order

1. Fix structured-criteria rendering (crash)
2. Raise `max_len`; expose `truncate_left`; warn on truncation
3. Typed answers + exception hierarchy
4. Async client + retries
5. Port 4–6 cookbooks as `laya.patterns`
6. Two-stage scoring for high-cardinality choice

---

## Implementation status

Landed in v0.2.0. 182 tests run without a model; 17 more run against the real checkpoint
under `LAYA_INTEGRATION=1`.

| Item | Status | Where |
|---|---|---|
| P0 #1 structured criteria | **Done** | `render_criterion` in `laya/common.py`; the noul `TypeError` is gone and all three branches emit JSON |
| P0 #2 truncation reporting | **Done** | `truncate_left` + `max_len` are constructor args; truncation is logged and returned as `Response.truncated` |
| P0 #2 long context | **Validated on GPU** | 8k fine-tuning measured: 0.594 → 1.000 accuracy, 0.350 → 0.000 ECE in 50 steps (~2.5 min, 9.7GB). Needs real data for a production checkpoint. 32k still not recommended. See [GPU validation report](gpu_validation_report.md) |
| P0 #3 option ceiling | **Resolved on GPU** | Measured: single call wins below ~100 options (0.750 vs 0.229); above that use `two_stage_choice` **with `group_of`** (0.646 vs 0.229 for default chunking). Two bugs found and fixed. See [GPU validation report](gpu_validation_report.md) |
| P1 typed errors | **Done** | `laya/errors.py`, 10 classes under `LayaError`, keeping builtin bases for compatibility |
| P1 typed answers | **Done** | `laya/types.py`; dict-compatible so existing callers and the notebooks keep working |
| P1 async + retries | **Done** | `laya/aio.py`: `AsyncAgent`, `RetryPolicy`, `batch(max_concurrency=…)` |
| P1 request_id + logging | **Done** | `Response.request_id`; warnings go through the `laya` logger instead of `print` |
| P1 JS/TS SDK | **Not started** | Out of scope for this pass |
| P2 patterns | **Done** | `laya/patterns.py`, 14 helpers — including `select_tool` (function calling) and `check_citation` (citation checking), the two the priority list named that were initially missed |
| P3 tooling | **Not started** | Playground, hosted docs, known-issues page |
| Benchmark caveat | **Done** | README now states the latency and accuracy comparisons are not head-to-head |

### GPU work: done

Both items were validated on an L40S on 2026-09-18 — see the
[GPU validation report](gpu_validation_report.md) for full numbers.

1. **Context window.** 8k fine-tuning works and is cheap (2.9 s/step, 9.7GB peak, converged in
50 steps). Accuracy and ECE both resolved on synthetic data. What remains is *data*, not
compute: a production 8k checkpoint needs real long-document examples.
2. **High-cardinality choice.** Resolved, with a negative result: `two_stage_choice` as
originally shipped was substantially *worse* than a single call, because stage 1 could not
discriminate between arbitrarily-chunked groups (stage-1 accuracy 0.042, stage-2 1.000).
Fixed two bugs and documented semantic grouping as the recommended path.

---

### Sources

[Docs index](https://docs.typesafe.ai/llms.txt) · [Advanced structure](https://docs.typesafe.ai/primitives/advanced.md) · [Question types](https://docs.typesafe.ai/sdk/python/api/types/questions.md) · [Response types](https://docs.typesafe.ai/sdk/python/api/types/responses.md) · [Confidence](https://docs.typesafe.ai/confidence.md) · [Parallel questions](https://docs.typesafe.ai/cookbooks/parallel_questions)
Loading