Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Tidy the Laya page: versions, plausibility numbers and IT caveats
Label the demo comparison as measured on Laya 0.3.11 and drop its
plausibility row, which disagreed with the controlled 0.3.21 numbers
in the world-knowledge note; the note now carries both phrasings.
List validation status codes among the ITs' expected failures, drop
the CI claim from the intro, and say in the README that the ITs talk
to TYPESAFE_BASE_URL when it is set.

Signed-off-by: Christian Tzolov <christian.tzolov@broadcom.com>
  • Loading branch information
tzolov committed Sep 29, 2026
commit dc9be8e174ba37e19f53524b410f5e341b884d6c
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,8 @@ versions — see
```

The whole default suite runs offline, with HTTP exercised through `MockRestServiceServer`.
The `*IT.java` classes talk to the real API and are opt-in twice over — behind the profile
The `*IT.java` classes talk to the live API (or to a compatible server named by
`TYPESAFE_BASE_URL`) and are opt-in twice over — behind the profile
*and* behind `@EnabledIfEnvironmentVariable` on `TYPESAFE_API_KEY` — so an exported key
never turns an ordinary build into a billed one.

Expand Down
17 changes: 8 additions & 9 deletions docs/client/Laya.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
[Laya](https://github.com/NandhaKishorM/laya) is an open-source (Apache-2.0) System One
engine. Its `laya-serve` server, added in Laya 0.3.7, exposes the same `POST /v1/systemone`
wire protocol as Jev. That means `TypeSafeClient` and everything built on it can run against a
Laya server on your own machine, unchanged. It is useful for offline development, local
experiments and CI without an API key.
Laya server on your own machine, unchanged. It is useful for offline development and local
experiments without an API key, at the cost of a model download.

!!! warning "A stand-in, not a replacement"
The protocol matches closely; the model does not. Laya's answers differ from Jev's, and
Expand Down Expand Up @@ -85,18 +85,16 @@ score legends and the 401 on a wrong key, behaves as it does with Jev.
## How it compares

Running this project's [demos](../demos.md) against Laya's English checkpoint and against
Jev, on the same inputs:
Jev, on the same inputs. Measured on Laya 0.3.11:

| Workload | Laya | Jev |
|---|---|---|
| Plausibility (−125 °C or −255 °C in Paris) | rated 0.66–0.74, close to plausible | rated 0.05–0.53 |
| RAG: a passage contradicting the others | `INCLUDED` and ranked first | `CONFLICTING` |
| Tool search on four paraphrased requests | 2 of 4 | 4 of 4 |
| Cascade: clean invoices flagged for escalation | 2 of 3 | 0 of 3 |
| Guardrails: a self-harm message | flagged for review | routed to support |

The table shows one run. The tool search, cascade and judge results changed from run to
run.
The table shows one run. The tool search and cascade results changed from run to run.

In short:

Expand All @@ -112,8 +110,9 @@ thresholds from measurements against Laya itself.

### Criteria that need world knowledge

Physical plausibility is the clearest weak spot. Asked whether the values in an answer are
physically plausible, Laya scores −125 °C and −255 °C close to 15 °C. Stating the bounds in
Physical plausibility is the clearest weak spot. Measured on Laya 0.3.21: asked whether
the values in an answer are physically plausible, Laya scores −125 °C and −255 °C close to
15 °C (0.48–0.72 against 0.81), where Jev gives 0.02–0.20 against 0.97. Stating the bounds in
the question ("is every temperature between −90 and +60 °C?") narrows the gap without
closing it: 0.67 for 15 °C against 0.41–0.52 for the impossible values. Jev answers the same
question 0.97 against 0.02.
Expand All @@ -133,7 +132,7 @@ TYPESAFE_BASE_URL=http://localhost:8002 TYPESAFE_API_KEY=local-test \
```

Expect failures. Some ITs assert protocol details Laya lacks (model listing, request id,
output tokens, error types). Others assert answers tuned on Jev.
output tokens, error types, validation status codes). Others assert answers tuned on Jev.

## See Also

Expand Down
Loading