Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,14 +96,17 @@ versions — see
```

The whole default suite runs offline, with HTTP exercised through `MockRestServiceServer`.
The `*IT.java` classes talk to the real API and are opt-in twice over — behind the profile
The `*IT.java` classes talk to the live API (or to a compatible server named by
`TYPESAFE_BASE_URL`) and are opt-in twice over — behind the profile
*and* behind `@EnabledIfEnvironmentVariable` on `TYPESAFE_API_KEY` — so an exported key
never turns an ordinary build into a billed one.

> [!WARNING]
> While working on the SDK, do not export `TYPESAFE_BASE_URL` or `TYPESAFE_DEFAULT_MODEL`.
> `TypeSafeClient.builder()` falls back to both, and any throwaway `main` you write will
> pick them up.
> `TypeSafeClient.builder()` falls back to both, and any throwaway `main` you write, and
> the live ITs, will pick them up. Export `TYPESAFE_BASE_URL` only to run the ITs against a
> compatible server on purpose, such as a local
> [Laya](https://spring-ai-community.github.io/spring-ai-typesafe/latest-snapshot/client/Laya/).

Run the demos from
[Demos](https://spring-ai-community.github.io/spring-ai-typesafe/latest-snapshot/demos/).
Expand Down
141 changes: 141 additions & 0 deletions docs/client/Laya.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# Using Laya

[Laya](https://github.com/NandhaKishorM/laya) is an open-source (Apache-2.0) System One
engine. Its `laya-serve` server, added in Laya 0.3.7, exposes the same `POST /v1/systemone`
wire protocol as Jev. That means `TypeSafeClient` and everything built on it can run against a
Laya server on your own machine, unchanged. It is useful for offline development and local
experiments without an API key, at the cost of a model download.

!!! warning "A stand-in, not a replacement"
The protocol matches closely; the model does not. Laya's answers differ from Jev's, and
the thresholds in this project's examples and tests were tuned on Jev. Check your
criteria against Laya before relying on its decisions. See [how it compares](#how-it-compares).

## Run `laya-serve`

```bash
uv venv --python 3.10
uv pip install "laya[serve]"

LAYA_DEVICE=mps \
LAYA_MODELS=english \
LAYA_PRELOAD=1 \
LAYA_API_KEY=local-test \
LAYA_PORT=8002 \
.venv/bin/laya-serve
```

- **`LAYA_DEVICE`:** `mps` on Apple Silicon, `cuda` or `cpu` elsewhere. On CPU, set
`LAYA_THREADS` to the number of physical cores.
- **`LAYA_MODELS`:** the checkpoints to load. The first start downloads them from Hugging Face.
- **`LAYA_PORT`:** Laya defaults to 8000, which `mkdocs serve` and many dev servers also use.
This page uses 8002.
- **`LAYA_API_KEY`:** optional. When set, requests need `Authorization: Bearer <key>`.

`curl -s localhost:8002/health` reports the device and the loaded checkpoints once it's ready.
See Laya's own docs for Docker images and the other settings.

## Point the client at it

The client always sends a key, so give it one. When `LAYA_API_KEY` is set, it must be the
same value.

=== "Plain Java"

```java
TypeSafeClient client = TypeSafeClient.builder()
.baseUrl("http://localhost:8002")
.apiKey("local-test")
.build();
```

Or leave both out and export `TYPESAFE_BASE_URL=http://localhost:8002` and
`TYPESAFE_API_KEY=local-test`. `builder().build()` falls back to them.

=== "Spring Boot"

```properties
spring.ai.typesafe.base-url=http://localhost:8002
spring.ai.typesafe.api-key=local-test
```

The starter does not read `TYPESAFE_BASE_URL`. Set the property, or
`SPRING_AI_TYPESAFE_BASE_URL`, for example in a `laya` profile.

## What differs

Measured against Laya 0.3.21:

| Area | Laya | Effect |
|---|---|---|
| `GET /v1/models` | not implemented, 404 | `listModels()` throws `TypeSafeNotFoundException` |
| `model` | honoured only for Laya's checkpoint names (`english`, `multilingual`, `typed-decisions`). Anything else, such as `jev-latest`, is auto-routed | `TypeSafeModels` constants have no effect. `response.model()` names Laya's model |
| Noul without instructions | rejected, 422 | Give every noul `instructions`, even when it has `whenTrue` / `whenFalse` criteria |
| Request id | not sent | `requestId()` is `null` |
| Output tokens | always `0` | `usage().outputTokens()` is `0` |
| Error body | `{"detail": "..."}`, no error type | `errorMessage()` is set, `errorType()` is `null` |
| Validation errors | a malformed question is 422, where Jev answers 400; a `null` state is 400, where Jev answers 422 | Don't rely on the exact exception subtype |
| Extra answer fields | `action`, `routing` | Ignored by the client |
| Limits | 64 questions, 50,000 characters of state, 2 MB request | Larger requests fail with 413 |
| Concurrency | one forward pass at a time | [Batches](Batches.md) are served one request after another |

Everything else, including the noul, choice and score answers, probabilities, confidence,
score legends and the 401 on a wrong key, behaves as it does with Jev.

## How it compares

Running this project's [demos](../demos.md) against Laya's English checkpoint and against
Jev, on the same inputs. Measured on Laya 0.3.11:

| Workload | Laya | Jev |
|---|---|---|
| RAG: a passage contradicting the others | `INCLUDED` and ranked first | `CONFLICTING` |
| Tool search on four paraphrased requests | 2 of 4 | 4 of 4 |
| Cascade: clean invoices flagged for escalation | 2 of 3 | 0 of 3 |
| Guardrails: a self-harm message | flagged for review | routed to support |

The table shows one run. The tool search and cascade results changed from run to run.

In short:

- Laya's answers are less decisive: confidences sit nearer the middle.
- It misses subtler signals.
- It is sensitive to the order of fields in the state. The same record with its keys
reordered can score quite differently, so build states in a fixed order (`LinkedHashMap`,
not `Map.of`) when you compare runs.
- A warm three-question call takes about 30 ms on a laptop GPU.

Use Laya where speed, cost or data locality matter more than Jev's accuracy, and set
thresholds from measurements against Laya itself.

### Criteria that need world knowledge

Physical plausibility is the clearest weak spot. Measured on Laya 0.3.21: asked whether
the values in an answer are physically plausible, Laya scores −125 °C and −255 °C close to
15 °C (0.48–0.72 against 0.81), where Jev gives 0.02–0.20 against 0.97. Stating the bounds in
the question ("is every temperature between −90 and +60 °C?") narrows the gap without
closing it: 0.67 for 15 °C against 0.41–0.52 for the impossible values. Jev answers the same
question 0.97 against 0.02.

So the [Model-as-a-judge demo](../demos.md#modeljudgedemoapplication), whose `is_plausible`
criterion depends on exactly this, isn't a meaningful test of Laya: it often passes an
impossible temperature. On Laya, check such things in code instead, with a
[code criterion](../judge/JevJudge.md#code-criteria) such as "no temperature below −90 °C".

## Integration tests against Laya

The live ITs follow `TYPESAFE_BASE_URL`, so they run against Laya too:

```bash
TYPESAFE_BASE_URL=http://localhost:8002 TYPESAFE_API_KEY=local-test \
./mvnw -Pintegration-tests -pl typesafe-java-sdk,typesafe-spring-ai verify
```

Expect failures. Some ITs assert protocol details Laya lacks (model listing, request id,
output tokens, error types, validation status codes). Others assert answers tuned on Jev.

## See Also

- [TypeSafeClient](TypeSafeClient.md) — builder settings and environment variables
- [Spring Boot Starter](SpringBootStarter.md) — properties
- [Errors and Retries](ErrorsAndRetries.md) — the exception types above
3 changes: 2 additions & 1 deletion docs/client/SpringBootStarter.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,8 @@ properties.

!!! warning "Do not export TYPESAFE_BASE_URL while developing"
The plain-Java builder falls back to it, so an exported value silently redirects any
throwaway `main` you write, and any test that does not pin the base URL itself.
throwaway `main` you write, and the live ITs. That is also how to point them at a
compatible server on purpose; see [Using Laya](Laya.md).

## Endpoints bean

Expand Down
1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ nav:
- Batches: client/Batches.md
- Errors and Retries: client/ErrorsAndRetries.md
- Spring Boot Starter: client/SpringBootStarter.md
- Using Laya: client/Laya.md
- Model-as-a-Judge:
- JevJudge: judge/JevJudge.md
- JevSelfRefineAdvisor: judge/JevSelfRefineAdvisor.md
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@
class TypeSafeLiveApiIT {

private final TypeSafeClient client = TypeSafeClient.builder()
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.build();

Expand Down Expand Up @@ -124,7 +123,6 @@ void reportsTokenUsage() {
void mapsARejectedKeyOntoAnAuthenticationException() {
TypeSafeClient badKey = TypeSafeClient.builder()
.apiKey("sk-not-a-real-key")
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.retryPolicy(RetryPolicy.noRetry())
.build();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@ record Triage(String team, double urgent, double severity) {

private final ChatClient chatClient = ChatClient.create(JevChatModel
.builder(TypeSafeClient.builder()
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.build())
.question("team",
Expand Down Expand Up @@ -109,7 +108,6 @@ void reportsRealTokenUsageOnTheStandardMeter() {
registry.observationConfig().observationHandler(new ChatModelMeterObservationHandler(meters));
JevChatModel observed = JevChatModel
.builder(TypeSafeClient.builder()
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.build())
.question("urgent", Noul.of("Does the user's ticket need attention right now?"))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,6 @@ class JevJudgeIT {
.build();

private final TypeSafeClient client = TypeSafeClient.builder()
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.build();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,6 @@ class JevRagPostProcessorIT {
.build();

private final TypeSafeClient client = TypeSafeClient.builder()
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.build();

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,6 @@ class JevToolIndexIT {
private static final String SESSION = "it-session";

private final TypeSafeClient client = TypeSafeClient.builder()
.baseUrl(TypeSafeConstants.DEFAULT_BASE_URL)
.defaultModel(TypeSafeModels.JEV_LATEST)
.build();

Expand Down
Loading