Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions laya-dotnet/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,19 @@ goldens: 0.4.12).
`ONNXAgent.predict_batch` and `Router.predict_batch` / `route_batch`: shared forward passes, `batch_size`
chunking, `sort_by_length` inside windows of eight batches (effective only for `1 < batch_size < len(states)`),
routing grouped by checkpoint then question schema with input order restored, and per-request `lang_guess`.
Hooks, per-call `lang` / `max_len` / `head_max_len` and the router `diagnostics`
dictionary are not ported yet. Existing signatures are unchanged.
Hooks and the router `diagnostics` dictionary are not ported yet. Existing signatures are unchanged.
- Per-language temperatures and per-call overrides, ported from Python `lang_temperatures` /
`resolve_lang_temperatures` and the `lang`, `max_len` and `head_max_len` arguments: `LayaOptions.LangTemperatures`
(a table of `LayaLangTemperature`, resolved when the engine is created, with Python's key normalisation, clamping,
warning and `ValueError` text; a router hands it to every engine it builds), `Lang`, `MaxLen` and `HeadMaxLen` on
`LayaPredictOptions` and `LayaBatchOptions`, `MaxLen` and `HeadMaxLen` on `LayaBatchRequest`, and
`LayaEngine.PredictLong(state, questions, LayaLongOptions)` with `Lang`. A language the table holds replaces both
the checkpoint's temperature and its per-bucket map. A router forwards its `lang` argument, else the language it
detected, to the checkpoint, and `PredictBatch` splits a checkpoint's requests by token budget and, for a
checkpoint with a table, by language. Token budgets are unchecked as in Python: `0`, a negative value or one too
small for the options' markers throws the "option markers fit" `ArgumentException`. New interfaces
`ILayaCallPredictor`, `ILayaLongLangPredictor` and `ILayaLangTemperatures` let a router pass these on; existing
signatures are unchanged.
- Abstention gate, ported from Python `laya/confidence.py` (`min_confidence`, `jev_confidence`): `MinConfidence`
(`Of` for one threshold, `ForBuckets` for a per-option-count map, `BucketKeys`), `LayaPredictOptions`
with new `LayaEngine.Predict(state, questions, options)` and `LayaRouter.Predict(..., langGuess, options)`
Expand Down Expand Up @@ -126,6 +137,5 @@ goldens: 0.4.12).

### Known gaps

- Python-only opt-in features are not yet ported: `lang_temperatures`, hooks, question option
ordering and digest controls. Batching is ported without hooks and per-call
`lang` / `max_len` / `head_max_len`.
- Python-only opt-in features are not yet ported: hooks, question option ordering, digest controls and
calibration binning (`load_calibration`). Batching is ported without hooks.
77 changes: 64 additions & 13 deletions laya-dotnet/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,18 +367,61 @@ IReadOnlyList<LayaResult> results = engine.PredictBatch(
window at a time, so an unanswerable state fails after the earlier windows have run, as in Python.
- Each result carries the usage diagnostics Python's ONNX path reports (`StateTokens`,
`StateTokensDropped`, `Truncated`, `TruncatedQuestions`, `Options`), the same as `Predict` reports for a single state.
- `LayaBatchOptions` is a class of init-only properties, so later per-call options extend it without
- `LayaBatchOptions` is a class of init-only properties, so further per-call options extend it without
changing a signature. `ILayaBatchPredictor` is a separate interface; `ILayaPredictor` is unchanged.

**Router.** `LayaRouter.PredictBatch(requests, options)` takes `LayaBatchRequest`s (`State`, `Questions`,
and the optional routing overrides `Model`, `Task`, `Lang`, `LangGuess`), routes all of them first, groups them
by routed checkpoint (first-seen order, so each checkpoint loads at most once), splits each group by question
schema (order-sensitive) so same-schema states share a pass, and returns results in request order, each with
`Routing`. `LayaRouter.RouteBatch(requests)` returns just the `RouteDecision`s and loads nothing. A null
and the optional routing overrides `Model`, `Task`, `Lang`, `LangGuess`, and the token budgets `MaxLen`,
`HeadMaxLen`), routes all of them first, groups them by routed checkpoint (first-seen order, so each checkpoint
loads at most once), splits each group by question schema (order-sensitive), token budgets and, for a checkpoint
that has [per-language temperatures](#per-language-temperatures-and-per-call-options), language, so only requests
that agree on all of those share a pass, and returns results in request order, each with `Routing`. `LayaRouter.RouteBatch(requests)` returns just the `RouteDecision`s and loads nothing. A null
state or questions, or an unknown model, fails the whole call before anything loads. Attached predictors must
implement `ILayaBatchPredictor`, else `NotSupportedException`. Hooks, per-call
`lang` / `max_len` / `head_max_len` and the `diagnostics` dictionary are not ported yet.
implement `ILayaBatchPredictor`, else `NotSupportedException`. Hooks and the `diagnostics` dictionary are not
ported yet.

### Per-language temperatures and per-call options

Python's opt-in `lang_temperatures`, and the per-call `lang`, `max_len` and `head_max_len`. All are off by default,
so default predictions are unchanged.

```csharp
using var engine = LayaEngine.Create(new LayaOptions
{
LangTemperatures = new Dictionary<string, LayaLangTemperature?>
{
["de"] = new() { Temperature = [1.4, 1.8, 2.0] }, // choice, score, noul
["fr"] = null, // the checkpoint's own temperature
["es"] = new() { TemperatureByOptions = new Dictionary<string, double> { ["choice:3-5"] = 2.5 } },
},
});

var result = engine.Predict(state, questions, new LayaPredictOptions { Lang = "de-AT", MaxLen = 256, HeadMaxLen = 96 });
engine.PredictBatch(states, questions, new LayaBatchOptions { Lang = "de" });
engine.PredictLong(document, questions, new LayaLongOptions { Lang = "de", BatchSize = 2 });
```

- **The table** (`LayaOptions.LangTemperatures`) maps a language code to `LayaLangTemperature` (`Temperature`: exactly three
values in `choice`, `score`, `noul` order, or `null` to inherit the checkpoint's; `TemperatureByOptions`: bucket to
value). It is resolved when the engine is created, as Python's `resolve_lang_temperatures` does: a key is matched on
its part before the first `-`, lower-cased (`"de-AT"`, `"DE"` and `"de"` are one language, `"de_DE"` is not), the last
of several keys for one language wins, every value is clamped to 0.5..5, and a value the clamp moved is reported once
per language through `System.Diagnostics.Trace` (the text Python warns with). A `Temperature` that is not three values
throws `ArgumentException` carrying Python's `ValueError` text. A router hands the table to every engine it builds
(`LayaRouterOptions.EngineOptions`).
- **A call that names a language the table holds** (`Lang` on `LayaPredictOptions` / `LayaBatchOptions` /
`LayaLongOptions`) replaces the checkpoint's per-type temperature **and** its per-bucket map with that language's; there
is no fallback to the checkpoint's bucket map. A missing, empty or unknown language changes nothing.
- **On a router** the language is the `lang` argument of `Predict` / `PredictLong` (it also routes), else the language the
router's own detection found (a `lang_guess` hint or an explicit `model` leaves none), and it is handed to the engine.
In `PredictBatch` the request's `Lang` (or the detected one) joins the grouping key, as written (`"de"` and `"de-AT"`
are separate passes), only for a checkpoint that has a table.
- **`MaxLen` / `HeadMaxLen`** replace the checkpoint's `max_len` / `head_max_len` for one call (`LayaPredictOptions`,
`LayaBatchOptions`, and per request on `LayaBatchRequest`; not on `PredictLong`, which takes none in Python). Python
checks nothing, and neither does this: `0`, a negative value or one too small for a question's option markers makes the
call throw the same `ArgumentException` a checkpoint with such a budget would (`only N of its M option markers fit in
max_len=...`), while a small or negative `HeadMaxLen` squeezes each option to 4 tokens and the instruction to 8.
Booleans are not a .NET `int`: `True`/`False` are `1`/`0`.
### Abstention: `MinConfidence`

Python's opt-in `min_confidence`, off by default. Pass a `MinConfidence` per call and every answer says
Expand Down Expand Up @@ -1006,7 +1049,7 @@ with `--solution` rather than as a positional argument.
## Not in v1

- Quantized artifacts.
- Async `Predict`, and runtime overrides of `max_len` / `head_max_len`.
- Async `Predict`.
- `embed_fn_from_agent` (mean-pooling the checkpoint's own encoder into a shortlist embedder) —
`LayaShortlist` takes only a caller-supplied embedding function; see [Routing, language
detection, email and shortlist](#routing-language-detection-email-and-shortlist).
Expand All @@ -1019,12 +1062,12 @@ identifies the foreign line or field that overrides a mostly-English state. Unde
Latin text uses the router's configured default. Email cleaning follows current request,
disclaimer, signature and reply-header rules, including Portuguese, Spanish and French markers.

- **Opt-in features added in Python 0.3.21 are not ported yet:** per-language temperatures
(`lang_temperatures`). They are off by default in Python, so default predictions are unaffected. They
are planned as follow-ups. Abstention (`min_confidence`) is ported (see "Abstention" above).
- **Opt-in features added in Python 0.3.21:** abstention (`min_confidence`) and per-language temperatures
(`lang_temperatures`, with per-call `lang`) are ported (see "Abstention" and "Per-language temperatures" above).
Calibration binning (`load_calibration`) is not, so the binning a language override skips in Python has nothing to skip here.
- **Other Python-only APIs are not ported:** hooks, question option ordering and digest controls.
Batching is ported (`PredictBatch`, `LayaRouter.PredictBatch` / `RouteBatch`) without hooks,
per-call `lang` / `max_len` / `head_max_len` and the router `diagnostics` dictionary.
Batching is ported (`PredictBatch`, `LayaRouter.PredictBatch` / `RouteBatch`) without hooks and the router
`diagnostics` dictionary.
- **Usage diagnostics:** Python's `state_tokens`, `truncated`, `state_tokens_dropped`, `truncated_questions`
and `options` are reported by `Predict`, `PredictBatch` and `PredictLong`, as Python's `system_one` is
`predict_batch([state])`.
Expand All @@ -1040,6 +1083,14 @@ Where this SDK deliberately does not match the Python reference, so nobody has t
Python then slices from the end, which this port does not support. Python loads such a checkpoint and fails later. The
related corner: a small *fractional* `head_max_len` that Python happens to survive (roughly up to 8 plus 4 per option)
is refused here too.
- **`LangTemperatures` is a typed table.** Python also refuses a key that is not a string, an entry or
`temperature_by_options` that is not a mapping, and a `temperature` that is not a list; a typed table cannot hold
those, so only the three-value check (and the inherited-list check) remains. A warning prints a float the way Python
prints a float (`99.0`), where Python prints the `99` an integer was written as.
- **A router takes the language, and the token budgets, in the places Python does.** `LayaPredictOptions.Lang` on
`LayaRouter.Predict` and `LayaBatchOptions.Lang` / `MaxLen` / `HeadMaxLen` on `LayaRouter.PredictBatch` throw
`ArgumentException`: Python's `predict` has the one `lang` argument, and `predict_batch` has no such arguments (it is a
`TypeError` there), only the per-request keys.
- **JSON nesting.** Config and tokenizer JSON is read like `json.loads`, with the nesting cut-off set to 2,997 levels,
what CPython 3.12 measured from a shallow call. Python's real limit depends on the call depth at the point of loading.
- **Config files are decoded as strict UTF-8.** Python opens them with the locale's encoding (cp1252 on a default Windows
Expand Down
37 changes: 37 additions & 0 deletions laya-dotnet/src/Laya/ILayaLongPredictor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,40 @@ public interface ILayaLongPredictor : ILayaPredictor
LayaResult PredictLong(object state, QuestionSet questions, int? window = null, int? stride = null,
string aggregate = "auto", int? batchSize = null);
}

/// <summary>
/// Per-call knobs for <see cref="LayaEngine.PredictLong(object, QuestionSet, LayaLongOptions?)"/> and the router's
/// <c>PredictLong</c>: the arguments of <see cref="ILayaLongPredictor.PredictLong"/> plus <see cref="Lang"/>.
/// <c>max_len</c> and <c>head_max_len</c> are deliberately absent: Python's <c>predict_long</c> takes neither, the
/// windows are sized from <see cref="Window"/> and the checkpoint's own budget.
/// </summary>
public sealed class LayaLongOptions
{
/// <summary>State tokens per window; nonpositive or <see langword="null"/> uses the checkpoint default.</summary>
public int? Window { get; init; }

/// <summary>Token step; nonpositive or <see langword="null"/> uses half the effective window.</summary>
public int? Stride { get; init; }

/// <summary>Only <c>"auto"</c> is supported.</summary>
public string Aggregate { get; init; } = "auto";

/// <summary>Maximum windows per forward pass; nonpositive or <see langword="null"/> uses Python's automatic cap.</summary>
public int? BatchSize { get; init; }

/// <summary>
/// Language of the state (Python <c>lang</c>), applied to every window; see <see cref="LayaPredictOptions.Lang"/>.
/// </summary>
public string? Lang { get; init; }
}

/// <summary>
/// A scanner that takes the per-call <see cref="LayaLongOptions.Lang"/>: a <see cref="LayaEngine"/>. A
/// <see cref="LayaRouter"/> scanning with one that does not implement this runs the scan without the language and
/// warns, as Python does for a <c>predict_long</c> with no <c>lang</c> parameter.
/// </summary>
public interface ILayaLongLangPredictor : ILayaLongPredictor
{
/// <summary><see cref="ILayaLongPredictor.PredictLong"/> with <paramref name="options"/>.</summary>
LayaResult PredictLong(object state, QuestionSet questions, LayaLongOptions? options);
}
25 changes: 25 additions & 0 deletions laya-dotnet/src/Laya/ILayaPredictor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,28 @@ public interface ILayaPredictor
/// <summary>Answer every question in <paramref name="questions"/> about <paramref name="state"/>.</summary>
LayaResult Predict(object? state, QuestionSet questions);
}

/// <summary>
/// A predictor that also takes the per-call <see cref="LayaPredictOptions.Lang"/>, <see cref="LayaPredictOptions.MaxLen"/>
/// and <see cref="LayaPredictOptions.HeadMaxLen"/>: a <see cref="LayaEngine"/>. A <see cref="LayaRouter"/> forwards them
/// to the checkpoint it routed to; one that does not implement this is called without them (and refused when a token
/// budget was asked for, as Python refuses a keyword the agent does not accept).
/// </summary>
public interface ILayaCallPredictor : ILayaPredictor
{
/// <summary>
/// <see cref="ILayaPredictor.Predict"/> with the per-call language and token budgets of
/// <paramref name="options"/>. <see cref="LayaPredictOptions.MinConfidence"/> is the caller's to apply.
/// </summary>
LayaResult Predict(object? state, QuestionSet questions, LayaPredictOptions? options);
}

/// <summary>
/// Whether a predictor carries per-language temperatures (Python <c>getattr(agent, "lang_temperatures", None)</c> being
/// non-empty). A <see cref="LayaRouter"/> only splits a batch by language for a checkpoint that does.
/// </summary>
public interface ILayaLangTemperatures
{
/// <summary>Whether <see cref="LayaOptions.LangTemperatures"/> named at least one language.</summary>
bool HasLangTemperatures { get; }
}
Loading
Loading