Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions docs/evals.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,48 @@ metrics per slice value, so a regression in one language or one question is visi
reading the aggregate. The `model` slice holds the checkpoint that answered each row: the
`Router`'s own choice per request, or the runner's `model` for a runner that does not route.

### Opt-in slice gates

The overall baseline gate can pass while a smaller language or question slice regresses. To make
one reviewed slice a CI requirement, save a JSON policy such as `gates.json`:

```json
{
"version": 1,
"rules": [
{"slice": {"language": "zh"}, "metric": "choice_accuracy",
"min_count": 50, "max_drop": 0.05},
{"slice": {"qid": "intent"}, "metric": "ece",
"min_count": 50, "max": 0.10}
]
}
```

```bash
laya-evals run data.jsonl --baseline baseline.json --tolerance choice_accuracy=0.02 \
--gate-policy gates.json --json report.json
laya-evals compare report.json --baseline baseline.json \
--tolerance choice_accuracy=0.02 --gate-policy gates.json
```

Each rule selects exactly one `language`, `model`, `qid`, or `tag` value and names the metric
exactly as it appears in the slice report. It has a positive `min_count` and exactly one limit:
`min` or `max` checks the candidate value; `max_drop` permits at most that decrease from the
baseline; `max_increase` permits at most that increase. The latter two require `--baseline`.
The count is the number of scored answers for that metric in the selected slice, in **both**
reports for a relative rule. For `ece`, it is the number of answers with a finite confidence and
boolean `correct` value. A missing slice or metric, too few scored answers, or skipped/errored
cases fails the opted-in gate. Relative rules also require both reports to carry matching run
identities, so missing evidence cannot appear as a pass. A measured regression reports the slice,
metric, counts, values, and limit. Invalid policy syntax exits 2 before a checkpoint loads; a
quality failure exits 1. The policy is recorded in `config.gate_policy` of a `run --json` report.
`compare --gate-policy` applies the policy supplied on that command line to the saved measurements.
If it differs from the report's recorded policy, `compare` says so; an explicit re-check under a
new policy does not change the policy under which the original run was made.

The regular overall comparison still applies, including its tolerance and legacy-baseline
behavior. Without `--gate-policy`, slice reporting and comparison behave as before.

## Run identity

`run` records what it measured in the report's `config` block, so the artifact a reviewer reads
Expand All @@ -152,6 +194,7 @@ is reviewable on its own:
| `questions_sha256` | a fingerprint of the question schema: every question's id, type, `instructions` and `criteria`, over the whole dataset |
| `laya_version` | the `laya` that computed the numbers |
| `thresholds` | the gate this run applied: `min`, `max` and `baseline_tolerance` |
| `gate_policy` | the optional slice gate policy applied by `run --gate-policy` |
| `revisions` | the commit each checkpoint that answered was loaded from (see [below](#baseline-and-ci-gate)) |

`dataset` is a path, and a path is not an identity: a dataset can be edited in place, moved, or
Expand Down
2 changes: 1 addition & 1 deletion docs/typescript-sdk.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,6 @@ authentication and request limits. CI runs the SDK checks on Node.js 22 and 24.
Tiny random weights verify transport and numerical parity, not pretrained
quality or performance.

See the [SDK guide](../sdk/typescript/README.md) for setup, examples, and npm
See the [SDK guide](https://github.com/NandhaKishorM/laya/blob/main/sdk/typescript/README.md) for setup, examples, and npm
publishing. The package will be published under the `laya-client` name. Python release workflows
are unchanged.
167 changes: 167 additions & 0 deletions laya/_eval_policy.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
"""Opt-in, weight-free quality gates over the slices already in an eval report."""

import json
import math
from typing import Any, Dict, List, Optional

from . import evals


_DIMENSIONS = ("language", "model", "qid", "tag")
_LIMITS = ("min", "max", "max_drop", "max_increase")
_RELATIVE = ("max_drop", "max_increase")


def _finite_number(value: Any) -> bool:
if type(value) not in (int, float):
return False
try:
return math.isfinite(value)
except OverflowError:
return False


def load_policy(path: str) -> Dict[str, Any]:
"""Read and validate a small, versioned JSON gate before a checkpoint is loaded."""
try:
with open(path, "r", encoding="utf-8") as handle:
policy = json.load(handle)
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
raise evals.EvalError("invalid gate policy %r: %s" % (path, exc)) from exc
if not isinstance(policy, dict) or set(policy) != {"version", "rules"}:
raise evals.EvalError("gate policy needs exactly 'version' and 'rules'")
if type(policy["version"]) is not int or policy["version"] != 1:
raise evals.EvalError("gate policy version must be 1")
rules = policy["rules"]
if not isinstance(rules, list) or not rules:
raise evals.EvalError("gate policy 'rules' must be a non-empty list")
for index, rule in enumerate(rules, 1):
where = "gate policy rule %d" % index
if not isinstance(rule, dict):
raise evals.EvalError("%s must be an object" % where)
limits = set(rule) & set(_LIMITS)
if (set(rule) - {"slice", "metric", "min_count"} - set(_LIMITS)
or len(limits) != 1 or not {"slice", "metric", "min_count"} <= set(rule)):
raise evals.EvalError("%s needs slice, metric, min_count and exactly one limit" % where)
selector = rule["slice"]
if (not isinstance(selector, dict) or len(selector) != 1
or next(iter(selector)) not in _DIMENSIONS
or not isinstance(next(iter(selector.values())), str)
or not next(iter(selector.values())).strip()):
raise evals.EvalError("%s slice must name one non-empty language/model/qid/tag value" % where)
if not isinstance(rule["metric"], str) or not rule["metric"].strip():
raise evals.EvalError("%s metric must be a non-empty string" % where)
if type(rule["min_count"]) is not int or rule["min_count"] < 1:
raise evals.EvalError("%s min_count must be a positive integer" % where)
limit_name = next(iter(limits))
limit = rule[limit_name]
if (not _finite_number(limit)
or (limit_name in _RELATIVE and limit < 0)):
raise evals.EvalError("%s %s must be finite%s" %
(where, limit_name,
" and non-negative" if limit_name in _RELATIVE else ""))
return policy


def needs_baseline(policy: Dict[str, Any]) -> bool:
return any(set(rule) & set(_RELATIVE) for rule in policy["rules"])


def _count(report: evals.EvalReport, dimension: str, value: str, metric: str) -> int:
cases = report.cases if isinstance(report.cases, list) else []
count = 0
for case in cases:
if not isinstance(case, dict):
continue
if dimension == "tag":
tags = case.get("tags")
values = tags if isinstance(tags, (list, tuple)) else []
else:
values = [case.get(dimension)]
if value not in [str(item) for item in values if item is not None]:
continue
if metric == "ece":
confidence, correct = case.get("confidence"), case.get("correct")
valid = _finite_number(confidence) and isinstance(correct, bool)
else:
scores = case.get("scores")
score = scores.get(metric) if isinstance(scores, dict) else None
valid = _finite_number(score)
count += bool(valid)
return count


def _value(report: evals.EvalReport, dimension: str, value: str,
metric: str) -> Optional[float]:
slices = report.slices if isinstance(report.slices, dict) else {}
dimension_slices = slices.get(dimension)
if not isinstance(dimension_slices, dict):
return None
group = dimension_slices.get(value)
if not isinstance(group, dict):
return None
result = group.get(metric)
if not _finite_number(result):
return None
return float(result)


def check_policy(report: evals.EvalReport, policy: Dict[str, Any],
baseline: Optional[Dict[str, Any]] = None) -> List[str]:
"""Return actionable failures. Missing measurements never count as a pass."""
failures: List[str] = []
base_report = None
if baseline is not None:
base_report = evals.EvalReport(
config=evals._identity_of(baseline),
slices=baseline.get("slices", {}), cases=baseline.get("cases", []))
if report.config.get("errored"):
failures.append("slice gate: candidate has skipped/errored cases")
if base_report is not None and base_report.config.get("errored"):
failures.append("slice gate: baseline has skipped/errored cases")
if needs_baseline(policy):
if base_report is None:
return failures + ["slice gate: relative rule needs a baseline"]
missing = [key for key in evals._IDENTITY_KEYS
if report.config.get(key) is None or base_report.config.get(key) is None]
if missing:
failures.append("slice gate: baseline and candidate need %s identity"
% ", ".join(missing))
comparable, reasons = report.comparable_to(baseline)
if not comparable:
failures.append("slice gate: baseline is not comparable: " + "; ".join(reasons))
for index, rule in enumerate(policy["rules"], 1):
dimension, value = next(iter(rule["slice"].items()))
metric = rule["metric"]
label = "slice gate rule %d (%s=%s, %s)" % (index, dimension, value, metric)
current = _value(report, dimension, value, metric)
current_count = _count(report, dimension, value, metric)
minimum = rule["min_count"]
if current is None:
failures.append("%s: metric or slice is missing" % label)
continue
if current_count < minimum:
failures.append("%s: candidate count %d is below min_count %d" %
(label, current_count, minimum))
continue
comparator = next(name for name in _LIMITS if name in rule)
limit = rule[comparator]
if comparator in _RELATIVE:
previous = _value(base_report, dimension, value, metric)
previous_count = _count(base_report, dimension, value, metric)
if previous is None:
failures.append("%s: baseline metric or slice is missing" % label)
continue
if previous_count < minimum:
failures.append("%s: baseline count %d is below min_count %d" %
(label, previous_count, minimum))
continue
drift = previous - current if comparator == "max_drop" else current - previous
if drift > limit and not math.isclose(drift, limit, rel_tol=1e-12, abs_tol=1e-12):
failures.append("%s: candidate=%.4f (n=%d), baseline=%.4f (n=%d), %s=%.4f exceeded "
"by %.4f" % (label, current, current_count, previous,
previous_count, comparator, limit, drift - limit))
elif (comparator == "min" and current < limit) or (comparator == "max" and current > limit):
failures.append("%s: candidate=%.4f (n=%d), %s=%.4f failed" %
(label, current, current_count, comparator, limit))
return failures
30 changes: 25 additions & 5 deletions laya/evals_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
import sys
from typing import Any, Dict, List, Optional, Sequence, Tuple

from . import evals
from . import _eval_policy, evals
from .evals import EvalError


Expand Down Expand Up @@ -170,6 +170,8 @@ def _build_parser() -> argparse.ArgumentParser:
run.add_argument("--max", action="append", metavar="METRIC=VALUE", help="maximum for any metric")
run.add_argument("--slice", action="append", choices=("language", "model", "qid", "tag"),
help="also report this slice dimension; repeatable")
run.add_argument("--gate-policy", metavar="FILE",
help="apply opt-in per-slice quality rules from a JSON policy")
run.add_argument("--json", dest="json_out", help="write the full report JSON here")
run.add_argument("--markdown", dest="markdown_out", help="write a Markdown summary here")

Expand All @@ -178,6 +180,8 @@ def _build_parser() -> argparse.ArgumentParser:
compare.add_argument("--baseline", required=True)
compare.add_argument("--tolerance", action="append", metavar="METRIC=VALUE",
help="allowed absolute drift; repeatable")
compare.add_argument("--gate-policy", metavar="FILE",
help="apply opt-in per-slice quality rules from a JSON policy")

return parser

Expand Down Expand Up @@ -256,6 +260,10 @@ def _warn_no_value(extra: Sequence[evals.Evaluator], report: evals.EvalReport) -

def _cmd_run(args) -> int:
dataset = evals.Dataset.from_jsonl(args.dataset)
policy = _eval_policy.load_policy(args.gate_policy) if args.gate_policy else None
if policy and _eval_policy.needs_baseline(policy) and not args.baseline:
raise EvalError("--gate-policy has a relative rule; pass --baseline")
baseline = _load_report(args.baseline) if args.baseline and policy else None
if args.model:
for example in dataset.examples: # --model is authoritative over per-row model
example.model = args.model
Expand Down Expand Up @@ -295,6 +303,8 @@ def _cmd_run(args) -> int:
# refetched under the same name, and a reviewer comparing two reports needs the
# report -- not a file mtime -- to say so.
"dataset_sha256": evals.file_fingerprint(args.dataset)}
if policy:
config["gate_policy"] = policy
if args.onnx:
config["onnx"] = args.onnx
if extra:
Expand Down Expand Up @@ -328,7 +338,8 @@ def _cmd_run(args) -> int:

failures = _check_thresholds(report.overall, mins, maxs)
if args.baseline:
baseline = _load_report(args.baseline)
if baseline is None:
baseline = _load_report(args.baseline)
ok, deltas = report.compare(baseline, tolerances)
_print_deltas(deltas)
if not ok:
Expand All @@ -338,6 +349,8 @@ def _cmd_run(args) -> int:
refusal = _comparability_failure(report, baseline)
if refusal:
failures.append(refusal)
if policy:
failures.extend(_eval_policy.check_policy(report, policy, baseline))

for name in sorted(report.overall):
print("%-18s %.4f" % (name, report.overall[name]))
Expand Down Expand Up @@ -368,20 +381,27 @@ def _cmd_compare(args) -> int:
# that disagreed with its baseline passed the gate while the same disagreement stated in
# `config` was refused. The baseline has always gone through `_identity_of`; this makes the
# candidate side symmetric.
policy = _eval_policy.load_policy(args.gate_policy) if args.gate_policy else None
document = _load_report(args.report)
report = evals.EvalReport(
config=evals._identity_of(document),
**{k: v for k, v in document.items() if k in ("overall", "slices", "cases")})
if policy and report.config.get("gate_policy") not in (None, policy):
print("laya-evals: warning: --gate-policy differs from the report's recorded gate_policy; "
"this verdict uses --gate-policy", file=sys.stderr)
baseline = _load_report(args.baseline)
ok, deltas = report.compare(baseline, _parse_pairs(args.tolerance))
_print_deltas(deltas)
# `compare` is the offline re-check of a report a reviewer already read, so the same
# comparability refusal has to apply here: a saved report that cannot say which experiment
# it came from must not pass a gate either.
refusal = _comparability_failure(report, baseline)
if refusal:
print("FAIL: " + refusal, file=sys.stderr)
return 0 if ok and not refusal else 1
failures = [refusal] if refusal else []
if policy:
failures.extend(_eval_policy.check_policy(report, policy, baseline))
for failure in failures:
print("FAIL: " + failure, file=sys.stderr)
return 0 if ok and not failures else 1


def main(argv: Optional[Sequence[str]] = None) -> int:
Expand Down
Loading
Loading