Skip to content

feat(evals): add opt-in per-slice quality gates - #755

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
xinpingzh:feat/evals-slice-gates
Oct 1, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
xinpingzh:feat/evals-slice-gates

Conversation

@xinpingzh

@xinpingzh xinpingzh commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Add an opt-in, versioned JSON policy for existing language, model, qid, and tag slices. Each rule names an exact metric, a minimum scored count, and either an absolute bound or a directional baseline drift limit.
  • Apply the same policy in laya-evals run and saved-report compare; record the run policy in the JSON report. Missing evidence, skipped cases, and unsuitable relative baselines fail closed. Existing commands without the flag keep their behavior.
  • Document the policy and add weight-free tests for masked regressions, both-side counts, missing metrics, identity, validation before checkpoint load, and both CLI paths. A separate one-line docs commit corrects the pre-existing SDK guide link that blocked the required strict docs check.

Why

The current baseline gate compares overall only. In the regression test, English improves from 90% to 95% across 180 answers while Chinese falls from 90% to 60% across 20. Overall rises from 90% to 91.5%, so an overall tolerance of 0.02 passes. A language=zh rule with max_drop: 0.05 catches the 0.30 drop. This makes a reviewed critical slice enforceable in CI without changing the default comparison.

How it was verified

  • python -m pytest tests/test_evals.py -q -k "slice_gate or slice_policy or slice_rule or invalid_policy_json" — 12 passed locally.
  • python tests/test_evals_api.py — 25 passed locally.
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 and python -m compileall -q laya/ tests/ — passed locally.
  • zensical build --strict --clean — passed locally with no warnings after the SDK link correction.
  • GitHub CI: full eval harness, Python test matrix, Windows tests, lint, and strict docs all passed on the updated branch.

The complete eval suite could not run locally because the CPU torch download was interrupted; the repository CI installed it and ran the suite successfully.

Checklist

  • Focused feature change; the separate one-line docs commit unblocks a required check
  • Based on the latest main
  • Full eval suite passes in CI
  • CLI documentation updated

Fixes #754

@xinpingzh

xinpingzh commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks @Bruce-Yii for the thorough review. I pushed 136ae2a to address the follow-ups:

  • compare --gate-policy now warns when the explicit policy differs from the report's recorded config.gate_policy. The supplied policy remains authoritative for intentional offline re-evaluation; tests cover strict-saved/lax-applied, matching, and no-flag paths.
  • Tightened the duplicate-tag count boundary test, added an unknown-dimension validation case, removed the unreachable relative-rule skip, and added gate_policy to the derived report-key contract. The docs and a test now state that ECE gate evidence requires boolean correct values.

Local verification: 98 eval tests, 25 eval API checks, Ruff, compileall, and the strict docs build. On this head, GitHub reports 24 successful checks and one skipped deployment check.

The separate SDK link fix overlaps #742. Since #742 is still open and the current strict docs build needs the correction, I will drop the overlapping commit once #742 lands.

Maintainer help needed: tests (py3.9) is still required, but .github/workflows/ci.yml runs only Python 3.10–3.13, so that status cannot be reported. Could someone with repository settings access replace this stale requirement with an active check?

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This holds up under adversarial probing, which is the part I expected a quality gate to get wrong. I threw 30 false-green vectors at it and every one fails closed with a specific message:

  • A slice regressing while the overall metric improves. Baseline en n=180 @0.90 + zh n=20 @0.90, candidate en 0.95 + zh 0.60 — overall goes 0.9000 → 0.9150. The legacy gate passes it (exit 0); with --gate-policy it fails, and the message names the numbers: candidate=0.6000 (n=20), baseline=0.9000 (n=20), max_drop=0.0500 exceeded by 0.2500. That is the whole point of the feature and it works.
  • Insufficient evidence on either side. A baseline whose aggregate still reads 0.9000 while cases holds 19 rows fails on baseline count 19 is below min_count 20 — the count is derived from the cases that actually carry a finite score, not from the stored aggregate. Removing either count guard turns a test red.
  • Skipped rows, including where the count check cannot be what fires. run --on-error skip where one case raises leaves zh at n=9; under a loose max_drop 0.9 that the numbers would happily pass, compare still exits 1 with slice gate: candidate has skipped/errored cases. This works because evals._identity_of copies the whole config rather than just the identity keys, so errored survives the compare filter.
  • Order independence. Three rules, all six permutations, one distinct verdict. The loop has no break and no state carries between rules, so there is nothing for order to change.
  • Also: slice present on one side only, unknown metric / slice value / dimension / policy version, NaN and Infinity and true-as-1 in limit fields, zero-count and empty groups, non-comparable and identity-less baselines, ece drift on a lower-is-better metric, a baseline already below the floor, and reports with missing or malformed cases.

I could not find a path through check_policy that returns an empty failure list for a violated rule. Exit codes match the repo convention (0 / 1 / 2) across 24 observed cases, and a report round-trips: config.gate_policy survives load/dump(sort_keys=True), and reloading through compare with the same file reproduces the identical exit code and message. cases is what makes that work, so it surviving serialization is load-bearing.

Corroborating this independently: 11 of 19 mutations of the new code go red, including both load-bearing wirings (run not applying the policy, compare not applying the policy), the pre-checkpoint validation, and the identity guard. The no-flag path is byte-identical to main across 11 invocations — plain run, run with baseline, run with thresholds, a failing run, markdown output, compare both ways, validate, two usage errors, and both report JSONs. All 24 checks on this head are green, including docs (strict build).

Not blockers, but four things I would fix

  1. A report records the policy it ran under, and compare never reads it back. Run with {min: 0.99} on a regressed candidate exits 1 and stamps that strict policy into the report. Then compare report.json --gate-policy {min: 0.0} exits 0, with nothing indicating the artifact was produced under a stricter gate. The applied verdict is still computed from the policy named on the command line, so this is a reporting gap rather than a wrong answer — but evals_cli.py:223-235 argues that "a gate that cannot say which experiment produced a number cannot support a promotion decision", and this is the remaining hole in that. One if comparing config.gate_policy against the applied policy would close it.
  2. test_slice_gate_counts_each_tagged_answer_once_and_accepts_boundary does not test the first half of its name. The assertion is min_count: 200 over 200 cases; inflating the count 3× still satisfies >= 200. _count uses tag membership rather than iteration and is correct today, but nothing would catch a regression.
  3. The slice-dimension whitelist in load_policy:48 is untested. The malformed-policy cases cover two-limits, two-dimensions, min_count: 0 and negative max_drop, never an unknown dimension name.
  4. _eval_policy.py:150-151, if base_report is None: continue, is unreachable — any relative rule sets needs_baseline, which returns at :122-124 first. It is also the only place in the rule loop that skips a rule without recording a failure, so it is fail-open-shaped code that is currently harmless. Deleting it removes the chance of a future refactor turning it into a real bypass.

On the bundled docs fix

The docs/typescript-sdk.md hunk here is byte-equivalent to #742's: same removed line, same added URL, same result. Once #742 lands, your hunk becomes a no-op; if #752's rewrap wins instead, your one-line version conflicts with it. Worth dropping commit 5656848 and rebasing — that is all it is, and the Docs workflow runs for laya/ changes either way, so bundling it does not buy you a green check any sooner.

Two smaller notes: config.gate_policy is documented in docs/evals.md:176 but is not in the derived key list at tests/test_evals.py:943-948, whose stated contract is that it is derived from the code and therefore cannot drift. And _count requires isinstance(correct, bool) for ece while evals.py:488 accepts any non-None, so a third-party report storing correct: 1/0 as ints fails every ece rule on min_count — fail-closed and consistent with "missing evidence never passes", just undocumented.

Not verified: I ran no real checkpoint or Router-backed evaluation, so a real router's model slice values interacting with a policy is untested end to end. mergeable_state=blocked here is a stale tests (py3.9) required check — ci.yml's matrix is 3.10–3.13 and no run on this SHA produces that job. That is a repo-settings matter, not a change to make in this PR, and your note asking about it is the right call.

@Bruce-Yii Bruce-Yii left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the current head after your follow-ups. The policy-mismatch path now behaves as intended: a saved report that records one gate policy and is compared with another produces a warning, while reports with no recorded policy remain intentionally silent. The duplicate-tag boundary, unknown-dimension validation, unreachable skip, and report-key follow-ups are also addressed.

I reran the changed behavior and did not find a new false-green path in the areas touched by this update. APPROVE on 136ae2a54796ece76eadc917434f9eff9e305388.

The overlapping SDK-link hunk can still drop once #742 lands; I do not consider that a blocker here.

@NandhaKishorM
NandhaKishorM merged commit a064a62 into NandhaKishorM:main Oct 1, 2026
25 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged for 0.3.23.

The adversarial pass on this is what earned it: a slice regressing while the overall metric improves is the exact false green a quality gate exists to catch, and failing closed on missing evidence, skipped cases and unsuitable relative baselines is the part most implementations get wrong. The policy-mismatch warning on compare --gate-policy was the right follow-up to take rather than defer.

The overlapping SDK-link hunk dropped out cleanly, since #742 came in via #684 in the same pass.

Verified on the integrated branch: tests/test_evals.py and the evals API and ONNX suites pass, full test lane green.

Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature]: opt-in per-slice regression gates for laya-evals

3 participants