Skip to content

feat(eval): make MASSIVE English release checks reproducible - #838

Open
xinpingzh wants to merge 4 commits into
NandhaKishorM:mainfrom
xinpingzh:research/massive-en-release-evidence
Open

xinpingzh wants to merge 4 commits into
NandhaKishorM:mainfrom
xinpingzh:research/massive-en-release-evidence

Conversation

@xinpingzh

@xinpingzh xinpingzh commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

What

  • Add an opt-in release_candidate mode to the existing Evals workflow. It requires full checkpoint and dataset commit SHAs and a complete 100-case MASSIVE English CPU report.
  • Record the loaded checkpoint, dataset and source revisions, runtime versions, ordered option keys and criteria descriptions, an input fingerprint, and every decision. A pinned dataset is read from that revision's gzip file so an unavailable pin cannot silently use the latest prepared cache.
  • Validate the saved decisions before accepting candidate evidence: required fields, finite probabilities, label/index consistency, the probability constraints the saved top/gold values can establish, and all five recomputed metrics. Preserve floating-point precision for ECE bins and coverage ordering, and reject a laya import from a different checkout. Invalid candidate results return nonzero and retain diagnostic JSON when it can be written.
  • Commit the measured English CPU report under research/results/massive_en_cpu_release_20261001.json, document its scope in BENCHMARKS.md, and check the published cells against that JSON.

Why

#285 asks for a reproducible refresh before release. A numeric baseline alone cannot establish which inputs and checkpoint produced a rerun. This extends the existing research harness and gate for the English CPU slice already claimed there.

The weekly and published-release paths retain their existing invocation. Metric definitions, sampling and model scoring are unchanged. The candidate validator uses the harness's own metric functions rather than introducing another implementation.

How it was verified

The committed report was measured on laya 0.3.23 at 9bf2b51. I reran the same 100 cases on CPU after rebasing onto 0.3.24, with checkpoint 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 and dataset 940fd47a81eaa7f2cc7b129674d945d618ac38c2. The ordered input fingerprint is unchanged (a6363b6c…e3ddf7), no decision changed, and accuracy 0.8200, macro F1 0.7876, ECE 0.1382, mean confidence 0.9582 and accuracy at 50% coverage 0.9800 reproduce.

python research/eval/laya_eval.py --model convaiinnovations/laya \
  --langs en --device cpu --per-lang 100 --n-opts 20 \
  --model-revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851 \
  --dataset-revision 940fd47a81eaa7f2cc7b129674d945d618ac38c2 \
  --require-complete --out report.json
  • python research/eval/test_laya_eval.py — 158 passed, including missing/non-finite metrics, invalid cases, probability counterexamples, ECE/coverage precision boundaries, failed-candidate JSON, source identity, and unavailable dataset pins.
  • python -m research.eval.test_metamorphic — 41 tests passed.
  • python research/eval/test_presentation_checks.py — 219 passed.
  • python -m pytest tests/test_evals.py tests/test_evals_shortlist.py -q — 118 passed on Windows, using a fresh temporary directory.
  • python tests/test_evals_api.py — 40 passed; python tests/test_doc_tables.py — 126 passed.
  • The documented router, criteria, hooks and hooks API script suites passed.
  • python research/evals/check_regression.py report.json research/results/eval_english_51_languages.json research/evals/thresholds.json — passed.
  • Required ruff checks on laya/ and the changed Python files, compileall, and zensical build --strict --clean — passed, with no griffe: diagnostics.

Independent checks also exercised 3,010 valid probability distributions without a false rejection, and verified that the pinned raw English data and its prepared cache have identical rows and ordering.

Scope

Refs #285. This does not complete the broader T4, latency or 51-language refresh. The historical numeric baseline has no input fingerprint, so matching its numbers does not establish identical inputs. The committed original measurement is preserved; the new rerun verifies the final code. Thresholds are unchanged because one CPU rerun does not establish a new noise tolerance.

@xinpingzh

Copy link
Copy Markdown
Contributor Author

All 23 reported checks have passed. GitHub still lists a required tests (py3.9) check as “Expected — Waiting for status to be reported.” The current CI matrix runs Python 3.10–3.13, consistent with requires-python = ">=3.10" in pyproject.toml, so this job is not scheduled for this PR. The required-check setting may need to be updated before this can merge.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant