Repository navigation
Conversation
7 tasks
Contributor
Author
|
All 23 reported checks have passed. GitHub still lists a required |
xinpingzh
force-pushed
the
research/massive-en-release-evidence
branch
from
October 2, 2026 17:29
9932bfa to
62269c8
Compare
This was referenced Oct 10, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
release_candidatemode to the existing Evals workflow. It requires full checkpoint and dataset commit SHAs and a complete 100-case MASSIVE English CPU report.layaimport from a different checkout. Invalid candidate results return nonzero and retain diagnostic JSON when it can be written.research/results/massive_en_cpu_release_20261001.json, document its scope inBENCHMARKS.md, and check the published cells against that JSON.Why
#285 asks for a reproducible refresh before release. A numeric baseline alone cannot establish which inputs and checkpoint produced a rerun. This extends the existing research harness and gate for the English CPU slice already claimed there.
The weekly and published-release paths retain their existing invocation. Metric definitions, sampling and model scoring are unchanged. The candidate validator uses the harness's own metric functions rather than introducing another implementation.
How it was verified
The committed report was measured on laya 0.3.23 at
9bf2b51. I reran the same 100 cases on CPU after rebasing onto 0.3.24, with checkpoint55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851and dataset940fd47a81eaa7f2cc7b129674d945d618ac38c2. The ordered input fingerprint is unchanged (a6363b6c…e3ddf7), no decision changed, and accuracy 0.8200, macro F1 0.7876, ECE 0.1382, mean confidence 0.9582 and accuracy at 50% coverage 0.9800 reproduce.python research/eval/test_laya_eval.py— 158 passed, including missing/non-finite metrics, invalid cases, probability counterexamples, ECE/coverage precision boundaries, failed-candidate JSON, source identity, and unavailable dataset pins.python -m research.eval.test_metamorphic— 41 tests passed.python research/eval/test_presentation_checks.py— 219 passed.python -m pytest tests/test_evals.py tests/test_evals_shortlist.py -q— 118 passed on Windows, using a fresh temporary directory.python tests/test_evals_api.py— 40 passed;python tests/test_doc_tables.py— 126 passed.python research/evals/check_regression.py report.json research/results/eval_english_51_languages.json research/evals/thresholds.json— passed.laya/and the changed Python files,compileall, andzensical build --strict --clean— passed, with nogriffe:diagnostics.Independent checks also exercised 3,010 valid probability distributions without a false rejection, and verified that the pinned raw English data and its prepared cache have identical rows and ordering.
Scope
Refs #285. This does not complete the broader T4, latency or 51-language refresh. The historical numeric baseline has no input fingerprint, so matching its numbers does not establish identical inputs. The committed original measurement is preserved; the new rerun verifies the final code. Thresholds are unchanged because one CPU rerun does not establish a new noise tolerance.