Repository navigation
Conversation
A high model score does not establish a low automation error rate. Require per-language error and coverage bounds with multiple-comparison correction, then reject reused groups when testing the selected policy on independent evidence.
This was referenced Sep 22, 2026
alpha912
marked this pull request as ready for review
September 22, 2026 13:54
Owner
|
Thank you, requiring independent evidence before a threshold is trusted is the right standard. As on #151: this is one of six related PRs, so could you pick the one you most want reviewed first? Then we'll go one at a time. |
alpha912
marked this pull request as draft
September 23, 2026 17:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add opt-in offline selection and independent testing of categorical decision thresholds. A high probability, an observed zero-error slice or the best of many tried thresholds is insufficient evidence of a low automation error rate. This utility requires per-language error and coverage bounds and rejects reuse of selection-set examples during final testing.
Changes
Testing
Additional validation on the minimum supported Python version: all 16 risk-gate tests pass on Windows with Python 3.10.11, CPU Torch 2.14.0, Transformers 5.17.0 and NumPy 2.2.6. The isolated test run used synthetic/local fixtures with Hub access disabled; it establishes compatibility, not model quality.
All checks ran locally on synthetic predictions; no pretrained checkpoint download or model training.
Notes for reviewer
This is a standalone part of a broader domain-adaptation pipeline and is independent of the data contract in #151. It complements calibration fitting in #19; it does not implement another fitter or alter model inference.
The statistical interpretation requires independent representatives from a fixed population, a model/calibration frozen before selection, a predeclared threshold grid and a single untouched final evaluation. Reports must later be bound to the actual checkpoint, question order, calibration and data manifest in a trusted bundle. Hashes provide consistency, not authentication or protection against somebody replacing all inputs.
These tests establish statistical/configuration behavior only. They do not show that Laya meets the default 5% error / 50% coverage targets, improves accuracy, or remains reliable under distribution shift. Runtime review outcomes and deployment-bundle verification remain separate integration work.