Skip to content

feat(risk): add independent selective decision gates - #152

Closed
alpha912 wants to merge 1 commit into
NandhaKishorM:mainfrom
MycelAI:alpha912/fix/decision-risk-gates
Closed

alpha912 wants to merge 1 commit into
NandhaKishorM:mainfrom
MycelAI:alpha912/fix/decision-risk-gates

Conversation

@alpha912

@alpha912 alpha912 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add opt-in offline selection and independent testing of categorical decision thresholds. A high probability, an observed zero-error slice or the best of many tried thresholds is insufficient evidence of a low automation error rate. This utility requires per-language error and coverage bounds and rejects reuse of selection-set examples during final testing.

Changes

  • Select from a predeclared grid using maximum class probability, keeping the existing entropy-based SDK confidence and prediction API unchanged.
  • Compute one-sided Clopper-Pearson bounds by binomial CDF inversion, without adding a dependency.
  • Apply Bonferroni correction across error/coverage bounds, languages and candidate thresholds. Require both a bounded selected error rate and minimum coverage.
  • Require every declared language to qualify. Empty slices, too few observations, overconfident mistakes and deferring everything cannot pass.
  • Reject duplicate group representatives within a language and reject selection/test identity or family overlap, including across languages.
  • Record protocol, sample hashes, counts and uncertainty, and document the sampling assumptions and limits.

Testing

Additional validation on the minimum supported Python version: all 16 risk-gate tests pass on Windows with Python 3.10.11, CPU Torch 2.14.0, Transformers 5.17.0 and NumPy 2.2.6. The isolated test run used synthetic/local fixtures with Hub access disabled; it establishes compatibility, not model quality.

All checks ran locally on synthetic predictions; no pretrained checkpoint download or model training.

  • All 16 new tests pass on Windows and Linux with Python 3.12.11; source/test SHA-256 hashes match across hosts.
  • Check a published exact interval, closed-form boundaries, exact small-sample coverage by enumeration, and larger/tiny-tail results using independent 60-digit Decimal probability recurrence.
  • Exercise nontrivial threshold selection, multiple-comparison correction, language-specific failure, invalid probabilities, duplicate evidence, JSON round trips and independent-test reuse/failure.
  • Existing criteria (34) and packaging (9) checks pass on Windows. Byte compilation passes on both platforms; repository-selected Ruff rules and git whitespace checks pass.

Notes for reviewer

This is a standalone part of a broader domain-adaptation pipeline and is independent of the data contract in #151. It complements calibration fitting in #19; it does not implement another fitter or alter model inference.

The statistical interpretation requires independent representatives from a fixed population, a model/calibration frozen before selection, a predeclared threshold grid and a single untouched final evaluation. Reports must later be bound to the actual checkpoint, question order, calibration and data manifest in a trusted bundle. Hashes provide consistency, not authentication or protection against somebody replacing all inputs.

These tests establish statistical/configuration behavior only. They do not show that Laya meets the default 5% error / 50% coverage targets, improves accuracy, or remains reliable under distribution shift. Runtime review outcomes and deployment-bundle verification remain separate integration work.

A high model score does not establish a low automation error rate. Require per-language error and coverage bounds with multiple-comparison correction, then reject reused groups when testing the selected policy on independent evidence.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you, requiring independent evidence before a threshold is trusted is the right standard. As on #151: this is one of six related PRs, so could you pick the one you most want reviewed first? Then we'll go one at a time.

@alpha912
alpha912 marked this pull request as draft September 23, 2026 17:29
@alpha912 alpha912 closed this Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants