Repository navigation
feat(evals): evidence inspection over checkpoint and eval artifacts - #964
Conversation
…tative calibration issues
|
Merged in 0.3.29. The case this exists for is real and I reproduced both halves of it. On a fine-tune with two calibration items: And on a shipped checkpoint: Both answers are right, and the second is the one that shows the design is sound. Reading persisted #933 evidence rather than duplicating the thresholds is what makes this a view over the contract instead of a second opinion about it. If Not treating The scope section is why this merged in one pass. No readiness score, no READY verdict, no GO or NO-GO, no rollout policy, and the header saying in the output itself that this is not a deployment verdict. A command that reports evidence is useful; a command that reports a verdict would have to be right about everyone's deployment, and it would be wrong quietly. Leaving the machine-readable schema out until there is a concrete consumer is the same instinct. The torch-blocked subprocess check proving the inspection path stays weight-free and torch-free is the right way to assert that, since it is the property that makes the command usable where it is most needed. All 83 suites pass on the merged tree, with |
`laya/evidence.py` opened with a five-line note promising that the module prefers the live `MIN_TYPE_N`/`CALIB_WARN_N` when torch imports and falls back to documented values with a provenance note when it does not. #964 removed `_thresholds()`, both `_FALLBACK_*` constants and the `threshold_source` key that note described, and left the note orphaned above `_entry`, contradicting the comment that replaced it. Move the half that is still true -- both upstream modules import torch at module scope -- into the comment where the policy lives, and guard it: states derive from the persisted `issues` alone, at counts on both sides of both floors.
…t a count
`isinstance(True, int)` is True, so `{"items": true}` in a checkpoint's persisted calibration block
read as a count of one and reported PRESENT -- "fitted with no recorded issues on 1 items" -- and
`{"items": false}` read as zero and reported MISSING, the state a genuinely unfitted question type
gets. Both are type errors on an artifact this module only reads, so they belong in the same branch as
a missing or non-dict entry. A negative count still reports, because NandhaKishorM#964 deleted the floor that
judged the number and this guard is about its type.
What
Add a read-only evidence summary:
laya-evals evidence --checkpoint DIR [--report R]The command reads the selected checkpoint's persisted calibration metadata and, when provided, an existing EvalReport. It reports human-readable evidence states:
PRESENT,MISSING,INSUFFICIENT,UNKNOWN, andINCOMPARABLEonly where current upstream artifacts can justify them.Calibration status comes from persisted #933 evidence rather than duplicated thresholds.
laya_versionis not treated as checkpoint identity, and the command does not load model weights or require torch.Why
Training and eval already persist useful evidence, but it is spread across checkpoint config, calibration metadata, and EvalReport identity. Users currently have to stitch those artifacts together and know internal semantics to tell whether evidence is present, absent, weak, unknown, or deterministically conflicting.
That makes cases such as zero calibration items easy to misread as healthy. This command exposes the existing evidence contract directly without introducing a second manifest or deployment policy layer.
How verified
python -m pytest tests/test_evidence.py -q→ 21 passedpython tests/test_evals_api.py→ 48 passedpython tests/test_cli.py→ 187 passedpython tests/test_evals.py→ exit 0ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120→ cleanpython -m compileall -q laya/ tests/→ cleanThe focused suite includes same/different-
laya_versionnegative controls, deterministic identity-conflict coverage, malformed/legacy evidence behavior, and a torch-blocked subprocess check proving the inspection path stays weight-free and torch-free.Follow-ups left out