Skip to content

feat(evals): evidence inspection over checkpoint and eval artifacts - #964

Merged
NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
Bruce-Yii:feat/evidence-layer-readiness
Oct 7, 2026
Merged

NandhaKishorM merged 5 commits into
NandhaKishorM:mainfrom
Bruce-Yii:feat/evidence-layer-readiness

Conversation

@Bruce-Yii

Copy link
Copy Markdown
Contributor

What

Add a read-only evidence summary:

laya-evals evidence --checkpoint DIR [--report R]

The command reads the selected checkpoint's persisted calibration metadata and, when provided, an existing EvalReport. It reports human-readable evidence states: PRESENT, MISSING, INSUFFICIENT, UNKNOWN, and INCOMPARABLE only where current upstream artifacts can justify them.

Calibration status comes from persisted #933 evidence rather than duplicated thresholds. laya_version is not treated as checkpoint identity, and the command does not load model weights or require torch.

Why

Training and eval already persist useful evidence, but it is spread across checkpoint config, calibration metadata, and EvalReport identity. Users currently have to stitch those artifacts together and know internal semantics to tell whether evidence is present, absent, weak, unknown, or deterministically conflicting.

That makes cases such as zero calibration items easy to misread as healthy. This command exposes the existing evidence contract directly without introducing a second manifest or deployment policy layer.

How verified

  • python -m pytest tests/test_evidence.py -q → 21 passed
  • python tests/test_evals_api.py → 48 passed
  • python tests/test_cli.py → 187 passed
  • python tests/test_evals.py → exit 0
  • ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 → clean
  • python -m compileall -q laya/ tests/ → clean

The focused suite includes same/different-laya_version negative controls, deterministic identity-conflict coverage, malformed/legacy evidence behavior, and a torch-blocked subprocess check proving the inspection path stays weight-free and torch-free.

Follow-ups left out

@NandhaKishorM
NandhaKishorM merged commit f98d571 into NandhaKishorM:main Oct 7, 2026
32 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.3.29.

The case this exists for is real and I reproduced both halves of it. On a fine-tune with two calibration items:

evidence summary for /tmp/t967/ft
(reports evidence and limitations found in existing artifacts; this is not a deployment verdict)
calibration: INSUFFICIENT
  choice    INSUFFICIENT -- fitted with recorded limitations: not fitted: 2 calibration items,
                            fewer than 10, so the temperature stays 1.0
  score     MISSING -- question type present but zero calibration items
  noul      MISSING -- question type present but zero calibration items

And on a shipped checkpoint:

calibration: UNKNOWN
  training metadata present but no calibration block; checkpoint predates #933

Both answers are right, and the second is the one that shows the design is sound. UNKNOWN rather than MISSING for a checkpoint that predates the evidence format is the distinction that stops this from becoming a tool people learn to ignore, and distinguishing "type present but zero items" from "fitted with recorded limitations" is the distinction the original problem statement is about. Zero calibration items reading as healthy was the thing to fix.

Reading persisted #933 evidence rather than duplicating the thresholds is what makes this a view over the contract instead of a second opinion about it. If CALIB_WARN_N moves, this command moves with it.

Not treating laya_version as checkpoint identity is a small decision with a large consequence, and I am glad it came with same-version and different-version negative controls rather than as an assertion.

The scope section is why this merged in one pass. No readiness score, no READY verdict, no GO or NO-GO, no rollout policy, and the header saying in the output itself that this is not a deployment verdict. A command that reports evidence is useful; a command that reports a verdict would have to be right about everyone's deployment, and it would be wrong quietly. Leaving the machine-readable schema out until there is a concrete consumer is the same instinct.

The torch-blocked subprocess check proving the inspection path stays weight-free and torch-free is the right way to assert that, since it is the property that makes the command usable where it is most needed.

All 83 suites pass on the merged tree, with tests/test_evidence.py in the shared gate.

NandhaKishorM pushed a commit that referenced this pull request Oct 11, 2026
`laya/evidence.py` opened with a five-line note promising that the module prefers the live
`MIN_TYPE_N`/`CALIB_WARN_N` when torch imports and falls back to documented values with a
provenance note when it does not. #964 removed `_thresholds()`, both `_FALLBACK_*` constants and
the `threshold_source` key that note described, and left the note orphaned above `_entry`,
contradicting the comment that replaced it. Move the half that is still true -- both upstream
modules import torch at module scope -- into the comment where the policy lives, and guard it:
states derive from the persisted `issues` alone, at counts on both sides of both floors.
joyanes97 pushed a commit to joyanes97/laya that referenced this pull request Oct 17, 2026
…t a count

`isinstance(True, int)` is True, so `{"items": true}` in a checkpoint's persisted calibration block
read as a count of one and reported PRESENT -- "fitted with no recorded issues on 1 items" -- and
`{"items": false}` read as zero and reported MISSING, the state a genuinely unfitted question type
gets. Both are type errors on an artifact this module only reads, so they belong in the same branch as
a missing or non-dict entry. A negative count still reports, because NandhaKishorM#964 deleted the floor that
judged the number and this guard is about its type.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants