Skip to content

docs(train): say which per-type keys calibration_report actually returns - #1168

Open
aashish254 wants to merge 1 commit into
NandhaKishorM:mainfrom
aashish254:docs-calibration-report-keys-046
Open

aashish254 wants to merge 1 commit into
NandhaKishorM:mainfrom
aashish254:docs-calibration-report-keys-046

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya/train.py::calibration_report's docstring said every question type comes back with
accuracy and mean_confidence. It does not give those keys to a type that has no calibration
items, and the sentence is the only thing in the tree that claimed otherwise.

  • laya/train.py +6/-4 -- two sentences of that docstring, no code touched.
  • tests/test_train.py +16 -- test_metric_keys_follow_the_data, which pins the key surface for
    all three types instead of one key at a time.

No signature, no return shape, no public name changes.

Why

The docstring read:

Each type gets items, temperature, accuracy and mean_confidence (top probability at the
fitted temperature) on the calibration items, and a list of issues, empty when nothing looks
wrong:

but both metrics are computed only inside if pairs: (laya/train.py:782-791 on this branch), so
an absent type has three keys, not five. The closing sentence named the absent case -- "A type with
no calibration items at all has no issues; it was not in the data" -- and mentioned only issues,
which is where the misreading starts: a reader takes the first sentence as the schema and the last
as a note about the warning list.

The code is not the bug here. tests/test_train.py:1038 has asserted
self.assertNotIn("accuracy", report["noul"]) for this exact case, under the comment "a type absent
from the data is reported, but not as a problem". Code and test agree; the prose is the outlier, and
the prose is what ships -- docs/reference/train.md:21 publishes it verbatim through
::: laya.train.calibration_report.

Measured on current main (c73908d9, release 0.4.6; laya/train.py is byte-identical to the 0.4.4
and 0.4.5 blobs -- git hash-object on a tree extracted from c73908d9 and git rev-parse c73908d9:laya/train.py and git rev-parse 682ac671:laya/train.py all give 442ed27c). The run
loaded that tree's own copy: the same process printed import laya.train; laya.train.__file__, which
resolved to the extracted tree's laya/train.py rather than any branch's:

choice items=0   keys=['issues', 'items', 'temperature']
score  items=60  keys=['accuracy', 'issues', 'items', 'mean_confidence', 'temperature']
noul   items=0   keys=['issues', 'items', 'temperature']
KeyError on the documented key: KeyError('accuracy')

Input was calibration_report(recs, [1.0, 1.3, 1.0]) with 60 score items and none for the other
two types, so nothing about the checkpoint or a model is involved.

Nothing in-tree reads those keys: the only caller is finetune at laya/train.py:1141, and it
walks entry["issues"]. The exposure is to anyone following the reference page, which is why this
is worth a doc fix rather than a shrug. calibration_report is not in laya.__all__, so the
reference page is where a caller meets it.

How it was verified

python -m pytest tests/test_train.py -q -k CalibrationReport
python tests/test_train.py
python -m pytest tests/test_train.py -q
python scripts/test_suites.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
zensical build --strict --clean

CalibrationReportTests 3 -> 4 passed (the same -k run on the extracted c73908d9 tree gives
3 passed, 55 deselected); tests/test_train.py 59 passed both as a script (Ran 59 tests ... OK) and under pytest (59 passed, 28 warnings, 10 subtests passed), where the same tree before this
change gives Ran 58 tests and 58 passed, 28 warnings, 10 subtests passed. Full gate
all 87 suites passed, rc 0; its pytest leg prints 500 passed, 1 skipped on both arms (16.17s on
this branch, 16.69s on the extracted tree), which is worth stating so nobody reads it as this test
going unnoticed -- tests/test_train.py is a script suite in scripts/test_suites.py, and the 500
counts the separate PYTEST_SUITES invocation, so the count this change moves is the script's own
58 -> 59. ruff rc 0, compileall rc 0, docs strict build No issues found rc 0 (1.72s).

The new test asserts the sorted key list for the type that has items and for both types that have
none, then reads each metric off an absent type inside assertRaises(KeyError) -- so it pins the
consequence the docstring used to hide, not just the absence.

Rendered page (site/reference/train/index.html) after the build, read with tags stripped and
whitespace collapsed, so identifiers appear without their code spans -- verbatim:

Each type gets items , temperature and a list of issues , empty when nothing looks wrong, and a
type that has items also gets accuracy and mean_confidence (top probability at the fitted
temperature) on them.

and the closing sentence reads "A type with no calibration items at all has no issues and no metric
keys; it was not in the data, so there is nothing to score it on." The superseded string "Each type
gets items, temperature, accuracy and mean_confidence" is at zero occurrences in that stripped text.

Mutation check -- two faithful edits to the if pairs: block, pytest tests/test_train.py -k CalibrationReport after each, file restored with git checkout -- between runs (git status clean
before each run, the branch committed first):

mutant killed by
accuracy / mean_confidence moved out of if pairs: so every type carries them (right, conf = 0, 0.0 hoisted above the guard, the two stores dedented, denominators len(pairs) or 1 so the mutant crashes on nothing) 2 failed, 2 passed -- test_metric_keys_follow_the_data (AssertionError: Lists differ: ['accuracy', 'issues', 'items', 'mean_confidence', 'temperature'] != ['issues', 'items', 'temperature']) and the pre-existing test_clamped_and_healthy_fits (AssertionError: 'accuracy' unexpectedly found in {'items': 0, 'temperature': 1.0, 'issues': [], 'accuracy': 0.0, 'mean_confidence': 0.0})
the two metrics never stored, even for a type with items 3 failed, 1 passed -- test_metric_keys_follow_the_data (AssertionError: Lists differ: ['issues', 'items', 'temperature'] != ['accuracy', 'issues', 'items', 'mean_confidence', 'temperature']), test_each_kind_of_weak_fit_is_named, test_clamped_and_healthy_fits
control: no edit 4 passed

The pin has teeth in both directions, one assertion per direction: the first mutant is caught by the
absent-type check (['accuracy', 'issues', 'items', 'mean_confidence', 'temperature'] != ['issues', 'items', 'temperature']), the second by the with-items check (['issues', 'items', 'temperature'] != ['accuracy', 'issues', 'items', 'mean_confidence', 'temperature']).

Deconfliction: 44 open PRs as at 2026-10-12T10:05Z -- main has taken releases 0.4.5 (08:53Z) and
0.4.6 (09:29Z) during this session, so this was re-derived from the current file lists and not carried
over. Three of them edit laya/train.py and none reaches this docstring --
#921 at @@ -668,6 +668,12 @@ and @@ -696,7 +702,7 @@ (old lines end at 702); #982 from
@@ -61,6 +61,7 @@ to @@ -1314,6 +1475,8 @@, whose nearest hunk below that (@@ -731,6 +844,31 @@)
ends at old line 736 while this edit spans old 756-772, and whose next hunk starts at old 1012 inside
finetune; #1129 (mine) at @@ -35,6 +35,7 @@, @@ -171,9 +172,14 @@ and @@ -320,10 +326,14 @@. In
tests/test_train.py the foreign hunks end at old line 557 (#982's @@ -552,6 +685,23 @@) and old 354
(#1129's @@ -349,6 +403,164 @@), while CalibrationReportTests starts at 1008 and this test is
inserted at @@ -1039,6 +1039,22 @@. Overlap was read off every open PR's file list and patch hunks,
not off titles.

Not fixed here, worth knowing

  • The bullet this change leaves untouched, "it came back unchanged at 1.0 ... which usually means the
    items gave the fit nothing to correct", is a diagnosis the same function's own numbers contradict. The
    message prints accuracy and mean confidence next to it, and on c73908d9 (calibration_report called
    directly with a temperature of 1.0, 60 score items whose argmax is the target in every row):

    logits [2.2, 0.05, 0.05] -> "the fit came back unchanged at 1.0 (accuracy 1.00, mean confidence 0.811
        on the calibration items), which usually means they gave it nothing to correct; this type's
        confidences are uncalibrated"
    logits [0.35, 0.3, 0.3]  -> "... (accuracy 1.00, mean confidence 0.345 on the calibration items) ..."
    

    Nothing is "nothing to correct" about a pool the model gets every one right on: what the sentence
    summarises is a pair, accuracy and mean confidence, and in both arms they differ (1.00 against 0.811,
    1.00 against 0.345), which is the fit's whole subject. These numbers come from a hand-passed
    temperature of 1.0, so they witness the wording only -- not why any particular fit landed on 1.0,
    which is the mechanism fix(calibrate): keep a ragged pool's fit from vanishing into 1.0 #1158 (merged) settled. Different sentence, different bullet, so it stays out
    of this change.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (c73908d9, release 0.4.6)
  • Tests pass locally
  • Docs or examples updated when the public API changed

`calibration_report`'s docstring listed `accuracy` and `mean_confidence` among
what "each type gets", but a type with no calibration items carries neither:
the two are only computed inside `if pairs:`, and
`test_clamped_and_healthy_fits` already asserts `assertNotIn("accuracy", ...)`.
Reading `report["noul"]["accuracy"]` on a run whose data had no noul items is a
KeyError, which the reference page invited.

Reword the two sentences to state the real key surface, and pin it with
`test_metric_keys_follow_the_data`, which asserts the sorted keys for a present
type and for both absent ones.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant