Repository navigation
docs(train): say which per-type keys calibration_report actually returns - #1168
Open
aashish254 wants to merge 1 commit into
Open
aashish254 wants to merge 1 commit into
aashish254 wants to merge 1 commit into
Conversation
`calibration_report`'s docstring listed `accuracy` and `mean_confidence` among
what "each type gets", but a type with no calibration items carries neither:
the two are only computed inside `if pairs:`, and
`test_clamped_and_healthy_fits` already asserts `assertNotIn("accuracy", ...)`.
Reading `report["noul"]["accuracy"]` on a run whose data had no noul items is a
KeyError, which the reference page invited.
Reword the two sentences to state the real key surface, and pin it with
`test_metric_keys_follow_the_data`, which asserts the sorted keys for a present
type and for both absent ones.
This was referenced Oct 15, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
laya/train.py::calibration_report's docstring said every question type comes back withaccuracyandmean_confidence. It does not give those keys to a type that has no calibrationitems, and the sentence is the only thing in the tree that claimed otherwise.
laya/train.py+6/-4 -- two sentences of that docstring, no code touched.tests/test_train.py+16 --test_metric_keys_follow_the_data, which pins the key surface forall three types instead of one key at a time.
No signature, no return shape, no public name changes.
Why
The docstring read:
but both metrics are computed only inside
if pairs:(laya/train.py:782-791on this branch), soan absent type has three keys, not five. The closing sentence named the absent case -- "A type with
no calibration items at all has no issues; it was not in the data" -- and mentioned only
issues,which is where the misreading starts: a reader takes the first sentence as the schema and the last
as a note about the warning list.
The code is not the bug here.
tests/test_train.py:1038has assertedself.assertNotIn("accuracy", report["noul"])for this exact case, under the comment "a type absentfrom the data is reported, but not as a problem". Code and test agree; the prose is the outlier, and
the prose is what ships --
docs/reference/train.md:21publishes it verbatim through::: laya.train.calibration_report.Measured on current
main(c73908d9, release 0.4.6;laya/train.pyis byte-identical to the 0.4.4and 0.4.5 blobs --
git hash-objecton a tree extracted fromc73908d9andgit rev-parse c73908d9:laya/train.pyandgit rev-parse 682ac671:laya/train.pyall give442ed27c). The runloaded that tree's own copy: the same process printed
import laya.train; laya.train.__file__, whichresolved to the extracted tree's
laya/train.pyrather than any branch's:Input was
calibration_report(recs, [1.0, 1.3, 1.0])with 60scoreitems and none for the othertwo types, so nothing about the checkpoint or a model is involved.
Nothing in-tree reads those keys: the only caller is
finetuneatlaya/train.py:1141, and itwalks
entry["issues"]. The exposure is to anyone following the reference page, which is why thisis worth a doc fix rather than a shrug.
calibration_reportis not inlaya.__all__, so thereference page is where a caller meets it.
How it was verified
CalibrationReportTests3 -> 4 passed (the same-krun on the extractedc73908d9tree gives3 passed, 55 deselected);tests/test_train.py59 passed both as a script (Ran 59 tests ... OK) and under pytest (59 passed, 28 warnings, 10 subtests passed), where the same tree before thischange gives
Ran 58 testsand58 passed, 28 warnings, 10 subtests passed. Full gateall 87 suites passed, rc 0; its pytest leg prints500 passed, 1 skippedon both arms (16.17s onthis branch, 16.69s on the extracted tree), which is worth stating so nobody reads it as this test
going unnoticed --
tests/test_train.pyis a script suite inscripts/test_suites.py, and the 500counts the separate
PYTEST_SUITESinvocation, so the count this change moves is the script's own58 -> 59. ruff rc 0, compileall rc 0, docs strict build
No issues foundrc 0 (1.72s).The new test asserts the sorted key list for the type that has items and for both types that have
none, then reads each metric off an absent type inside
assertRaises(KeyError)-- so it pins theconsequence the docstring used to hide, not just the absence.
Rendered page (
site/reference/train/index.html) after the build, read with tags stripped andwhitespace collapsed, so identifiers appear without their code spans -- verbatim:
and the closing sentence reads "A type with no calibration items at all has no issues and no metric
keys; it was not in the data, so there is nothing to score it on." The superseded string "Each type
gets items, temperature, accuracy and mean_confidence" is at zero occurrences in that stripped text.
Mutation check -- two faithful edits to the
if pairs:block,pytest tests/test_train.py -k CalibrationReportafter each, file restored withgit checkout --between runs (git statuscleanbefore each run, the branch committed first):
accuracy/mean_confidencemoved out ofif pairs:so every type carries them (right, conf = 0, 0.0hoisted above the guard, the two stores dedented, denominatorslen(pairs) or 1so the mutant crashes on nothing)2 failed, 2 passed--test_metric_keys_follow_the_data(AssertionError: Lists differ: ['accuracy', 'issues', 'items', 'mean_confidence', 'temperature'] != ['issues', 'items', 'temperature']) and the pre-existingtest_clamped_and_healthy_fits(AssertionError: 'accuracy' unexpectedly found in {'items': 0, 'temperature': 1.0, 'issues': [], 'accuracy': 0.0, 'mean_confidence': 0.0})3 failed, 1 passed--test_metric_keys_follow_the_data(AssertionError: Lists differ: ['issues', 'items', 'temperature'] != ['accuracy', 'issues', 'items', 'mean_confidence', 'temperature']),test_each_kind_of_weak_fit_is_named,test_clamped_and_healthy_fits4 passedThe pin has teeth in both directions, one assertion per direction: the first mutant is caught by the
absent-type check (
['accuracy', 'issues', 'items', 'mean_confidence', 'temperature'] != ['issues', 'items', 'temperature']), the second by the with-items check (['issues', 'items', 'temperature'] != ['accuracy', 'issues', 'items', 'mean_confidence', 'temperature']).Deconfliction: 44 open PRs as at 2026-10-12T10:05Z --
mainhas taken releases 0.4.5 (08:53Z) and0.4.6 (09:29Z) during this session, so this was re-derived from the current file lists and not carried
over. Three of them edit
laya/train.pyand none reaches this docstring --#921 at
@@ -668,6 +668,12 @@and@@ -696,7 +702,7 @@(old lines end at 702); #982 from@@ -61,6 +61,7 @@to@@ -1314,6 +1475,8 @@, whose nearest hunk below that (@@ -731,6 +844,31 @@)ends at old line 736 while this edit spans old 756-772, and whose next hunk starts at old 1012 inside
finetune; #1129 (mine) at@@ -35,6 +35,7 @@,@@ -171,9 +172,14 @@and@@ -320,10 +326,14 @@. Intests/test_train.pythe foreign hunks end at old line 557 (#982's@@ -552,6 +685,23 @@) and old 354(#1129's
@@ -349,6 +403,164 @@), whileCalibrationReportTestsstarts at 1008 and this test isinserted at
@@ -1039,6 +1039,22 @@. Overlap was read off every open PR's file list and patch hunks,not off titles.
Not fixed here, worth knowing
The bullet this change leaves untouched, "it came back unchanged at 1.0 ... which usually means the
items gave the fit nothing to correct", is a diagnosis the same function's own numbers contradict. The
message prints accuracy and mean confidence next to it, and on
c73908d9(calibration_reportcalleddirectly with a temperature of 1.0, 60
scoreitems whose argmax is the target in every row):Nothing is "nothing to correct" about a pool the model gets every one right on: what the sentence
summarises is a pair, accuracy and mean confidence, and in both arms they differ (1.00 against 0.811,
1.00 against 0.345), which is the fit's whole subject. These numbers come from a hand-passed
temperature of 1.0, so they witness the wording only -- not why any particular fit landed on 1.0,
which is the mechanism fix(calibrate): keep a ragged pool's fit from vanishing into 1.0 #1158 (merged) settled. Different sentence, different bullet, so it stays out
of this change.
Checklist
main(c73908d9, release 0.4.6)