Repository navigation
docs(calibrate): the calibration payload's contract must name binning_map and every shape it refuses - #1003
Merged
NandhaKishorM merged 4 commits intoOct 7, 2026
Conversation
…_map and every shape it refuses
`apply_calibration_payload` installs three fields -- `temperature`,
`temperature_by_options` and `binning_map` -- and its own summary line named two, so a reader of
the page had no way to learn that loading a file written before histogram binning existed clears
the map the agent fitted locally (`obj.binning_map = binning`, and `binning` is `None` when the
key is absent). The shape paragraph made the same class of error in the other direction: it
promised a refusal for "the three mistakes below" while the code refuses a dozen shapes, every
one of them already driven by tests/test_calibrate.py, and never mentioned `binning_map`'s
entries at all.
It also states the one policy a caller cannot guess: a bad *temperature* is clamped into
[TEMP_MIN, TEMP_MAX], while a bad *binning value* is refused outright, because an out-of-range
binning value would move a confidence with nothing defensible to fall back to. Neither
`Agent.load_calibration` nor `ONNXAgent.load_calibration` said so; both described clamping only,
and `Agent.save_calibration` claimed to write temperatures when it writes the map too.
No behaviour changes: every edit is docstring prose, and `calibration_payload`'s "the key is
omitted when no map is installed" is kept honest by naming what "no map" means (`None`, not `{}`
-- a map that fitted nothing is still written and round-trips as `{}`).
19 checks that make the prose above expensive to lose again:
- the refused-field set is read out of the function's own `raise ValueError` messages by AST, so
it is what the code refuses rather than what a reader remembers, and compared two ways to the
fields the docstring names: nothing the code refuses may go unmentioned, and nothing the
docstring mentions may be a shape the code accepts. A non-vacuity arm requires the scan to
reach at least 11 refusal messages, which is what `_refuses` already drives.
- the summary line must name all three installed fields, and the page must carry the
clamp-vs-refuse split and the keyless-file-clears-the-map semantics.
- the pre-fix wording is banned ("three mistakes", " two fields") so the count claim cannot come
back attached to a different number.
- `Agent.load_calibration`, `Agent.save_calibration` and `ONNXAgent.load_calibration` are read by
AST from their module source -- `ONNXAgent` is not imported, so this suite stays runnable
without the `onnx` extra -- and each must name `binning_map`.
- the semantics those words claim are driven: a payload installs its map, a payload without the
key clears an installed map to `None`, an explicit `null` clears it too, no map omits the key,
and `{}` round-trips as `{}` rather than collapsing to `None`.
Both words appear elsewhere in this docstring whatever it says -- `clamp_temperature` in the version paragraph, "refused with a ValueError" in the shape paragraph -- so a page that dropped the contrast between the two fields still passed. The check now reads the window after "opposite value policies", which is the sentence that claims the split. 234 passed, 0 failed, unchanged.
NandhaKishorM#731 rewrites that one-liner and NandhaKishorM#783 edits the lines around it, so this PR's only claim about ONNXAgent was a guaranteed merge conflict on a site another contributor already has open. Drop the docstring edit and the gate arm that read it; the contract this PR holds is the one in laya/calibrate.py and the one Agent.save_calibration/Agent.load_calibration publish. tests/test_calibrate.py goes 234 -> 233 passed, 0 failed.
Owner
|
Merged in 0.3.29. The calibration payload's contract now names This pairs with #957 from the last release, which put the abstention cut on the 4-decimal scale the gate reads, and with #1002 in this one. The calibration surface is now documented and checked at the same level of detail, which it was not two releases ago. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
apply_calibration_payload()installs three fields onto the object it is handed and refuses twelvedifferent payload shapes. Its docstring described two fields and three refusals, and described one
value policy where the code runs two opposite ones. Same for
Agent.save_calibration(which writesbinning_maptoo) andAgent.load_calibration.This corrects those three docstrings and adds a gate in
tests/test_calibrate.pythat reads therefusal messages out of
laya/calibrate.pyby AST and holds the docstring to them field by field, sothe page cannot drift from the raises again.
Lines are 154 insertions / 14 deletions across
laya/calibrate.py,laya/agent.py,tests/test_calibrate.py. No behaviour changes: every install/omit/refuse the gate asserts is whatthe code already does.
Why
load_calibrationis the page someone opens before hand-writing acalibration.jsonor loading afile an older agent wrote. Three claims on it were wrong in ways that cost real time:
temperatureandtemperature_by_optionsare copied onto the object. Italso writes
binning_map— unconditionally (obj.binning_map = binning), so loading a filewith no
binning_mapkey clears a map the agent had just fitted. A docstring that reads "copiesthe temperatures" implies the map survives; the code drops it. That is the difference between a
fine and a silent accuracy regression when someone loads a pre-binning file over a fitted agent.
an integer, or a bucket map that is not an object" — three. There are twelve raises, and the nine
it did not mention are the
binning_mapinner-shape ones (binsnot an int >= 1,valuesnot oflength
bins, non-numeric, non-finite, outside[0, 1]). Someone writing a map by hand has toknow which of those is a hard error.
clamp_temperature". Only the temperatures are clamped. Abinning value outside
[0, 1]is refused with aValueError. The docstring promised one policyfor two opposite ones, and the difference is not cosmetic: nothing recalibrates a confidence the
way an out-of-range binning value would, so there is no defensible fallback for it.
calibration_payload's page also said "the key is omitted when no map is installed" without sayingthat
{}— whatfit_binning_mapreturns when no bucket reaches the sample floor — is written. So{}andNonelook like the same thing on the page and are not: one round-trips as{}, the otheras
None.The gate is the point, not the prose. It parses
laya/calibrate.py, walks toapply_calibration_payload, pulls the literal message out of everyraise(stripping the%formatting), keeps the twelve that begin
calibration JSON, and derives from each message whichfield it refuses on. Then it requires the docstring to name every one of those fields and to name no
field the code does not refuse on. A docstring that adds a claim the code dropped, or drops a claim
the code still makes, fails. It also checks the two agent docstrings that publish the same contract,
reads them through
ast.get_docstringfrom module text so a-Orun cannot answer from a strippeddocstring, and drives the actual semantics (install / clear-on-keyless / clear-on-null / omit /
{}round-trips as
{}).How it was verified
The
215count is the same suite run from agit archiveofa4a8921(release: 0.3.28, thebranch point) — so the new checks are +18 (215 → 233), and no previously passing check was
removed or weakened.
Mutation table (
laya-bench/mutate_calibrate_contract.py, loglaya-bench/calibrate_contract_mutation.log). M0 is the committed state; each mutant is one edit,the tree is reset before every one, the suite runs with
-B, and an aborted run counts as a KILL:bins/valuesbinning_mapinstalls only when presentAttributeError: 'Stub' object has no attribute 'binning_map') — KILL{}stops round-trippingAgent.load_calibrationback to temperatures onlyAgent.save_calibrationstops naming the map it writes_fields_inmatches short words first (longest-first is load-bearing)12/12 killed, 0 survivors,
git status --shortempty after the run.M8 and M12 are the two that make the gate more than a substring test. M8 deletes the whole
clamp-vs-refuse paragraph while leaving the words "clamp" and "refuse" in the rest of the page (both
appear elsewhere whatever the page says), so the check reads the 500 characters after the phrase
"opposite value policies" and requires the split inside that window. M12 keeps the scan but matches
FIELD_WORDSshortest-first, so"temperature"is consumed out of"temperature_by_options"andthe field set comes back wrong — which is why
_fields_inreplaces the longest names first.No TypeScript is touched.
laya-ts/has no calibration-payload loader that repeats these claims —its only binning statement is the map validator at
laya-ts/src/common.ts:367— and that lane has aseparate owner.
Deconfliction against the 69 open PRs, file by file:
laya/calibrate.py— no open PR touches it.tests/test_calibrate.py— only docs(onnx): document the calibration knob test_calibrate.py pins #731, whose hunk sits at pre-image line 362 while this PR'sadditions start after line 653; 291 lines apart.
laya/agent.py— this PR's hunks are at 1995 and 2030. The nearest open-PR hunks in the file arefix(agent): read a state predict_long's questions leave room for in one pass #966 at 1735/1746 and perf(agent): tokenize only the prefix of state a sequence holds, and make state_tokens opt-in #687 at 1751/1763, so at least 230 lines away. fix(agent): prevent concurrent OOM fallback device races #660, fix(agent): comprehensive input validation for questions, instructions, and state (#707) #716, fix: make choice criteria permutation-invariant#779 #783, feat(encoder): add opt-in residual bottleneck adapters #921 also
edit
laya/agent.py, all above line 1870.laya/onnx_agent.py— this PR editedONNXAgent.load_calibration's docstring until commit2f0646f, which reverts it: docs(onnx): document the calibration knob test_calibrate.py pins #731 rewrites exactly that one-liner and fix: make choice criteria permutation-invariant#779 #783 changes lines inside thesame function, so keeping it meant a guaranteed conflict on a site another contributor already has
open.
tests/test_calibrate.pyaccordingly gives up theONNXAgentarm (that is the 234 → 233between my earlier local run and the numbers above).
Nothing here moves a number in the library: the three docstrings describe behaviour the code already
has, and the only count that changes is this suite's check total, 215 → 233.
tests/test_hooks_api.pyis untouched because no signature, no__all__member and no publicbehaviour changed: the diff is three docstrings plus one test file.
Checklist
mainFixes #