Skip to content

docs(calibrate): the calibration payload's contract must name binning_map and every shape it refuses - #1003

Merged
NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
aashish254:fix-calibration-payload-contract
Oct 7, 2026
Merged

NandhaKishorM merged 4 commits into
NandhaKishorM:mainfrom
aashish254:fix-calibration-payload-contract

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

apply_calibration_payload() installs three fields onto the object it is handed and refuses twelve
different payload shapes. Its docstring described two fields and three refusals, and described one
value policy where the code runs two opposite ones. Same for Agent.save_calibration (which writes
binning_map too) and Agent.load_calibration.

This corrects those three docstrings and adds a gate in tests/test_calibrate.py that reads the
refusal messages out of laya/calibrate.py by AST and holds the docstring to them field by field, so
the page cannot drift from the raises again.

Lines are 154 insertions / 14 deletions across laya/calibrate.py, laya/agent.py,
tests/test_calibrate.py. No behaviour changes: every install/omit/refuse the gate asserts is what
the code already does.

Why

load_calibration is the page someone opens before hand-writing a calibration.json or loading a
file an older agent wrote. Three claims on it were wrong in ways that cost real time:

  • It said the payload's temperature and temperature_by_options are copied onto the object. It
    also writes binning_map — unconditionally (obj.binning_map = binning), so loading a file
    with no binning_map key clears a map the agent had just fitted. A docstring that reads "copies
    the temperatures" implies the map survives; the code drops it. That is the difference between a
    fine and a silent accuracy regression when someone loads a pre-binning file over a fitted agent.
  • It said the shape checks refuse "a temperature that is not a list of three, a version that is not
    an integer, or a bucket map that is not an object" — three. There are twelve raises, and the nine
    it did not mention are the binning_map inner-shape ones (bins not an int >= 1, values not of
    length bins, non-numeric, non-finite, outside [0, 1]). Someone writing a map by hand has to
    know which of those is a hard error.
  • It said "each value is passed through clamp_temperature". Only the temperatures are clamped. A
    binning value outside [0, 1] is refused with a ValueError. The docstring promised one policy
    for two opposite ones, and the difference is not cosmetic: nothing recalibrates a confidence the
    way an out-of-range binning value would, so there is no defensible fallback for it.

calibration_payload's page also said "the key is omitted when no map is installed" without saying
that {} — what fit_binning_map returns when no bucket reaches the sample floor — is written. So
{} and None look like the same thing on the page and are not: one round-trips as {}, the other
as None.

The gate is the point, not the prose. It parses laya/calibrate.py, walks to
apply_calibration_payload, pulls the literal message out of every raise (stripping the %
formatting), keeps the twelve that begin calibration JSON, and derives from each message which
field it refuses on. Then it requires the docstring to name every one of those fields and to name no
field the code does not refuse on. A docstring that adds a claim the code dropped, or drops a claim
the code still makes, fails. It also checks the two agent docstrings that publish the same contract,
reads them through ast.get_docstring from module text so a -O run cannot answer from a stripped
docstring, and drives the actual semantics (install / clear-on-keyless / clear-on-null / omit / {}
round-trips as {}).

How it was verified

python -B tests/test_calibrate.py    # branch: 233 passed, 0 failed
python -B tests/test_confidence.py   # 134 passed, 0 failed
python -B tests/test_evals_api.py    # 48 passed, 0 failed
python -B tests/test_agent_gate.py   # 11 passed, 0 failed, exit 0
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/                                   # exit 0

The 215 count is the same suite run from a git archive of a4a8921 (release: 0.3.28, the
branch point) — so the new checks are +18 (215 → 233), and no previously passing check was
removed or weakened.

Mutation table (laya-bench/mutate_calibrate_contract.py, log
laya-bench/calibrate_contract_mutation.log). M0 is the committed state; each mutant is one edit,
the tree is reset before every one, the suite runs with -B, and an aborted run counts as a KILL:

Mutant Result
M0 baseline (committed state) 233 passed, 0 failed — green, as required
M1 summary line names two of the three installed fields 231/2 — KILL
M2 refusal paragraph stops naming bins/values 232/1 — KILL
M3 a refusal message stops naming its field 229/4 — KILL
M4 binning_map installs only when present aborted (AttributeError: 'Stub' object has no attribute 'binning_map') — KILL
M5 writer drops a falsy map, {} stops round-tripping 231/2 — KILL
M6 Agent.load_calibration back to temperatures only 231/2 — KILL
M7 Agent.save_calibration stops naming the map it writes 232/1 — KILL
M8 the clamp-vs-refuse contrast deleted 232/1 — KILL
M9 page says a keyless file leaves the map alone 232/1 — KILL
M10 the count claim comes back 232/1 — KILL
M11 AST scan blinded (non-vacuity self-check) 230/3 — KILL
M12 _fields_in matches short words first (longest-first is load-bearing) 232/1 — KILL

12/12 killed, 0 survivors, git status --short empty after the run.

M8 and M12 are the two that make the gate more than a substring test. M8 deletes the whole
clamp-vs-refuse paragraph while leaving the words "clamp" and "refuse" in the rest of the page (both
appear elsewhere whatever the page says), so the check reads the 500 characters after the phrase
"opposite value policies" and requires the split inside that window. M12 keeps the scan but matches
FIELD_WORDS shortest-first, so "temperature" is consumed out of "temperature_by_options" and
the field set comes back wrong — which is why _fields_in replaces the longest names first.

No TypeScript is touched. laya-ts/ has no calibration-payload loader that repeats these claims —
its only binning statement is the map validator at laya-ts/src/common.ts:367 — and that lane has a
separate owner.

Deconfliction against the 69 open PRs, file by file:

Nothing here moves a number in the library: the three docstrings describe behaviour the code already
has, and the only count that changes is this suite's check total, 215 → 233.

tests/test_hooks_api.py is untouched because no signature, no __all__ member and no public
behaviour changed: the diff is three docstrings plus one test file.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

Fixes #

…_map and every shape it refuses

`apply_calibration_payload` installs three fields -- `temperature`,
`temperature_by_options` and `binning_map` -- and its own summary line named two, so a reader of
the page had no way to learn that loading a file written before histogram binning existed clears
the map the agent fitted locally (`obj.binning_map = binning`, and `binning` is `None` when the
key is absent). The shape paragraph made the same class of error in the other direction: it
promised a refusal for "the three mistakes below" while the code refuses a dozen shapes, every
one of them already driven by tests/test_calibrate.py, and never mentioned `binning_map`'s
entries at all.

It also states the one policy a caller cannot guess: a bad *temperature* is clamped into
[TEMP_MIN, TEMP_MAX], while a bad *binning value* is refused outright, because an out-of-range
binning value would move a confidence with nothing defensible to fall back to. Neither
`Agent.load_calibration` nor `ONNXAgent.load_calibration` said so; both described clamping only,
and `Agent.save_calibration` claimed to write temperatures when it writes the map too.

No behaviour changes: every edit is docstring prose, and `calibration_payload`'s "the key is
omitted when no map is installed" is kept honest by naming what "no map" means (`None`, not `{}`
-- a map that fitted nothing is still written and round-trips as `{}`).
19 checks that make the prose above expensive to lose again:

- the refused-field set is read out of the function's own `raise ValueError` messages by AST, so
  it is what the code refuses rather than what a reader remembers, and compared two ways to the
  fields the docstring names: nothing the code refuses may go unmentioned, and nothing the
  docstring mentions may be a shape the code accepts. A non-vacuity arm requires the scan to
  reach at least 11 refusal messages, which is what `_refuses` already drives.
- the summary line must name all three installed fields, and the page must carry the
  clamp-vs-refuse split and the keyless-file-clears-the-map semantics.
- the pre-fix wording is banned ("three mistakes", " two fields") so the count claim cannot come
  back attached to a different number.
- `Agent.load_calibration`, `Agent.save_calibration` and `ONNXAgent.load_calibration` are read by
  AST from their module source -- `ONNXAgent` is not imported, so this suite stays runnable
  without the `onnx` extra -- and each must name `binning_map`.
- the semantics those words claim are driven: a payload installs its map, a payload without the
  key clears an installed map to `None`, an explicit `null` clears it too, no map omits the key,
  and `{}` round-trips as `{}` rather than collapsing to `None`.
Both words appear elsewhere in this docstring whatever it says -- `clamp_temperature` in the
version paragraph, "refused with a ValueError" in the shape paragraph -- so a page that dropped
the contrast between the two fields still passed. The check now reads the window after
"opposite value policies", which is the sentence that claims the split. 234 passed, 0 failed,
unchanged.
NandhaKishorM#731 rewrites that one-liner and NandhaKishorM#783 edits the lines around it, so this PR's
only claim about ONNXAgent was a guaranteed merge conflict on a site another
contributor already has open. Drop the docstring edit and the gate arm that
read it; the contract this PR holds is the one in laya/calibrate.py and the one
Agent.save_calibration/Agent.load_calibration publish. tests/test_calibrate.py
goes 234 -> 233 passed, 0 failed.
@NandhaKishorM
NandhaKishorM merged commit b65687a into NandhaKishorM:main Oct 7, 2026
32 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.3.29.

The calibration payload's contract now names binning_map and every shape it refuses. This is the payload that decides what answer_confidence reports, so a reader writing one by hand needs the refusals as much as the accepted shape, and a validator whose rejections are undocumented is a validator people work around.

This pairs with #957 from the last release, which put the abstention cut on the 4-decimal scale the gate reads, and with #1002 in this one. The calibration surface is now documented and checked at the same level of detail, which it was not two releases ago.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants