Skip to content

fix(calibrate): keep a ragged pool's fit from vanishing into 1.0 - #1158

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-calibrate-ragged-padded-nan
Oct 12, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
aashish254:fix-calibrate-ragged-padded-nan

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

laya/calibrate.py::fit_one_temperature stops losing fits. Two edits, both inside that one
function's objective:

  • the pairs are grouped by option count instead of being padded into one matrix
    (_pairs_by_width replaces _pairs_to_tensors);
  • the LBFGS search on log T is bounded to LOG_T_MARGIN (10.0) e-folds outside the
    [TEMP_MIN, TEMP_MAX] band it has to answer inside.

laya/calibrate.py +34/-14, tests/test_calibrate.py +120. No signature, no payload schema,
no public name changes: LOG_T_MARGIN is a module constant beside MIN_TYPE_N, and the answer
is still whatever clamp_temperature(fitted, TEMP_MIN, TEMP_MAX) returns.

Why

A question type is fit on every record of that type pooled across option-count bands
(_group_records at laya/calibrate.py:205, consumed by _fit_groups at :254, which calls
fit_one_temperature(pairs, min_n=MIN_TYPE_N) at :260 for the type-level scalar and :266
for each bucket). Those pools are ragged: a band like 3-5 holds three widths on its own, and
the type-level pool spans all of them. So fit_one_temperature gets rows of different lengths
as routine input, not as an edge case.

On 682ac671 it used to square them off: _pairs_to_tensors at :88 allocated
torch.full((len(pairs), kmax), -1e4) (:90). Any
fill value is a real column in the single softmax every row then goes through, so it competes
for the probability mass the fit is allocating. Two things follow, and both end in
clamp_temperature (laya/common.py:828-843) reporting the neutral 1.0 for a fit that ran:

  1. a row whose own logits sit under the fill value stops being about that row;
  2. the search on log T is unbounded, and a pool whose optimum lies below TEMP_MIN walks it
    past float32's exp() underflow (~-88), divides the objective by zero, and turns every
    gradient into NaN.

Reproducible from the public API with no model weights:

import numpy as np
from laya.calibrate import fit_one_temperature, fit_temperature_map

def pool(width, scale, n, seed=0):
    rs = np.random.RandomState(seed)
    out = []
    for _ in range(n):
        z = list(rs.randn(width) * scale)
        t = [0.0] * width
        t[int(np.argmax(z))] = 1.0
        out.append((z, t))
    return out

low = [([-20000.0 - i, -20001.0 - i], [1.0, 0.0]) for i in range(12)]   # every logit under -1e4

What each arm returns, measured on 682ac671 and on this branch (one process per tree, each
printing the laya/calibrate.py path it loaded):

arm 682ac671 this branch
widths 2 and 9 pooled, 24 rows each 1.0000000000 0.5000000000
the same two widths on their own 0.5000000000, 0.5000000000 unchanged
12 low rows alone 0.5000000000 0.5000000000
12 low rows pooled with 12 width-4 rows 5.0000000000 0.5000000000
fit_temperature_map on 808 records of one type at widths 2 and 5, temperature[1] 5.0 0.5

The NaN path, with the intermediate values of the old objective exposed (its _pairs_to_tensors
and closure copied verbatim, so this is the old code, not a paraphrase):

[grid 51  w=3 acc=0.7 mag=0.3  n=400] -> 1.0  raw_fitted=nan log_t=nan pad_width=3 grad_finite=False steps=100
[grid 120 w=5 acc=0.5 mag=0.3  n=60 ] -> 1.0  raw_fitted=nan log_t=nan pad_width=5 grad_finite=False steps=100
[noisy 14 w=5 scale=0.5 n=200 seed=0] -> 1.0  raw_fitted=nan log_t=nan pad_width=5 grad_finite=False steps=100
[noisy 15 w=5 scale=0.5 n=200 seed=3] -> 1.0  raw_fitted=nan log_t=nan pad_width=5 grad_finite=False steps=100
[noisy 23 w=5 scale=3.0 n=200 seed=3] -> 1.0  raw_fitted=nan log_t=nan pad_width=5 grad_finite=False steps=100

The optimum is below the floor for all five, so TEMP_MIN is the right answer and they get 1.0.
They are not ragged, which is why the search bound is the part that matters: any pool of
low-spread, mostly-right rows can walk there, and mag/scale 0.3-3.0 is ordinary near-tied
option text, not a constructed worst case.

temp_bucket (laya/common.py:815) is untouched, MIN_BUCKET_N/MIN_TYPE_N are untouched, and
clamp_temperature is still the last word, so a value that is a non-number for some other reason
still lands on 1.0 rather than raising.

How it was verified

python tests/test_calibrate.py
python scripts/test_suites.py
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
python -m compileall -q laya/ tests/
zensical build --strict --clean

tests/test_calibrate.py goes 257 -> 278 passed, 0 failed; the new section is 21 checks in
five labelled groups (runaway/, pad/, public/, band/, weight/). Full gate
all 87 suites passed, 499 passed / 1 skipped, rc 0. ruff rc 0, compileall rc 0, docs
strict build No issues found rc 0.

No doc file changes here: the public API is unchanged, and
docs/reference/helpers.md:116 renders this function's docstring through
::: laya.calibrate.fit_one_temperature, so the strict build picked up the new wording on its
own. The built reference page, tags stripped, reads verbatim:

The search is bounded to LOG_T_MARGIN e-folds outside that band, which covers any minimum the
answer can use; past it the float32 scale drifts toward exp() underflow, where the objective
divides by zero and every gradient becomes NaN, and clamp_temperature reads that NaN back as
the neutral 1.0.

That the new checks are the fix's, not decoration: the same file run against 682ac671's
laya/calibrate.py (base module copied over the branch tree, laya.__file__ printed to prove
which copy answered) gives 266 passed, 8 failed, and the eight are exactly the behavioural
ones --

  FAIL runaway/ragged pool is not the neutral 1.0 T=1.0
  FAIL runaway/ragged pool keeps the floor T=1.0
  FAIL runaway/low-spread pool is not the neutral 1.0 T=1.0
  FAIL runaway/low-spread pool keeps the floor T=1.0
  FAIL pad/low rows pooled fit what they fit alone:
  FAIL pad/pool is not the ceiling T=5.0
  FAIL public/ragged type keeps the floor T=5.0
  FAIL public/ragged type is neither neutral nor the ceiling T=5.0

(The three checks that name _pairs_by_width / LOG_T_MARGIN are dropped from that run because
the base has no such symbol; with them left in, the base run cannot get past the import.)

One check does not compare against a hard-coded number: band/interior fit beats the floor
re-evaluates the objective in float64 and requires the returned T to beat TEMP_MIN and both
sides of itself, so it fails if bounding the search ever hides a real minimum.

Nothing that used to be right moves. 356 single-width configs (8 widths x 4 accuracies x 5 logit
magnitudes x 2 sizes, plus 18 random-logit configs at three widths) were fit on both trees and
diffed at full precision: 351 are bit-identical, including the ones that land strictly inside
the band (4.89012479782, 1.03938007355, 4.15759325027), and the 5 that move are exactly the
5 non-finite fits above. The base returns 1.0 for 36 of those 356 configs; the other 31 return
1.0 on this branch too, with steps=1 grad_finite=True log_t=0.000000 -- they converged where
they started because their NLL is already ~0, and this change leaves them alone. One config
(w=20 acc=0.7 mag=1.0 n=400) sits at 0.5222262144088745, genuinely inside the band; the
float64 objective at that scale is 1.4656239107 against 1.4686198900 at the floor, so it is a
real minimum and this branch keeps it to the digit. That is why the bound is 10 e-folds outside
the band rather than on its edge: clamping to the band itself turns the objective flat over the
region a near-floor minimum needs, and that config comes back as 0.5.

Mutation check -- five faithful edits to laya/calibrate.py, tests/test_calibrate.py after
each, file restored from 97bb3a60 between runs:

mutant killed by
objective unbound (log_t.exp()), grouping kept 4 checks, runaway/* at T=1.0
padded matrix restored, bound kept 4 checks, pad/* and public/* at T=5.0
LOG_T_MARGIN = 0.0 (bound on the band edge) 2 checks, band/interior fit stays interior at T=0.5
LOG_T_MARGIN = 90.0 (bound past the underflow) 1 check, band/the bound stops before float32 exp() underflows, lower edge=-90.69314718055995
mean of per-width means instead of mean over rows 1 check, weight/doubling a width pulls the fit to that width (even=2.0012264251708984 heavy=2.0012261867523193)
control: no edit 278 passed, 0 failed

The last mutant is why the weight/ checks exist: with per-width means the fit stops caring how
many rows a width has, and duplicating 8 rows leaves the answer unchanged to 2.4e-07.

Deconfliction: 50 open PRs as at 2026-10-12T06:07Z, re-derived per file -- none edits
laya/calibrate.py. Only #731 (open, last touched 2026-10-01) edits tests/test_calibrate.py,
at @@ -16,6 +16,7 @@ and @@ -362,6 +363,51 @@; this branch's two hunks are @@ -23,12 +23,14 @@
(both import additions in one hunk) and @@ -177,6 +179,124 @@, so no line is shared -- their
hunk ends at old line 21 and this one starts at 23. #1147 rewrites the prose beside this
(docs/finetune.md:287, which on 682ac671 still says the clamp is [0.1, 10]) and calls a type
the optimiser "wants to sharpen below the floor ... gets the bound rather than its own number" --
this branch is what makes that sentence true.

Not fixed here, worth knowing

  • calibration_report (laya/train.py:754) does notice a type left at 1.0, but its message
    (:800) says the items "gave it nothing to correct", which is the wrong story for a NaN fit --
    those rows had a clear answer. Only laya/train.py:1139 calls it, so a CPU-side
    fit_temperature_map / Agent.fit_temperatures caller gets no diagnostic at all.
  • The fitter installs with warn=False (laya/agent.py:1993, note at laya/calibrate.py:608), so
    temperature_raw stores the value after clamp_temperature turned the NaN into 1.0. A
    calibration file written from such a fit cannot be told apart from one that legitimately
    converged at 1.0. Fixing the fit removes the cause; the artifact still keeps no trace.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

A type-level fit pools every option count that type has seen, so the pairs
arrive ragged. Squaring them off put -1e4 in the same softmax each row runs
through, which stole probability from rows whose own logits sat under it and
overflowed to NaN in float32 once the optimiser drove the scale toward zero --
and clamp_temperature reads a NaN back as the neutral 1.0. Bound the log-space
search to LOG_T_MARGIN e-folds outside the answer band (no in-band minimum is
lost) and group the pairs by option count so no pad column competes.
…verage

Repeating one option count's rows has to move the fit toward the scale that width
wants on its own. Averaging the per-width averages instead leaves it where it was,
so the pooled objective is checked for a move big enough to be the weighting, not
for a strict inequality a float difference would satisfy.
@NandhaKishorM
NandhaKishorM merged commit 715b82d into NandhaKishorM:main Oct 12, 2026
44 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged in 0.4.5.

This is the one real behaviour fix in the release, so I measured the thing that matters most about it: whether it moves a fit that was already correct. It does not. Fifty-four single-width pools, every combination of k in 2/3/4/5/12/20, accuracy 0.5/0.7/0.9 and logit magnitude 2/4/8, come out identical to twelve decimal places before and after. With one option count there is one group, the sum over rows divided by the row count is the same mean, and the log-T clamp sits ten e-folds outside the band, so nothing in the ordinary path can reach it.

The ragged case is as broken as you describe. The twelve rows whose every logit sits under the old -1e4 pad:

before:  low rows alone -> 0.5    pooled with 4-option rows -> 5.0
after:   low rows alone -> 0.5    pooled with 4-option rows -> 0.5

Pooling used to flip the answer from the floor to the ceiling, and through fit_temperature_map the type-level scalar for the same records went [1.0, 5.0, 1.0] to [1.0, 0.5, 1.0]. An ordinary ragged pool moves only in the last digits, 1.305085 to 1.304285, which is the pad column's small residual share of the softmax.

No shipped payload changes: this is the fitting path, not the reading path, so the 94-row real-weights decision diff against 0.4.4 is identical. LOG_T_MARGIN beside MIN_TYPE_N and clamp_temperature still having the last word are the right shape for this.

Thanks for taking the trouble to show that the objective still averages over rows rather than over widths. That was the first thing I went looking for in the grouped version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants