Skip to content

feat: fit and persist per-bucket temperatures - #19

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
mvanhorn:cursor/fit-persist-bucket-temperatures-b1f1
Sep 29, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
mvanhorn:cursor/fit-persist-bucket-temperatures-b1f1

Conversation

@mvanhorn

@mvanhorn mvanhorn commented Sep 19, 2026 •

Copy link
Copy Markdown
Contributor

Demo

demo

Before: inference read temperature_by_options but nothing wrote it. After: Agent.fit_temperatures → save_calibration → load(…, calibration=) persists one T per (qtype, k) bucket.

Summary

Inference already looks up temperature_by_options before the type-level scalar. Nothing in the installed package fitted that map, wrote it, or loaded it. The notebook fitter still writes three scalars into cfg["temperature"].

This adds laya.calibrate (the notebook NLL + LBFGS on log T, grouped by temp_bucket) plus Agent.fit_temperatures / save_calibration / load_calibration and a calibration= path on Agent / load. Fitting takes CPU records (qtype, logits, target, k). No Hub download, no model.safetensors rewrite, no CLI.

From BENCHMARKS.md:

Both ship over-confident; laya-multilingual ships with no fitted temperatures at all. Refitting one temperature per (question type, option count) on held-out data is the single highest-value fix available, and takes ECE below Jev's measured 0.246.

as shipped temperature refit
laya 0.466 0.081
laya-multilingual 0.314 0.106

This PR is the writer for that refit. It does not re-run the Hub checkpoints.

from laya import fit_temperatures, load

result = fit_temperatures(records, compute_ece=True)
agent.fit_temperatures(records)
agent.save_calibration("calibration.json")
agent = load("convaiinnovations/laya", calibration="calibration.json")

Synthetic fixture (n=125, no Hub weights)

Overconfident one-hot records, 60% accuracy, peaked logits. Same fitter the tests run.

ECE
before (T=1) 0.3920
after fitted map 0.1728
bucket n T
choice:2 40 10.000
noul:2 40 10.000
choice:11+ 40 5.297
score:3-5 5 omitted (n < 10)

Type-level: choice 9.066, score 1.0 (only the five score rows), noul 10.0. save_calibration writes only temperature and temperature_by_options.

Tests

$ python tests/test_calibrate.py
48 passed, 0 failed
synthetic ECE before=0.3920 after=0.1728  choice:2 T=10.000  noul:2 T=10.000  choice:11+ T=5.297

Wired next to test_router.py / test_criteria.py in CI. The script does not construct a loaded Agent (no Hub download); live-map coverage uses Agent.__new__.

Files

  • laya/calibrate.py — new: fit_one_temperature, fit_temperature_map (fit_temperatures alias), JSON helpers
  • laya/agent.py — _forward_logits before T, fit/save/load, calibration= on __init__ / load
  • laya/__init__.py — export the fitter names
  • tests/test_calibrate.py — new
  • .github/workflows/ci.yml — run the new script

@NandhaKishorM

Copy link
Copy Markdown
Owner

This is the one I most want to land. You are right that nothing in the installed package ever wrote temperature_by_options, even though inference reads it first. Good catch.

One change I would like before merge: please clamp fitted temperatures to roughly [0.5, 5.0] and skip any bucket with fewer than about 2000 samples.

The reason is that the shipped English checkpoint has choice:11+ = 0.1006. That divides the logits by 0.1, so it sharpens them about 10x and reports around 0.89 confidence on 77 option questions it only gets right 42 percent of the time. Without a guard, a user fitting on a small held out set can land on exactly that again. A clamp plus a minimum sample count makes the API safe by default.

Also CI has not run on this branch. main requires tests (py3.9), tests (py3.11), tests (py3.13), lint and package. A rebase on latest main should trigger them.

@antonio-mello-ai

Copy link
Copy Markdown
Contributor

Hi, we're evaluating Laya locally for a PT-BR decision workload, and this PR addresses one of the main blockers we found: the multilingual checkpoint currently has no fitted temperature map, so we can't safely use its confidence scores for automation gates yet.

We'd be glad to help with the PR while respecting the work already done. Along with the [0.5, 5.0] clamp and the higher minimum sample count that were already requested, we ran into two issues that seem worth handling here:

  • The calibration file should say which model revision and config it belongs to. Without that, it would be easy to upgrade the weights and accidentally keep using an incompatible calibration map.
  • Fitting and evaluation should use different records. Right now, compute_ece evaluates on the same data used to fit the temperatures, which is likely to make the result look better than it will on unseen data.

We can put together a patch for those two points, plus basic payload versioning, stricter input validation, per-bucket sample counts, and atomic JSON writes.

Would you rather include that work in this PR or keep it for a focused follow-up? We don't want to duplicate the work already underway.

mvanhorn added a commit to mvanhorn/laya that referenced this pull request Sep 21, 2026
…t ECE

Addresses the review on NandhaKishorM#19.

Clamp fitted temperatures to [0.5, 5.0] and raise the per-bucket floor to
2000, so a small held-out set cannot reproduce the choice:11+ = 0.1006 case
that sharpens logits ~10x. MIN_TYPE_N stays at 10 as a separate floor for the
type-level scalar, so a dataset that fills no single bucket still gets a
temperature instead of staying at 1.0.

Record which checkpoint a calibration belongs to: the payload now carries
model_id_or_path, subfolder, the checkpoint config, and a schema version.
Loading a payload whose identity does not match warns and still loads, and a
payload with no version is read as version 1, so existing files keep working.

Stop scoring ECE on the fitting data. With compute_ece=True the records are
split per bucket by a seeded stratified partition, temperatures are fit on the
remainder, and ECE is reported on the held-out side only, with n_eval and any
buckets too small to split named in the report. compute_ece=False is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hbhAdQ6qFfwMgCNscPHwS
mvanhorn added a commit to mvanhorn/laya that referenced this pull request Sep 21, 2026
…t ECE

Addresses the review on NandhaKishorM#19.

Clamp fitted temperatures to [0.5, 5.0] and raise the per-bucket floor to
2000, so a small held-out set cannot reproduce the choice:11+ = 0.1006 case
that sharpens logits ~10x. MIN_TYPE_N stays at 10 as a separate floor for the
type-level scalar, so a dataset that fills no single bucket still gets a
temperature instead of staying at 1.0.

Record which checkpoint a calibration belongs to: the payload now carries
model_id_or_path, subfolder, the checkpoint config, and a schema version.
Loading a payload whose identity does not match warns and still loads, and a
payload with no version is read as version 1, so existing files keep working.

Stop scoring ECE on the fitting data. With compute_ece=True the records are
split per bucket by a seeded stratified partition, temperatures are fit on the
remainder, and ECE is reported on the held-out side only, with n_eval and any
buckets too small to split named in the report. compute_ece=False is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011hbhAdQ6qFfwMgCNscPHwS
@mvanhorn
mvanhorn force-pushed the cursor/fit-persist-bucket-temperatures-b1f1 branch from 885a200 to 4f2fa38 Compare September 21, 2026 17:24
@mvanhorn

Copy link
Copy Markdown
Contributor Author

Pushed 885a200, then rebased onto current main (v0.3.4) as 4f2fa38. All four items below are on that head.

@NandhaKishorM, your two:

Clamp is now [0.5, 5.0] and the per-bucket floor is 2000. One thing worth flagging: with the clamp tightened, the synthetic fixture in tests/test_calibrate.py now pins all three of its buckets at the ceiling (choice:2, noul:2 and choice:11+ all report T=5.000). That is the guard working rather than a fit result, and it is the mirror image of the choice:11+ = 0.1006 case you described, which sat on the old 0.1 floor. Synthetic data should not be read as evidence about real checkpoints either way.

I did not raise the floor uniformly. MIN_BUCKET_N = 2000 is the per-bucket floor, and MIN_TYPE_N = 10 is a separate, lower floor used only for the type-level scalar. If both were 2000, a realistic labeled set that fills no single bucket would leave temperature at [1.0, 1.0, 1.0] and the API would silently do nothing. The module docstring says which is which so the two do not get collapsed later.

@antonio-mello-ai, your two, and thank you for the specifics:

The calibration payload now records model_id_or_path, subfolder, and the checkpoint config (minus the temperature fields it already carries), plus a schema version. A payload whose recorded identity does not match the agent loading it warns rather than raising, and a payload with no version key is read as version 1, so nothing written before this stops loading.

compute_ece no longer scores on its fitting data. With compute_ece=True the records are split per bucket by a seeded stratified partition, temperatures are fit on the remainder, and ECE is reported on the held-out side only. report carries n_eval alongside n so the number is legible, and a bucket too small to split after the holdout is fit on all of its records and named in report["buckets_excluded_from_eval"] rather than dropped. compute_ece=False is byte-for-byte the old behavior, since that is the path the API uses.

On your question of scope: I took those two here because they change the payload format and the meaning of a reported number, and both are easier to reason about landing with the writer they belong to. n_by_bucket already existed, so per-bucket counts are covered. That leaves stricter input validation and atomic JSON writes from your list, which are self-contained and do not touch this format. A focused follow-up from you would be welcome for those, and I am happy to review rather than duplicate.

On CI: the rebase is done and the branch is level with main, but no checks have reported on it yet. If workflow runs from forks need maintainer approval in this repo, that is probably what it is waiting on.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for following through on the clamp and sample-floor requests. This is still the calibration feature I most want to land. It needs a rebase onto today's main. Two things landed that touch it: #142 (invalid temperatures no longer crash loading) and #191 (calibration fitted on a held-out slice). If the rebase keeps both behaviours, I'd like to merge it.

@antonio-mello-ai

Copy link
Copy Markdown
Contributor

Thanks — that makes sense. We’ll wait for the rebase and merge of this PR, and would be happy to contribute a focused follow-up PR for strict input validation and atomic JSON writes afterward.

@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you for sticking with this. It is still the calibration feature we most want, and main does not have it yet: the notebook fits only three per-type scalars, and nothing in the package writes temperature_by_options. When you rebase onto current main, please export the new names through the lazy _LAZY_ATTRS table in laya/init.py, reuse TEMP_MIN, TEMP_MAX and clamp_temperature from laya.common instead of separate constants, and build the logits from the new _encode_state and _decode_answers path. Once CI is green on the rebased branch we can merge it.

@mvanhorn
mvanhorn force-pushed the cursor/fit-persist-bucket-temperatures-b1f1 branch from 4f2fa38 to d29b918 Compare September 25, 2026 16:55
@mvanhorn

Copy link
Copy Markdown
Contributor Author

Thanks @NandhaKishorM. Rebuilt on today's main as a single commit (d29b918) rather than rebasing the old stack, since agent.py had moved a lot.

Both behaviours are kept:

The new names are lazy exports, so import laya still doesn't pull in torch. test_calibrate.py (121 checks) is in both CI lists. test_calibration_persistence, test_temperature_loading and test_packaging pass locally.

@Bruce-Yii

Copy link
Copy Markdown
Contributor

Checked the rebuilt head against the points above. The base is current main (4066d5d5). The new exports stay lazy, calibration reuses the shared temperature bounds/clamp helper, and record collection goes through _encode_state with the same option-logit slicing used by production decode. I also verified the #142 clamp/warn behavior is preserved and the ECE split keeps fit/eval rows disjoint. Locally, test_calibrate.py is 121/121 and the persistence/loading/packaging suites pass.

The only remaining gate I couldn't verify through CI yet: all four Actions runs on this head are currently action_required, so GitHub hasn't actually executed them. Once those run, the final “CI green” condition can be checked.

@NandhaKishorM
NandhaKishorM merged commit d8f47e1 into NandhaKishorM:main Sep 29, 2026
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @mvanhorn! It ships in 0.3.22.

@mvanhorn

mvanhorn commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @NandhaKishorM for merging per-bucket temperatures. Curious to see how the fitted values hold up on real traffic.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants