Repository navigation
feat: fit and persist per-bucket temperatures - #19
NandhaKishorM merged 1 commit into
Conversation
|
This is the one I most want to land. You are right that nothing in the installed package ever wrote One change I would like before merge: please clamp fitted temperatures to roughly The reason is that the shipped English checkpoint has Also CI has not run on this branch. |
|
Hi, we're evaluating Laya locally for a PT-BR decision workload, and this PR addresses one of the main blockers we found: the multilingual checkpoint currently has no fitted temperature map, so we can't safely use its confidence scores for automation gates yet. We'd be glad to help with the PR while respecting the work already done. Along with the
We can put together a patch for those two points, plus basic payload versioning, stricter input validation, per-bucket sample counts, and atomic JSON writes. Would you rather include that work in this PR or keep it for a focused follow-up? We don't want to duplicate the work already underway. |
…t ECE Addresses the review on NandhaKishorM#19. Clamp fitted temperatures to [0.5, 5.0] and raise the per-bucket floor to 2000, so a small held-out set cannot reproduce the choice:11+ = 0.1006 case that sharpens logits ~10x. MIN_TYPE_N stays at 10 as a separate floor for the type-level scalar, so a dataset that fills no single bucket still gets a temperature instead of staying at 1.0. Record which checkpoint a calibration belongs to: the payload now carries model_id_or_path, subfolder, the checkpoint config, and a schema version. Loading a payload whose identity does not match warns and still loads, and a payload with no version is read as version 1, so existing files keep working. Stop scoring ECE on the fitting data. With compute_ece=True the records are split per bucket by a seeded stratified partition, temperatures are fit on the remainder, and ECE is reported on the held-out side only, with n_eval and any buckets too small to split named in the report. compute_ece=False is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hbhAdQ6qFfwMgCNscPHwS
…t ECE Addresses the review on NandhaKishorM#19. Clamp fitted temperatures to [0.5, 5.0] and raise the per-bucket floor to 2000, so a small held-out set cannot reproduce the choice:11+ = 0.1006 case that sharpens logits ~10x. MIN_TYPE_N stays at 10 as a separate floor for the type-level scalar, so a dataset that fills no single bucket still gets a temperature instead of staying at 1.0. Record which checkpoint a calibration belongs to: the payload now carries model_id_or_path, subfolder, the checkpoint config, and a schema version. Loading a payload whose identity does not match warns and still loads, and a payload with no version is read as version 1, so existing files keep working. Stop scoring ECE on the fitting data. With compute_ece=True the records are split per bucket by a seeded stratified partition, temperatures are fit on the remainder, and ECE is reported on the held-out side only, with n_eval and any buckets too small to split named in the report. compute_ece=False is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011hbhAdQ6qFfwMgCNscPHwS
885a200 to
4f2fa38
Compare
|
Pushed 885a200, then rebased onto current main (v0.3.4) as 4f2fa38. All four items below are on that head. @NandhaKishorM, your two: Clamp is now I did not raise the floor uniformly. @antonio-mello-ai, your two, and thank you for the specifics: The calibration payload now records
On your question of scope: I took those two here because they change the payload format and the meaning of a reported number, and both are easier to reason about landing with the writer they belong to. On CI: the rebase is done and the branch is level with main, but no checks have reported on it yet. If workflow runs from forks need maintainer approval in this repo, that is probably what it is waiting on. |
|
Thank you for following through on the clamp and sample-floor requests. This is still the calibration feature I most want to land. It needs a rebase onto today's |
|
Thanks — that makes sense. We’ll wait for the rebase and merge of this PR, and would be happy to contribute a focused follow-up PR for strict input validation and atomic JSON writes afterward. |
|
Thank you for sticking with this. It is still the calibration feature we most want, and main does not have it yet: the notebook fits only three per-type scalars, and nothing in the package writes temperature_by_options. When you rebase onto current main, please export the new names through the lazy _LAZY_ATTRS table in laya/init.py, reuse TEMP_MIN, TEMP_MAX and clamp_temperature from laya.common instead of separate constants, and build the logits from the new _encode_state and _decode_answers path. Once CI is green on the rebased branch we can merge it. |
4f2fa38 to
d29b918
Compare
|
Thanks @NandhaKishorM. Rebuilt on today's Both behaviours are kept:
The new names are lazy exports, so |
|
Checked the rebuilt head against the points above. The base is current main ( The only remaining gate I couldn't verify through CI yet: all four Actions runs on this head are currently |
|
Passed checks and merged. Thank you @mvanhorn! It ships in 0.3.22. |
|
Thanks @NandhaKishorM for merging per-bucket temperatures. Curious to see how the fitted values hold up on real traffic. |
Demo
Before: inference read
temperature_by_optionsbut nothing wrote it. After:Agent.fit_temperatures→save_calibration→load(…, calibration=)persists one T per(qtype, k)bucket.Summary
Inference already looks up
temperature_by_optionsbefore the type-level scalar. Nothing in the installed package fitted that map, wrote it, or loaded it. The notebook fitter still writes three scalars intocfg["temperature"].This adds
laya.calibrate(the notebook NLL + LBFGS on log T, grouped bytemp_bucket) plusAgent.fit_temperatures/save_calibration/load_calibrationand acalibration=path onAgent/load. Fitting takes CPU records(qtype, logits, target, k). No Hub download, nomodel.safetensorsrewrite, no CLI.From BENCHMARKS.md:
layalaya-multilingualThis PR is the writer for that refit. It does not re-run the Hub checkpoints.
Synthetic fixture (n=125, no Hub weights)
Overconfident one-hot records, 60% accuracy, peaked logits. Same fitter the tests run.
choice:2noul:2choice:11+score:3-5n < 10)Type-level: choice 9.066, score 1.0 (only the five
scorerows), noul 10.0.save_calibrationwrites onlytemperatureandtemperature_by_options.Tests
Wired next to
test_router.py/test_criteria.pyin CI. The script does not construct a loadedAgent(no Hub download); live-map coverage usesAgent.__new__.Files
laya/calibrate.py— new:fit_one_temperature,fit_temperature_map(fit_temperaturesalias), JSON helperslaya/agent.py—_forward_logitsbefore T, fit/save/load,calibration=on__init__/loadlaya/__init__.py— export the fitter namestests/test_calibrate.py— new.github/workflows/ci.yml— run the new script