Skip to content

fix(calibration): clear stale bucket overrides from notebook exports - #139

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
MycelAI:alpha912/fix/calibration-persistence
Sep 23, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
MycelAI:alpha912/fix/calibration-persistence

Conversation

@alpha912

@alpha912 alpha912 commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fix the typed-decisions notebook export retaining inherited temperature_by_options after fitting new per-type temperature values. Inference prioritizes matching buckets, so reloading the export silently ignores the new fit for those questions. Clear the inherited map at export; preserve existing checkpoints' intentional bucket precedence and runtime clamping.

The published typed-decisions config, checked on 2026-09-22, has per-type temperatures approximately [1.0148, 1.0374, 1.0575] alongside six buckets exactly matching the base config. For example, noul:2 = 1.9834 overrides the new per-type noul = 1.0575. The existing clamp bounds temperatures but does not resolve their precedence.

Changes

  • Add cfg.pop("temperature_by_options", None) immediately after the notebook assigns its fitted per-type values, with a comment explaining why. No inference API or checkpoint-format changes.
  • Add five CPU-only regression tests that execute the actual notebook config-export tail, reload a tiny locally generated BERT checkpoint, and exercise real inference postprocessing with deterministic logits. Cover stale overrides, existing bucket precedence across bucket boundaries, absent/empty maps, missing-bucket fallback by type, default temperatures, and preservation of unrelated config fields.
  • Document the export behavior, compatibility, test command, and limits of the evidence in the README.

Testing

Original defect reproduction on Windows: before the export fix, the stale-map regression failed with 14 assertions while its other four tests passed. With logits [0, 1], inherited bucket 1.5 gave P(true)=0.6608 despite the newly saved per-type noul temperature 4.0, which should give 0.5622. All five calibration tests then passed on this PR alone. Its separate Windows download-test failure was a pre-existing path-separator issue; #140 fixes it, and the combined run below confirms that resolution.

Cross-platform validation completed on 2026-09-22 against #139 at 39649db and #140 at 5f40afe, independently based on upstream 573e5b6. A temporary combined checkout applied #140's patch on #139, producing Git tree 1bdb95a2e089de52b4832f29e6eabc2f7ed8b455; neither contribution branch was changed.

Actually run on Windows x86_64 and Ubuntu 26.04 LTS x86_64 (kernel 7.0.0-30-generic). Both used Python 3.12.11, torch 2.14.0+cpu, transformers 5.17.0, safetensors 0.8.0, huggingface-hub 1.32.0, and numpy 2.5.3. All 19 Python/config/notebook source files compared equal after normalizing CRLF to LF.

Combined checkout results on both platforms:

Command Windows Linux
python tests/test_calibration_persistence.py 5 tests pass 5 tests pass
python tests/test_download.py 4 tests pass 4 tests pass
python tests/test_router.py 150 checks pass 150 checks pass
python tests/test_criteria.py 34 checks pass 34 checks pass
python tests/test_shortlist.py 81 checks pass 81 checks pass
python tests/test_decision_model.py 3 tests pass 3 tests pass
python tests/test_packaging.py 9 checks pass 9 checks pass
python tests/test_email.py 11 checks pass 11 checks pass

Ruff (--select=E9,F63,F7,F82,F401,F811 --line-length=120) on laya/ and the two affected tests, plus python -m compileall -q laya/ tests/, passes on both platforms. Notebook JSON/nbformat schema and all 9 code cells validate on both; the full %%writefile script is compiled separately, without execution.

Linux also validated each PR independently in separate detached checkouts: #139 passes calibration (5) and download (4) tests, lint, compilation, and notebook validation; #140 passes download (4) tests, lint, and compilation.

Tests ran with HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, an isolated empty Hub cache, and CPU-only torch. Fixtures are tiny local models and mocked transport. No pretrained weights were downloaded, no training was run, and the real-checkpoint test_local_e2e.py was not executed. These are directly executed checks, not GitHub Actions results.

Notes for reviewer

Rechecked open contributions on 2026-09-22: #19 at 4f2fa38 and #53 at 77e116e. No review comments or requested changes were present on #139 or #140. #19 owns the broader per-bucket fitting/persistence API and does not modify this notebook. #53 rewrites this notebook into laya.finetune; its current export also assigns new per-type temperatures without clearing inherited buckets, so the same invalidation needs carrying over if it lands first. The export regression must also be retargeted to laya.finetune when that notebook rewrite lands. #127 adds language-specific temperatures and #126 changes reported confidence; neither fixes this writer. This PR adds no competing calibration API and changes no workflows.

This establishes configuration correctness only. No improvement in model accuracy, ECE, or other real-checkpoint metrics has been measured. The notebook still draws calibration examples from training items; the fitting policy is unchanged. Already published checkpoint files are not rewritten, because their bucket maps may be intentional.

Inherited option-count temperatures take precedence over newly fitted per-type values after the notebook export is reloaded. Clear them at the writer boundary while preserving intentional bucket calibration in existing checkpoints.
@NandhaKishorM

Copy link
Copy Markdown
Owner

Thank you. Clearing inherited bucket overrides before a per-type refit is the right fix, since otherwise the new values are silently hidden. Merging, and it pairs well with #191, which landed today and holds the calibration slice out of training.

@NandhaKishorM
NandhaKishorM merged commit bdce407 into NandhaKishorM:main Sep 23, 2026
baninaveen pushed a commit to baninaveen/laya that referenced this pull request Sep 24, 2026
…haKishorM#139)

Inherited option-count temperatures take precedence over newly fitted per-type values after the notebook export is reloaded. Clear them at the writer boundary while preserving intentional bucket calibration in existing checkpoints.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants