Repository navigation
fix(calibration): clear stale bucket overrides from notebook exports - #139
Merged
NandhaKishorM merged 1 commit intoSep 23, 2026
Merged
NandhaKishorM merged 1 commit into
NandhaKishorM merged 1 commit into
Conversation
Inherited option-count temperatures take precedence over newly fitted per-type values after the notebook export is reloaded. Clear them at the writer boundary while preserving intentional bucket calibration in existing checkpoints.
This was referenced Sep 22, 2026
Owner
|
Thank you. Clearing inherited bucket overrides before a per-type refit is the right fix, since otherwise the new values are silently hidden. Merging, and it pairs well with #191, which landed today and holds the calibration slice out of training. |
This was referenced Sep 23, 2026
baninaveen
pushed a commit
to baninaveen/laya
that referenced
this pull request
Sep 24, 2026
…haKishorM#139) Inherited option-count temperatures take precedence over newly fitted per-type values after the notebook export is reloaded. Clear them at the writer boundary while preserving intentional bucket calibration in existing checkpoints.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the typed-decisions notebook export retaining inherited
temperature_by_optionsafter fitting new per-typetemperaturevalues. Inference prioritizes matching buckets, so reloading the export silently ignores the new fit for those questions. Clear the inherited map at export; preserve existing checkpoints' intentional bucket precedence and runtime clamping.The published typed-decisions config, checked on 2026-09-22, has per-type temperatures approximately
[1.0148, 1.0374, 1.0575]alongside six buckets exactly matching the base config. For example,noul:2 = 1.9834overrides the new per-typenoul = 1.0575. The existing clamp bounds temperatures but does not resolve their precedence.Changes
cfg.pop("temperature_by_options", None)immediately after the notebook assigns its fitted per-type values, with a comment explaining why. No inference API or checkpoint-format changes.Testing
Original defect reproduction on Windows: before the export fix, the stale-map regression failed with 14 assertions while its other four tests passed. With logits
[0, 1], inherited bucket1.5gaveP(true)=0.6608despite the newly saved per-typenoultemperature4.0, which should give0.5622. All five calibration tests then passed on this PR alone. Its separate Windows download-test failure was a pre-existing path-separator issue; #140 fixes it, and the combined run below confirms that resolution.Cross-platform validation completed on 2026-09-22 against #139 at
39649dband #140 at5f40afe, independently based on upstream573e5b6. A temporary combined checkout applied #140's patch on #139, producing Git tree1bdb95a2e089de52b4832f29e6eabc2f7ed8b455; neither contribution branch was changed.Actually run on Windows x86_64 and Ubuntu 26.04 LTS x86_64 (kernel
7.0.0-30-generic). Both used Python 3.12.11, torch 2.14.0+cpu, transformers 5.17.0, safetensors 0.8.0, huggingface-hub 1.32.0, and numpy 2.5.3. All 19 Python/config/notebook source files compared equal after normalizing CRLF to LF.Combined checkout results on both platforms:
python tests/test_calibration_persistence.pypython tests/test_download.pypython tests/test_router.pypython tests/test_criteria.pypython tests/test_shortlist.pypython tests/test_decision_model.pypython tests/test_packaging.pypython tests/test_email.pyRuff (
--select=E9,F63,F7,F82,F401,F811 --line-length=120) onlaya/and the two affected tests, pluspython -m compileall -q laya/ tests/, passes on both platforms. Notebook JSON/nbformat schema and all 9 code cells validate on both; the full%%writefilescript is compiled separately, without execution.Linux also validated each PR independently in separate detached checkouts: #139 passes calibration (5) and download (4) tests, lint, compilation, and notebook validation; #140 passes download (4) tests, lint, and compilation.
Tests ran with
HF_HUB_OFFLINE=1,TRANSFORMERS_OFFLINE=1, an isolated empty Hub cache, and CPU-only torch. Fixtures are tiny local models and mocked transport. No pretrained weights were downloaded, no training was run, and the real-checkpointtest_local_e2e.pywas not executed. These are directly executed checks, not GitHub Actions results.Notes for reviewer
Rechecked open contributions on 2026-09-22: #19 at
4f2fa38and #53 at77e116e. No review comments or requested changes were present on #139 or #140. #19 owns the broader per-bucket fitting/persistence API and does not modify this notebook. #53 rewrites this notebook intolaya.finetune; its current export also assigns new per-type temperatures without clearing inherited buckets, so the same invalidation needs carrying over if it lands first. The export regression must also be retargeted tolaya.finetunewhen that notebook rewrite lands. #127 adds language-specific temperatures and #126 changes reported confidence; neither fixes this writer. This PR adds no competing calibration API and changes no workflows.This establishes configuration correctness only. No improvement in model accuracy, ECE, or other real-checkpoint metrics has been measured. The notebook still draws calibration examples from training items; the fitting policy is unchanged. Already published checkpoint files are not rewritten, because their bucket maps may be intentional.