Repository navigation
docs(finetune): the temperature fit clamps to [0.5, 5.0], and a failed fit keeps the inherited value - #1147
docs(finetune): the temperature fit clamps to [0.5, 5.0], and a failed fit keeps the inherited value#1147aashish254 wants to merge 2 commits into
Conversation
…d fit keeps the inherited value The page promised a "[0.1, 10]" clamp and a "1.2 if the fit raises" fallback; the fit clamps through clamp_temperature(fitted, TEMP_MIN, TEMP_MAX) = (0.5, 5.0), and the notebook it describes catches the failure and leaves the checkpoint temperature untouched. Gate both numbers against laya/common.py and laya/calibrate.py, and the failure path against the notebook's own rank-0 block.
|
Short answer: the one red check on this PR is a Windows runner crash in a suite this PR does not touch, and two other open PRs hit the identical crash today. What the job printed ( The loop at Same signature on two other branches today, both unrelated to docs:
Against that, the same commit is green everywhere else (37 checks pass), and my two neighbouring PRs #1143 ( This PR changes I can't re-run the job myself ( |
What
docs/finetune.md's post-training calibration paragraph promised three things the code does not do. Itsaid the fitted per-type temperatures are "clamped to
[0.1, 10](1.0for a slice under ten items,1.2if the fit raises)". Two numbers are wrong and one does not exist:[0.1, 10][TEMP_MIN, TEMP_MAX]=[0.5, 5.0](laya/common.py:824-825), applied byfit_one_temperature's last line (laya/calibrate.py:138)1.2if the fit raisesexceptbranch setsfitted_temps = None(…2xT4_kaggle.ipynb:324-326), soout_cfg["temperature"]is never written and the inherited checkpoint temperature ships; thetemperature_by_optionspop sits insideif fitted_temps is not None:(:331,:334), so a failed fit keeps the inherited buckets too1.0for a slice under ten items_fit_groupsonly fits whenlen(pairs) >= MIN_TYPE_N(laya/calibrate.py:239-240), andMIN_TYPE_N = 10(:39)The page's own next sentence ("removes any inherited
temperature_by_optionsin the same write") wastrue only on the success path and read as unconditional, so it moved inside that qualifier. The new
text:
11 insertions and 4 deletions in the page; 100 lines of new gate in
tests/test_finetune_entrypoints.py.Why
This is the paragraph a fine-tuner reads right before they gate a production decision on the
checkpoint's confidence.
README.md:1559already documents[0.5, 5.0]for checkpoint load, so thepage was the only copy in the repo claiming a window twice as wide — and
0.1/10are not bounds atall, they are LBFGS's
lrandmax_iterfrom the line above the clamp (laya/calibrate.py:128,torch.optim.LBFGS([log_t], lr=0.1, max_iter=100)),which is how the sentence likely got written. A reader sizing a threshold off
[0.1, 10]believes a fitcan be ~5x sharper than the floor actually allows.
The
1.2half is worse because it changes what a failed run means. Nothing inlaya/fits a failureto 1.2; that value exists only as a per-question-type no-data default in two other wrappers
(
research/scripts/finetune_single_device.py:91,notebooks/laya_finetune_typed_decisions_mps.py:136),keyed on
entry.get("items") == 0. In the path this page describes, a raising fit leaves the inheritedtemperature in place — so a run that never calibrated is indistinguishable, in its config, from a run
that inherited and shipped it. Expecting 1.2 means believing something was fitted when nothing was.
How it was verified
Read-only probe against this tree, no CUDA and no training run needed:
Both bounds are reachable, so the window is a real clamp and not decoration: an always-correct slice
lands the fit on the 0.5 floor, a half-wrong peaked slice lands it on the 5.0 ceiling.
The gate is three tests plus the ban, and every arm is derived from the code rather than transcribed
from the page:
test_the_fit_paragraph_states_the_code_clamp_windowreads every[lo, hi]out of theparagraph and requires the set to equal
(TEMP_MIN, TEMP_MAX)imported fromlaya/common.py, plus theMIN_TYPE_Ncount;test_the_fit_paragraph_keeps_the_inherited_temperature_on_a_failed_fitparses thenotebook's JSON, slices the rank-0 block between its marker comment and
destroy_process_group(), andrequires
fitted_temps = None, the page's matching phrase, no1.2in the block, and the pop to sitafter the
if fitted_temps is not None:guard.Five mutants, each applied to the committed tree and reverted before the next:
[0.1, 10]rc=1 KILLEDTEMP_MIN = 0.5->0.1(page untouched)rc=1 KILLEDMIN_TYPE_N = 10->12(page untouched)rc=1 KILLED1.2rc=1 KILLEDrc=1 KILLEDThe second and third arms are the point: a future change to the constants fails the gate even though
the page still looks self-consistent, which is exactly the drift that produced this bug.
Gates, run in this worktree:
Suite count goes 497 -> 500: the three new tests are the addition.
zensicalis not installed in thisvenv, so the
docs (strict build)job is the first time this page is built; the change adds no link,no nav entry and no heading, and no file under
docs/was renamed.Diff scan for secrets —
api[_-]?key|secret|token|password|bearer|sk-[A-Za-z0-9]|ghp_|AKIA|hf_[A-Za-z0-9]|private[_-]?keyover
git diff HEAD~1: 0 hits.Deconfliction:
docs/finetune.mdandtests/test_finetune_entrypoints.pyare touched by no open PR.The two files the gate reads are —
laya/common.py(#921, #783, #687) andlaya/calibrate.py(#1126,#1124) — but this PR edits neither, and
gh pr diffover all five matchesTEMP_MIN/TEMP_MAXonly in#783, where it is an added
clamp_temperatureimport line, not a bounds change.Checklist
main