Skip to content

docs(finetune): the temperature fit clamps to [0.5, 5.0], and a failed fit keeps the inherited value - #1147

Open
aashish254 wants to merge 2 commits into
NandhaKishorM:mainfrom
aashish254:docs-finetune-clamp-window
Open

aashish254 wants to merge 2 commits into
NandhaKishorM:mainfrom
aashish254:docs-finetune-clamp-window

Conversation

@aashish254

Copy link
Copy Markdown
Contributor

What

docs/finetune.md's post-training calibration paragraph promised three things the code does not do. It
said the fitted per-type temperatures are "clamped to [0.1, 10] (1.0 for a slice under ten items,
1.2 if the fit raises)". Two numbers are wrong and one does not exist:

the page said what the code does
clamped to [0.1, 10] clamped to [TEMP_MIN, TEMP_MAX] = [0.5, 5.0] (laya/common.py:824-825), applied by fit_one_temperature's last line (laya/calibrate.py:138)
1.2 if the fit raises the notebook's except branch sets fitted_temps = None (…2xT4_kaggle.ipynb:324-326), so out_cfg["temperature"] is never written and the inherited checkpoint temperature ships; the temperature_by_options pop sits inside if fitted_temps is not None: (:331, :334), so a failed fit keeps the inherited buckets too
1.0 for a slice under ten items correct, and now named by its constant: _fit_groups only fits when len(pairs) >= MIN_TYPE_N (laya/calibrate.py:239-240), and MIN_TYPE_N = 10 (:39)

The page's own next sentence ("removes any inherited temperature_by_options in the same write") was
true only on the success path and read as unconditional, so it moved inside that qualifier. The new
text:

After the last epoch, rank 0 fits **one temperature per question type** (`choice`, `score`,
`noul`) by LBFGS on the log-temperature, clamped to `[0.5, 5.0]` — the same
`laya.common.TEMP_MIN`/`TEMP_MAX` window checkpoint load applies, so a fitted value and a
loaded one are bounded alike. A type with fewer than `MIN_TYPE_N` (10) items in the slice
keeps the neutral `1.0`, and one the optimiser wants to sharpen below the floor or flatten
past the ceiling gets the bound rather than its own number. If the fit raises, rank 0 keeps
the **inherited checkpoint temperature**: the notebook catches the failure and skips the
write below, so a run that could not calibrate ships what it started with instead of a
partially calibrated config. On that success path the values go into `rl_agent_config.json`
as `temperature`, and the same write **removes any inherited `temperature_by_options`**, and
only there: those old bucket values take precedence at inference and would silently mask the
new fit.

11 insertions and 4 deletions in the page; 100 lines of new gate in tests/test_finetune_entrypoints.py.

Why

This is the paragraph a fine-tuner reads right before they gate a production decision on the
checkpoint's confidence. README.md:1559 already documents [0.5, 5.0] for checkpoint load, so the
page was the only copy in the repo claiming a window twice as wide — and 0.1/10 are not bounds at
all, they are LBFGS's lr and max_iter from the line above the clamp (laya/calibrate.py:128,
torch.optim.LBFGS([log_t], lr=0.1, max_iter=100)),
which is how the sentence likely got written. A reader sizing a threshold off [0.1, 10] believes a fit
can be ~5x sharper than the floor actually allows.

The 1.2 half is worse because it changes what a failed run means. Nothing in laya/ fits a failure
to 1.2; that value exists only as a per-question-type no-data default in two other wrappers
(research/scripts/finetune_single_device.py:91, notebooks/laya_finetune_typed_decisions_mps.py:136),
keyed on entry.get("items") == 0. In the path this page describes, a raising fit leaves the inherited
temperature in place — so a run that never calibrated is indistinguishable, in its config, from a run
that inherited and shipped it. Expecting 1.2 means believing something was fitted when nothing was.

How it was verified

Read-only probe against this tree, no CUDA and no training run needed:

$ PYTHONDONTWRITEBYTECODE=1 PYTHONPATH="$PWD" ../laya/.venv/bin/python -B /tmp/probe_fit_contract.py
laya resolved to: /Users/aashish/github contribution /GIT/laya-finetune-clamp
laya/common.py TEMP_MIN, TEMP_MAX = 0.5 5.0
laya/calibrate.py MIN_TYPE_N = 10
fit on 40 always-correct, lightly separated items -> 0.5
fit on 40 half-wrong, peaked items               -> 5.0
clamp_temperature(0.1) -> 0.5 | clamp_temperature(10.0) -> 5.0
fit_temperature_map on 9 items -> temperature [1.0, 1.0, 1.0]
fit on non-finite logits -> raises ValueError: logits must contain only finite values
laya/{calibrate,train,common,agent}.py: lines matching 1.2 near temp -> none (all four)
notebook rank-0 fit block, the lines that decide the failure path:
            try:
                fitted_temps = fitted["temperature"]
            except Exception as e:
                fitted_temps, fitted = None, {"temperature_by_options": {}}
            if fitted_temps is not None:
                out_cfg["temperature"] = fitted_temps
                out_cfg.pop("temperature_by_options", None)
'1.2' occurrences in that block: 0

Both bounds are reachable, so the window is a real clamp and not decoration: an always-correct slice
lands the fit on the 0.5 floor, a half-wrong peaked slice lands it on the 5.0 ceiling.

The gate is three tests plus the ban, and every arm is derived from the code rather than transcribed
from the page: test_the_fit_paragraph_states_the_code_clamp_window reads every [lo, hi] out of the
paragraph and requires the set to equal (TEMP_MIN, TEMP_MAX) imported from laya/common.py, plus the
MIN_TYPE_N count; test_the_fit_paragraph_keeps_the_inherited_temperature_on_a_failed_fit parses the
notebook's JSON, slices the rank-0 block between its marker comment and destroy_process_group(), and
requires fitted_temps = None, the page's matching phrase, no 1.2 in the block, and the pop to sit
after the if fitted_temps is not None: guard.

Five mutants, each applied to the committed tree and reverted before the next:

mutant result
page window back to [0.1, 10] rc=1 KILLED
TEMP_MIN = 0.5 -> 0.1 (page untouched) rc=1 KILLED
MIN_TYPE_N = 10 -> 12 (page untouched) rc=1 KILLED
page fallback -> 1.2 rc=1 KILLED
page reverted to the pre-fix sentence rc=1 KILLED

The second and third arms are the point: a future change to the constants fails the gate even though
the page still looks self-consistent, which is exactly the drift that produced this bug.

Gates, run in this worktree:

$ PYTHONDONTWRITEBYTECODE=1 ../laya/.venv/bin/python -B scripts/test_suites.py
FAIL: 0            (x4, no non-zero FAIL line)
500 passed, 1 skipped, 5 warnings in 16.10s
all 87 suites passed
$ ../laya/.venv/bin/python -B -m pytest tests/test_finetune_entrypoints.py -q
11 passed in 0.41s
$ ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
All checks passed!
$ ../laya/.venv/bin/python -B -m compileall -q laya/ tests/
(exit 0, no output)

Suite count goes 497 -> 500: the three new tests are the addition. zensical is not installed in this
venv, so the docs (strict build) job is the first time this page is built; the change adds no link,
no nav entry and no heading, and no file under docs/ was renamed.

Diff scan for secrets — api[_-]?key|secret|token|password|bearer|sk-[A-Za-z0-9]|ghp_|AKIA|hf_[A-Za-z0-9]|private[_-]?key
over git diff HEAD~1: 0 hits.

Deconfliction: docs/finetune.md and tests/test_finetune_entrypoints.py are touched by no open PR.
The two files the gate reads are — laya/common.py (#921, #783, #687) and laya/calibrate.py (#1126,
#1124) — but this PR edits neither, and gh pr diff over all five matches TEMP_MIN/TEMP_MAX only in
#783, where it is an added clamp_temperature import line, not a bounds change.

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed

…d fit keeps the inherited value

The page promised a "[0.1, 10]" clamp and a "1.2 if the fit raises" fallback; the fit
clamps through clamp_temperature(fitted, TEMP_MIN, TEMP_MAX) = (0.5, 5.0), and the
notebook it describes catches the failure and leaves the checkpoint temperature
untouched. Gate both numbers against laya/common.py and laya/calibrate.py, and the
failure path against the notebook's own rank-0 block.
@aashish254

Copy link
Copy Markdown
Contributor Author

Short answer: the one red check on this PR is a Windows runner crash in a suite this PR does not touch, and two other open PRs hit the identical crash today.

What the job printed (tests (windows), run 38185901630, step "Routing and language-detection tests"):

13 passed, 0 failed
all lazy-import tests passed
D:\a\_temp\d0f73b21-43d3-493e-ac4d-2a66fb04c794.sh: line 1:   547 Illegal instruction        python "tests/$t.py"
##[error]Process completed with exit code 132.

The loop at .github/workflows/ci.yml:244 runs test_lazy_import and then test_runtime_fixes, so the crashing process is tests/test_runtime_fixes.py — ~3 seconds in, before it prints anything.

Same signature on two other branches today, both unrelated to docs:

  • run 38171676353 / job 114565406185, head 2ff91a99 (fix-agreement-abstention-vocabulary), 2026-10-11T20:23:27Z: 1287 Illegal instruction python "tests/$t.py", exit 132, again right after the "13 passed" lazy-import line.
  • run 38166451685 / job 114549898121, head 35a9b9f9 (Ha1baraA11:pr/laya-171), 2026-10-11T19:07:05Z: 888 Illegal instruction, exit 132, same position in the loop.

Against that, the same commit is green everywhere else (37 checks pass), and my two neighbouring PRs #1143 (1a16cabe) and #1148 (c3b0d312) each show pass x38 skipping x2 including tests (windows).

This PR changes docs/finetune.md and tests/test_finetune_entrypoints.py. The finetune-entrypoints suite is not in the Windows loop; it runs through scripts/test_suites.py:117, which passed on the Linux lanes here. So nothing in the diff reaches the crashing process.

I can't re-run the job myself (gh run rerun returns "Must have admin rights to Repository"), so I'm leaving the commit as-is rather than pushing a no-op to force a fresh run. Happy to have someone re-run tests (windows) on this head; if it goes green that confirms the flake.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant