Repository navigation
docs(finetune): add the #963 positive-control and multi-seed diagnostic - #1016
Conversation
|
Merged. The positive-control trick is the most transferable thing in #963, and it belongs in the guide rather than in an issue comment. Prepending an arbitrary tag that encodes the label gives you a task the model can trivially read, so a run that cannot learn that tells you the budget is wrong rather than the task is hard. I have used it myself to verify #899, #931 and #967, each time to separate "not enough updates" from "not learnable". The multi-seed note earns its place for the same reason: 9 of 15 runs left the ln K plateau and 6 never did, on identical data and configuration. A default that works two thirds of the time is a worse failure than one that never works, because the two thirds teach you to trust it. Good pairing with #1014 in the same release, which prints the update budget before the run. Together they turn #963's diagnosis into something a user can act on before spending the compute. |
What
Adds one section to
docs/finetune.md, immediately after the calibration/collapse-warning discussion and before### Options:### Diagnose a run that stays at chance— a label-encoding positive control (prepend a tag that spells the answer) to separate "budget/optimization is wrong" from "the task or representation is hard", plus the seed-dependence of takeoff near the ln K plateau.It states the measured facts and nothing more: #963 saw the positive control reach 1.00 at 8 epochs while the real task still collapsed, and 9 of 15 runs left the plateau while 6 never did within 32 epochs under identical data and configuration. The section explicitly says a passing positive control is not proof that the real task is learnable or that the checkpoint generalizes, and it does not prescribe a universal seed count.
Why
This implements proposal 4 from #963, which Nandha's latest comment on that issue leaves unclaimed and says belongs in
docs/finetune.md:Use case: a fine-tune that finishes and reports calibration while sitting at the class prior currently has no documented way to tell a too-small budget from an unlearnable task;
--dry-runand the collapse warning describe the failure but not this diagnosis.How it was verified
No issues found, nogriffe:lines. Local zensical is 0.0.64;requirements-docs.txtpins 0.0.67, so CI may render slightly differently but the doc has no theme or reference changes.docs/finetune.mdis feat(train): print the optimizer-update budget before a run #1014 (proposal 3, optimizer-update budget), which inserts its### Sizing a run in optimizer updatessection between the skip-reason table and### What a run writes; this section goes after### What a run writes, so the two hunks do not overlap and the topics stay separate adjacent subsections. docs(finetune): worked example for fine-tuning the multilingual checkpoint on CPU #705 touches onlydocs/finetune_multilingual_cpu.mdand nav, feat(train): add early stopping, disjoint validation split, and budget-independent schedules (#963) #982 only train code and tests.Checklist
main(073afac8)Proposal 4 only: no training code, no new CLI flag, no #982 early-stopping semantics, no benchmark, and proposal 3 is not bundled.