Skip to content

docs(finetune): add the #963 positive-control and multi-seed diagnostic - #1016

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Bruce-Yii:docs/finetune-positive-control-multi-seed
Oct 8, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
Bruce-Yii:docs/finetune-positive-control-multi-seed

Conversation

@Bruce-Yii

Copy link
Copy Markdown
Contributor

What

Adds one section to docs/finetune.md, immediately after the calibration/collapse-warning discussion and before ### Options:

  • ### Diagnose a run that stays at chance — a label-encoding positive control (prepend a tag that spells the answer) to separate "budget/optimization is wrong" from "the task or representation is hard", plus the seed-dependence of takeoff near the ln K plateau.

It states the measured facts and nothing more: #963 saw the positive control reach 1.00 at 8 epochs while the real task still collapsed, and 9 of 15 runs left the plateau while 6 never did within 32 epochs under identical data and configuration. The section explicitly says a passing positive control is not proof that the real task is learnable or that the checkpoint generalizes, and it does not prescribe a universal seed count.

Why

This implements proposal 4 from #963, which Nandha's latest comment on that issue leaves unclaimed and says belongs in docs/finetune.md:

On proposal 4: the positive-control trick is the most transferable thing in your report. Prepending an arbitrary tag that encodes the label, so that a run which cannot learn a label the model can trivially read tells you the budget is wrong rather than the task is hard, is a diagnostic I have since used myself for exactly this purpose when verifying #899, #931 and #967. It belongs in docs/finetune.md ...

Use case: a fine-tune that finishes and reports calibration while sitting at the class prior currently has no documented way to tell a too-small budget from an unlearnable task; --dry-run and the collapse warning describe the failure but not this diagnosis.

How it was verified

python -m zensical build --strict --clean

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main (073afac8)
  • Tests pass locally
  • Docs or examples updated when the public API changed — this is a docs-only change; no code, CLI flag, default, or scheduler touched

Proposal 4 only: no training code, no new CLI flag, no #982 early-stopping semantics, no benchmark, and proposal 3 is not bundled.

@NandhaKishorM
NandhaKishorM merged commit 2577f93 into NandhaKishorM:main Oct 8, 2026
32 of 36 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged.

The positive-control trick is the most transferable thing in #963, and it belongs in the guide rather than in an issue comment. Prepending an arbitrary tag that encodes the label gives you a task the model can trivially read, so a run that cannot learn that tells you the budget is wrong rather than the task is hard. I have used it myself to verify #899, #931 and #967, each time to separate "not enough updates" from "not learnable".

The multi-seed note earns its place for the same reason: 9 of 15 runs left the ln K plateau and 6 never did, on identical data and configuration. A default that works two thirds of the time is a worse failure than one that never works, because the two thirds teach you to trust it.

Good pairing with #1014 in the same release, which prints the update budget before the run. Together they turn #963's diagnosis into something a user can act on before spending the compute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants