Skip to content

feat(train): add laya.train, one fine-tuning loop for the notebook and scripts to share - #899

Merged
NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
GuilhermeFusari:feat/train-module
Oct 4, 2026
Merged

NandhaKishorM merged 1 commit into
NandhaKishorM:mainfrom
GuilhermeFusari:feat/train-module

Conversation

@GuilhermeFusari

Copy link
Copy Markdown
Contributor

What

First step of #887. Adds laya/train.py, a single training loop the notebook and the two scripts can later share. The existing scripts are not changed here, because #851 and #885 are open against them and I didn't want to conflict with either.

  • TrainConfig: the notebook's training as the default (loss="rlcd", options in their listed order, effective batch 64 as micro_batch=8, grad_accum=8, the same learning rates, cosine schedule, grad clip, sigma annealing and reward weights).
  • loss="rlcd" (default) is the notebook objective: a GRPO term over noisy logits rewarded by proper_reward, plus soft cross-entropy. loss="soft-ce" trains on the soft cross-entropy alone, the variant Controlled A/B: GRPO + proper-reward term shows no measurable gain over (soft) cross-entropy in the fine-tune loop #741 tested.
  • Option-order augmentation, off by default. Each epoch, items whose type is in shuffle_options (for example ("choice",)) are re-encoded with a random option_order through build_sequence, the same path inference uses, and the target is permuted to match (target[option_order[s]] in slot s). Items keep the question and the tokenized state, and states are tokenized once per row and shared by its questions.
  • Calibration with laya.calibrate.fit_temperature_map on a slice held out before training, so the fine-tuned config uses the runtime clamp and buckets. An inherited temperature_by_options is removed unless the run refits it.
  • Questions whose options collapse into the same token span under the head budget (Should option-token identity collapse after head truncation be reported? #538), or whose target length does not match the option count, are counted as skipped instead of trained on.
  • The max_len / head_max_len used during training are written into the saved config, together with the TrainConfig under training.laya_train.
  • finetune(data, model_dir, output_dir, config) runs the whole thing on a {state, questions, gold} JSONL and a local checkpoint directory, writes checkpoint_latest/ after every epoch, and saves the final checkpoint in the layout laya.load reads.

No new dependency. Nothing is exported from laya/__init__.py, so the public API contract in tests/test_hooks_api.py is unchanged.

Why

The training loop lives in three copies that drift (#851 and #885 are the same temperature-clamp fix in two of them), and none of them is tested. With the loop in the package, the scripts can call one implementation, and the next PR in #887 can add a laya-train CLI and plain labelled-data input on top of it.

How it was verified

Unit tests

tests/test_train.py is weight-free: a one-layer BERT and a word-level tokenizer built in a temp directory, with downloads patched to fail. It covers:

  • gold → target conversion for the three question types, including list-form choices, non-string labels, and refusal of negative or non-finite probabilities;
  • questions rendered exactly as Agent renders them (six shapes, including custom noul labels), so training sees the option text inference shows;
  • shared state tokens and skip accounting;
  • canonical encoding being identical to build_sequence;
  • a permuted item having its options in the new slots, with the target permuted the same way;
  • shuffling applying only to the configured types, deterministically per seed;
  • a disjoint, seeded calibration split;
  • soft_ce_loss against a hand computation, including padded options;
  • rlcd_loss giving the same value and gradient as the loss in finetune_single_device.py under the same RNG state;
  • the defaults matching the notebook (rlcd, no shuffling, effective batch 64);
  • config validation;
  • skipped questions being reported by reason;
  • a frozen encoder running in eval mode;
  • end-to-end runs (soft-CE, rlcd, shuffled, frozen encoder) that save a checkpoint laya.load opens and answers with.

The suite is added to scripts/test_suites.py and to the Windows loop in ci.yml.

ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # passes
python -m compileall -q laya/ tests/                                   # passes
python tests/test_train.py            # 23 tests OK
python tests/test_training.py         # 61 passed
python tests/test_packaging.py        # passes
python tests/test_calibration_persistence.py   # passes
python tests/test_router.py           # 703 passed

Results on typed-decisions (full fine-tune)

Run on Kaggle (2x T4, one run per GPU) with a notebook that installs laya==0.3.24 and adds this branch's laya/train.py. Setup for every run:

  • base convaiinnovations/laya, the full train split (1,200 cases: 5,600 training items + 400 calibration items);
  • 4 epochs, micro-batch 8 x 8 accumulation (effective 64), encoder lr 2.5e-5, head lr 1e-4;
  • fp16 autocast, gradient checkpointing, max_len 1024, head_max_len 256, seed 0.

Evaluation covers all 400 test cases (2,000 decisions), scored like cell 14 of the published notebook. "Reversed" means the same choice questions answered with the options in reverse order (option_order), and "flip" is the share whose answer changes.

run accuracy choice noul score choice, reversed flip ECE Brier
base laya 0.362 0.290 0.487 0.323 0.257 0.357 0.174 0.329
loss="rlcd", no shuffling 0.7715 0.745 0.840 0.740 0.740 0.065 0.150 0.050
loss="soft-ce", no shuffling 0.790 0.757 0.870 0.755 0.757 0.078 0.157 0.047
loss="soft-ce", shuffle_options=("choice",) 0.7875 0.767 0.855 0.753 0.757 0.062 0.159 0.048
published laya-typed-decisions 0.766 0.213 0.061
Jev 1.13.0 (published) 0.727 0.144 0.148

Per workflow:

run agent trace customer service invoice security
base 0.390 0.382 0.360 0.316
rlcd 0.732 0.774 0.814 0.766
soft-CE 0.750 0.786 0.850 0.774
soft-CE + shuffling 0.764 0.774 0.840 0.772
published 0.730 0.764 0.804 0.766

Paired differences on the same 2,000 decisions, with 95% bootstrap intervals over test cases:

comparison accuracy flip rate
soft-CE − rlcd +0.0185 [+0.0035, +0.034] +0.013 [−0.010, +0.038]
soft-CE + shuffling − soft-CE −0.0025 [−0.015, +0.0095] −0.017 [−0.040, +0.007]

What I take from it:

  • The base row matches the published 0.362, and loss="rlcd" lands at 0.7715 against the published 0.766, so the module reproduces the notebook.
  • Soft cross-entropy alone is 1.85 points better than the notebook objective on the same split, which is consistent with Controlled A/B: GRPO + proper-reward term shows no measurable gain over (soft) cross-entropy in the fine-tune loop #741. I left rlcd as the default so TrainConfig() trains exactly like the notebook; switching the default, or retraining the published checkpoint with soft-ce, is your call.
  • Option shuffling does not change accuracy measurably once the encoder trains. Full fine-tuning by itself already brings the flip rate from 0.357 to 0.06–0.08 on these 4-option questions, and the extra drop from shuffling is not significant. It stays opt-in; it may matter more with many options (the 20-option numbers in BENCHMARKS.md), which I haven't measured.
  • ECE is 0.15–0.16 for all three runs, lower than the 0.213 listed for the published checkpoint but not below Jev's 0.144.
  • This is one seed. Time on one T4 for 4 epochs: about 45 minutes for the rlcd run alone. The two soft-CE runs went side by side on the two GPUs.

Smaller checks on my own machine (GTX 1650, 4 GB):

  • A full fine-tune on CPU (20 cases, 1 epoch) ran end to end: the encoder weights changed, and the checkpoint loads with laya.load.
  • Head-only runs (freeze_encoder=True) on the GPU ran end to end.
  • fp16 autocast was about 5x slower than fp32 on that card, which is why amp is a config option rather than always on for CUDA.

Not in this PR

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed (no public API change: nothing is exported from laya/__init__.py)

Refs #887

🤖 Generated with Claude Code

…d scripts to share

The training loop lives in three drifting copies (the Kaggle notebook, the MPS script and
research/scripts/finetune_single_device.py). This adds it to the package once, as the first
step of NandhaKishorM#887. TrainConfig() trains the way the published notebook does: loss="rlcd", options
in their listed order, and an effective batch of 64 (micro_batch=8 x grad_accum=8).

- loss="soft-ce" trains on the soft cross-entropy alone (the variant NandhaKishorM#741 tested).
- shuffle_options re-encodes items of the named types with a random option_order each epoch,
  through the same path inference uses, and permutes the target to match. Off by default.
- Questions go through Agent._check_question and Agent._to_internal, so training sees the
  option text inference shows (list-form choices, custom noul labels, non-string instructions).
- Calibration uses laya.calibrate.fit_temperature_map, so the fit follows the runtime clamp and
  buckets; an inherited temperature_by_options is dropped unless refitted.
- Questions that cannot be trained on are skipped and counted by reason (invalid_question,
  invalid_target, target_mismatch, options_collapsed per NandhaKishorM#538).
- The max_len / head_max_len used in training are written into the saved config, with the
  TrainConfig under training.laya_train.
- freeze_encoder trains the head only and keeps the encoder in eval mode.

The existing scripts are left untouched in this change; NandhaKishorM#851 and NandhaKishorM#885 are open against them.
tests/test_train.py is weight-free (one-layer BERT, word-level tokenizer, no downloads) and is
registered in the shared suite list and the Windows CI loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@GuilhermeFusari

Copy link
Copy Markdown
Contributor Author

Workflows here are waiting for approval, so I ran the same CI on my fork: all green (CI 14/14 jobs, including the Windows and py3.10–3.13 test jobs, plus Docker, Docs and Security): https://github.com/GuilhermeFusari/laya/actions/runs/37129213237

@NandhaKishorM
NandhaKishorM merged commit 5e5cec5 into NandhaKishorM:main Oct 4, 2026
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. I ran a real fine-tune with it rather than relying on the unit tests, because a training loop that passes weight-free tests and does not learn is the failure mode that matters here.

It trains

A task the base checkpoint cannot do zero-shot: three options with arbitrary names (tier_a / tier_b / tier_c) and descriptions carrying no signal about the text, so the mapping has to be learned. 180 training rows, 30 held-out eval rows, the cached english checkpoint, CPU, 4 epochs, micro_batch=4, grad_accum=2.

checkpoint held-out accuracy
base convaiinnovations/laya 10/30 = 0.333 (chance is 0.333)
fine-tuned, loss="rlcd" 30/30 = 1.000
fine-tuned, loss="soft-ce" 30/30 = 1.000

The output directory loads with laya.load() and returns an Agent, in the layout you documented (model.safetensors, rl_agent_config.json, encoder/, tokenizer/, plus checkpoint_latest/). skipped was empty on every run, so nothing was silently dropped by the head-budget or target-length checks.

Option shuffling works, and this is the result I was most interested in

#887's point 3 is that options are never shuffled during training and BENCHMARKS.md suggests that as the fix for order instability. Measured it directly: for each eval row, permute the three options three ways and count how often the answer changes.

run answer changed under reorder
shuffle_options=() (default) 14/90
shuffle_options=("choice",) 0/90

So the augmentation does what the issue predicted, on real weights. That is a strong argument for the feature existing, and it is also why I am glad it defaults to off: it changes what the model learns, so it should be a decision rather than a surprise.

On the design

Not touching the three existing scripts was the right call given #851 and #885 were open against two of them, and it is why this merged in one pass. Routing the re-encode through build_sequence(..., option_order=...), the same path inference uses, rather than a second shuffling implementation, is the detail that makes the augmentation trustworthy. So is permuting the target to match in the same place.

Using laya.calibrate.fit_temperature_map instead of a fourth local LBFGS copy addresses #887's point 5, and dropping an inherited temperature_by_options unless the run refits it is the subtle part: carrying a parent's option-count buckets onto a model whose option distribution changed would be worse than having none.

One observation rather than a request: on my small run the fitted temperatures came back [1.0, 1.0, 1.0] with temperature_by_options: {} and n_by_bucket: {"choice:3-5": 18}. That is plausible for a model that fits its training distribution perfectly on 18 calibration rows, but it does mean a short run produces a checkpoint with no useful calibration, and the report is the only place that says so. A line in the report when every bucket lands at the clamp midpoint, or a warning below some calib_max, would save someone trusting a flat temperature.

Verified

tests/test_train.py 23 tests pass; the shared suite list reports all 80 suites passed, up from 79, so your suite is wired in; and a decision diff against 0.3.27 on the real english and multilingual checkpoints over 74 answers shows zero decision flips and a max probability delta of 0.000000, which is what a new module that nothing imports by default should show.

This is the first step of #887 and it is the one that needed to exist before the CLI and the CSV loader have anywhere to live. Thank you.

NandhaKishorM pushed a commit that referenced this pull request Oct 5, 2026
From the review of #899: a short run can fit temperatures of [1.0, 1.0, 1.0] from a handful of
calibration items and save a checkpoint whose confidences were never calibrated, with only the
returned numbers to show it.

calibration_report() describes, per question type, the calibration items the fit used (count,
accuracy and mean confidence at the fitted temperature) and lists what to doubt:

- fewer than MIN_TYPE_N items, so the temperature was not fitted and stays 1.0;
- fewer than CALIB_WARN_N (50) items, so it was fitted on little evidence;
- a temperature on the [TEMP_MIN, TEMP_MAX] clamp;
- a fit that came back unchanged at 1.0, which usually means the items gave it nothing to
  correct.

finetune() raises a RuntimeWarning for each issue, returns the report under "calibration" and
saves it in the checkpoint config under training.laya_train_calibration. A type with no
calibration items is reported without an issue, since it was not in the data. Nothing changes
in what is trained or fitted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants