Skip to content

refactor(finetune): converge the Kaggle DDP path on shared training primitives - #1017

Merged
NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:yu344-ddp-convergence
Oct 8, 2026
Merged

NandhaKishorM merged 2 commits into
NandhaKishorM:mainfrom
Bruce-Yii:yu344-ddp-convergence

Conversation

@Bruce-Yii

@Bruce-Yii Bruce-Yii commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

What

Converge the Kaggle 2×T4 DDP notebook (notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) onto the current shared training primitives:

  • item construction/preprocessing → laya.train.items_from_rows
  • batch encoding/collation → laya.train.encode_item + laya.common.collate_items
  • training objective → laya.train.rlcd_loss
  • calibration holdout → laya.train.split_calibration
  • sigma schedule → laya.train.sigma_at
  • temperature fitting → laya.calibrate.fit_temperature_map (via laya.train.calibration_records)
  • checkpoint/config persistence → laya.train.save_checkpoint

DDP-specific orchestration stays in the notebook: process-group and rank/local-rank setup, DDP wrapping, equal-rank trim + sharding (the #678 deadlock guard), effective-batch accounting, per-rank shuffle and rolling checkpoint_latest metadata.

Two existing gates are updated for the same change, because they read this notebook:

  • scripts/test_suites.py — registers tests/test_kaggle_ddp_convergence.py so CI runs it (every suite in tests/ must be wired; tests/test_packaging.py enforces that).
  • tests/test_calibration_persistence.py — export_notebook_config now slices the notebook's out_cfg block instead of the retired cfg["fine_tuned"] block, and the clamp sweep is redirected as its own failure message asks once no entry point hand-rolls a fit: it pins that the notebook reaches fit_temperature_map and owns no private fit of its own, and drives that canonical fitter with the same peaked/dipped inputs so the [TEMP_MIN, TEMP_MAX] bound is still asserted rather than the sweep matching nothing.

Adds tests/test_kaggle_ddp_convergence.py, a weight-free guard that fails if the private collate / temperature fitter / item builder come back, or if the canonical wiring is no longer in place.

Why

#887 Point 1. #965 converged the two single-process wrappers onto laya.train and deliberately left the Kaggle/DDP path separate; #1007 then named that path as the remaining piece.

The notebook was the last copy of the training loop. Fixes to the calibration or training contract had to be re-implemented by hand here — #851 and #885 were the same temperature-clamp bug in two of the copies. This is not a proposal to replace DDP with single-process finetune(); it is so that changes to the canonical contract reach the distributed notebook instead of leaving a private copy behind.

Intentional behavior change

This is not a bit-for-bit replay of the retired loop.

  • The old notebook fitted only private per-type scalar temperatures. The converged path uses the current canonical fit_temperature_map, so it follows today's calibration artifact contract and will persist temperature_by_options when a bucket has enough evidence. In the 400-item validation run no option-count bucket met MIN_BUCKET_N=2000, so none was produced.
  • Gradient scaling now follows the canonical partial accumulation-window rule in train_model (divide by the window's own size) rather than the retired loop's flat /GRAD_ACCUM.

How it was verified

Weight-free gates:

python tests/test_kaggle_ddp_convergence.py                 # Ran 9 tests ... OK
python -m pytest tests/test_train.py tests/test_finetune_entrypoints.py -q   # 58 passed
python tests/test_calibration_persistence.py                # Ran 7 tests ... OK
python tests/test_packaging.py                              # 138 passed, 0 failed
python tests/test_calibrate.py                              # 230 passed, 0 failed
python tests/test_training.py                               # 61 passed, 0 failed
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed
python -m compileall -q laya/ tests/

Two gates had to be updated for the converged script, both found by CI rather than by inspection: the new suite is registered in scripts/test_suites.py (a test outside that list is not run by CI at all), and tests/test_calibration_persistence.py — which executes the notebook's own config-export block instead of re-implementing it — slices the out_cfg block and redirects its clamp sweep to the canonical fitter, exactly as that sweep's message asks for once no entry point hand-rolls a fit.

tests/test_calibration_persistence.py also reports 7 tests OK, the same as 073afac8 — the notebook-export gate reads the converged block rather than the retired one.

Real 2×T4 end-to-end (Kaggle kernel yuyi722333/yu344-ddp-convergence-validate, machine_shape=NvidiaTeslaT4, 2 visible Tesla T4s):

torchrun --standalone --nproc_per_node=2 /kaggle/working/train_ddp.py <model_dir> <output_dir>
  • both ranks entered and finished; 4 epochs completed, no hang, no unmatched all-reduce, no traceback;
  • 6000 items → 400 held out for calibration before sharding → 5600 train → 2800 per rank, 350 micro-batches per rank per epoch;
  • effective batch = 8 × 2 × 4 = 64; 350 % 4 = 2 exercised the partial accumulation window;
  • fitted temperatures [1.104, 1.110, 1.137], finite and within [TEMP_MIN, TEMP_MAX];
  • the script that ran was byte-identical to the notebook cell (fetched back from the run and diffed);
  • final checkpoint loaded through laya.Agent and completed 2,000 real predictions on the official test split: accuracy 0.7725, Brier 0.0517, ECE 0.1507 (published checkpoint: 0.766; same split/seed comparison point).

Checklist

  • Focused on one change (split unrelated work into another PR)
  • Rebased on the latest main
  • Tests pass locally
  • Docs or examples updated when the public API changed — no public API changed

…aining primitives

The Kaggle 2xT4 notebook was the last training copy with its own item construction, collate, RLCD loss, calibration holdout and temperature fitter. Delegate those to laya.train / laya.common / laya.calibrate. DDP orchestration (process group, rank wiring, DDP wrap, equal-rank trim and shard, effective-batch accounting, rolling checkpoint meta) stays notebook-private. Calibration moves to the canonical artifact contract: fit_temperature_map also persists temperature_by_options when fitted, unlike the old per-type-scalar-only private fitter.
@Bruce-Yii
Bruce-Yii force-pushed the yu344-ddp-convergence branch 3 times, most recently from 5087cf8 to 61c1455 Compare October 7, 2026 13:22
…training copies

Adds tests/test_kaggle_ddp_convergence.py and registers it in scripts/test_suites.py so CI runs it: the DDP cell keeps the canonical helpers wired in and keeps no private collate / temperature fitter / item builder, the calibration holdout stays before sharding, and split_calibration / collate_items / sigma_at / fit_temperature_map keep the contract the notebook relies on.

Adapts tests/test_calibration_persistence.py to the converged export: export_notebook_config now slices from the out_cfg block instead of the retired cfg["fine_tuned"] block, and the clamp sweep is redirected as its own message asks -- it now pins that the notebook reaches fit_temperature_map and owns no fit of its own, and drives that canonical fitter with the same peaked/dipped inputs so the [TEMP_MIN, TEMP_MAX] bound stays asserted instead of the sweep matching nothing.
@NandhaKishorM
NandhaKishorM merged commit 689579b into NandhaKishorM:main Oct 8, 2026
30 of 34 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Merged. This finishes Point 1 of #887, which is the item with the strongest argument in that whole issue: #851 and #885 were the same temperature-clamp bug found and fixed twice, in parallel, because the fit existed in three copies.

With #965 last release and this one, the three are now one. The notebook keeps what is genuinely DDP-specific, which is the right split: process-group and rank setup, DDP wrapping, the equal-rank trim and sharding that guards #678's deadlock, effective-batch accounting and the rolling checkpoint_latest. None of that belongs in a shared primitive, and everything that does is now imported rather than re-implemented.

tests/test_kaggle_ddp_convergence.py is what keeps it converged. A refactor that is not gated drifts back the first time someone debugs the notebook in a hurry, and a guard that fails if the private collate, temperature fitter or item builder reappear is the right shape for that. Registering it in scripts/test_suites.py in the same PR is the detail most people miss, and tests/test_packaging.py would have caught it.

The tests/test_calibration_persistence.py change deserves a note, because it is the one place this could have quietly weakened a gate. That suite's clamp sweep existed to catch a hand-rolled fit outside [TEMP_MIN, TEMP_MAX]. Once no entry point hand-rolls one, a sweep over notebook-local code matches nothing and silently passes. Redirecting it to pin that the notebook reaches fit_temperature_map and owns no private fit, then driving that canonical fitter with the same peaked and dipped inputs, keeps the bound asserted rather than retiring it. That is exactly what its own failure message asked for.

All 85 suites pass on the merged tree.

NandhaKishorM added a commit that referenced this pull request Oct 8, 2026
35 pull requests from 8 contributors.

* LAYA_EXTRA_MODELS registers extra checkpoints on laya-serve (#1047, #919)
* fine-tuning prints its optimizer-update budget before a run (#1014, #963)
* the Kaggle DDP notebook converges on the shared training primitives (#1017, #887)
* evals writes redirected output as utf-8 (#1045)
* the server serves again after a lifespan restart (#1049)
* Java router lifecycle hooks, plus an OSV dependency scan (#1046)
* twenty-four documentation corrections, each verified against the code

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants