Repository navigation
refactor(finetune): converge the Kaggle DDP path on shared training primitives - #1017
Conversation
…aining primitives The Kaggle 2xT4 notebook was the last training copy with its own item construction, collate, RLCD loss, calibration holdout and temperature fitter. Delegate those to laya.train / laya.common / laya.calibrate. DDP orchestration (process group, rank wiring, DDP wrap, equal-rank trim and shard, effective-batch accounting, rolling checkpoint meta) stays notebook-private. Calibration moves to the canonical artifact contract: fit_temperature_map also persists temperature_by_options when fitted, unlike the old per-type-scalar-only private fitter.
5087cf8 to
61c1455
Compare
…training copies Adds tests/test_kaggle_ddp_convergence.py and registers it in scripts/test_suites.py so CI runs it: the DDP cell keeps the canonical helpers wired in and keeps no private collate / temperature fitter / item builder, the calibration holdout stays before sharding, and split_calibration / collate_items / sigma_at / fit_temperature_map keep the contract the notebook relies on. Adapts tests/test_calibration_persistence.py to the converged export: export_notebook_config now slices from the out_cfg block instead of the retired cfg["fine_tuned"] block, and the clamp sweep is redirected as its own message asks -- it now pins that the notebook reaches fit_temperature_map and owns no fit of its own, and drives that canonical fitter with the same peaked/dipped inputs so the [TEMP_MIN, TEMP_MAX] bound stays asserted instead of the sweep matching nothing.
|
Merged. This finishes Point 1 of #887, which is the item with the strongest argument in that whole issue: #851 and #885 were the same temperature-clamp bug found and fixed twice, in parallel, because the fit existed in three copies. With #965 last release and this one, the three are now one. The notebook keeps what is genuinely DDP-specific, which is the right split: process-group and rank setup, DDP wrapping, the equal-rank trim and sharding that guards #678's deadlock, effective-batch accounting and the rolling
The All 85 suites pass on the merged tree. |
35 pull requests from 8 contributors. * LAYA_EXTRA_MODELS registers extra checkpoints on laya-serve (#1047, #919) * fine-tuning prints its optimizer-update budget before a run (#1014, #963) * the Kaggle DDP notebook converges on the shared training primitives (#1017, #887) * evals writes redirected output as utf-8 (#1045) * the server serves again after a lifespan restart (#1049) * Java router lifecycle hooks, plus an OSV dependency scan (#1046) * twenty-four documentation corrections, each verified against the code Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What
Converge the Kaggle 2×T4 DDP notebook (
notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb) onto the current shared training primitives:laya.train.items_from_rowslaya.train.encode_item+laya.common.collate_itemslaya.train.rlcd_losslaya.train.split_calibrationlaya.train.sigma_atlaya.calibrate.fit_temperature_map(vialaya.train.calibration_records)laya.train.save_checkpointDDP-specific orchestration stays in the notebook: process-group and rank/local-rank setup, DDP wrapping, equal-rank trim + sharding (the #678 deadlock guard), effective-batch accounting, per-rank shuffle and rolling
checkpoint_latestmetadata.Two existing gates are updated for the same change, because they read this notebook:
scripts/test_suites.py— registerstests/test_kaggle_ddp_convergence.pyso CI runs it (every suite intests/must be wired;tests/test_packaging.pyenforces that).tests/test_calibration_persistence.py—export_notebook_confignow slices the notebook'sout_cfgblock instead of the retiredcfg["fine_tuned"]block, and the clamp sweep is redirected as its own failure message asks once no entry point hand-rolls a fit: it pins that the notebook reachesfit_temperature_mapand owns no private fit of its own, and drives that canonical fitter with the same peaked/dipped inputs so the[TEMP_MIN, TEMP_MAX]bound is still asserted rather than the sweep matching nothing.Adds
tests/test_kaggle_ddp_convergence.py, a weight-free guard that fails if the private collate / temperature fitter / item builder come back, or if the canonical wiring is no longer in place.Why
#887 Point 1. #965 converged the two single-process wrappers onto
laya.trainand deliberately left the Kaggle/DDP path separate; #1007 then named that path as the remaining piece.The notebook was the last copy of the training loop. Fixes to the calibration or training contract had to be re-implemented by hand here — #851 and #885 were the same temperature-clamp bug in two of the copies. This is not a proposal to replace DDP with single-process
finetune(); it is so that changes to the canonical contract reach the distributed notebook instead of leaving a private copy behind.Intentional behavior change
This is not a bit-for-bit replay of the retired loop.
fit_temperature_map, so it follows today's calibration artifact contract and will persisttemperature_by_optionswhen a bucket has enough evidence. In the 400-item validation run no option-count bucket metMIN_BUCKET_N=2000, so none was produced.train_model(divide by the window's own size) rather than the retired loop's flat/GRAD_ACCUM.How it was verified
Weight-free gates:
Two gates had to be updated for the converged script, both found by CI rather than by inspection: the new suite is registered in
scripts/test_suites.py(a test outside that list is not run by CI at all), andtests/test_calibration_persistence.py— which executes the notebook's own config-export block instead of re-implementing it — slices theout_cfgblock and redirects its clamp sweep to the canonical fitter, exactly as that sweep's message asks for once no entry point hand-rolls a fit.tests/test_calibration_persistence.pyalso reports 7 tests OK, the same as073afac8— the notebook-export gate reads the converged block rather than the retired one.Real 2×T4 end-to-end (Kaggle kernel
yuyi722333/yu344-ddp-convergence-validate,machine_shape=NvidiaTeslaT4, 2 visible Tesla T4s):[1.104, 1.110, 1.137], finite and within[TEMP_MIN, TEMP_MAX];laya.Agentand completed 2,000 real predictions on the official test split: accuracy 0.7725, Brier 0.0517, ECE 0.1507 (published checkpoint: 0.766; same split/seed comparison point).Checklist
main