Fix fit() crashes in tuning: rare class missing from holdout, Generator random_state - #1140
Open
LeoGrin wants to merge 2 commits into
Open
Fix fit() crashes in tuning: rare class missing from holdout, Generator random_state#1140LeoGrin wants to merge 2 commits into
LeoGrin wants to merge 2 commits into
Conversation
Temperature calibration and the log_loss threshold objective called sklearn's log_loss without the labels argument, so fit() raised when a rare class was absent from the tuning holdout (deterministic for a given dataset shape, since StratifiedKFold's per-fold class allocation ignores random_state). Pass the full label set explicitly; results are unchanged whenever every class is present. get_tuning_splits forwarded random_state raw to StratifiedKFold, which rejects np.random.Generator despite it being a documented random_state type. Convert it to a static seed via infer_random_state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LeoGrin
force-pushed
the
fix-tuning-rare-class-and-generator
branch
from
July 27, 2026 21:09
fc08f4c to
e3af60f
Compare
bejaeger
requested changes
Jul 29, 2026
| ), | ||
| "log_loss": log_loss, | ||
| # y_true is binarized one-vs-rest and may contain a single label. | ||
| "log_loss": lambda y_true, y_pred: log_loss(y_true, y_pred, labels=[0, 1]), |
Collaborator
There was a problem hiding this comment.
what if we deal with multiclass? should we take the labels from y_true unless it's a single label?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two crashes in the
tuning_configpath ofTabPFNClassifier.fit(), both found by auditingmain:1. Temperature calibration / threshold tuning crashes when a rare class is absent from the tuning holdout.
find_optimal_temperaturecalledlog_loss(y_true, probas)withoutlabels=.probasalways hasn_classescolumns, but the holdout labels come from only the firsttuning_n_foldsfolds of theStratifiedKFold, so a rare class can be entirely absent and sklearn raisesValueError: y_true and y_prob contain different number of classes. Repro: 25,002 samples with a 2-sample class andtuning_config={'calibrate_temperature': True}(auto-resolves to 1 of 3 folds).Notably the failure is deterministic, not seed-dependent:
StratifiedKFold's per-fold class allocation depends only on the class counts,shuffle/random_stateonly permute which samples fill each fold's quota. A scheduled retrain can start crashing because two rows of a new rare class were added.The
log_lossentry ofMETRIC_NAME_TO_OBJECTIVE(used by threshold tuning on binarized one-vs-rest labels) had the same missing-labelspattern and failed withy_true contains only one label; fixed withlabels=[0, 1].2.
fit()crashes whenrandom_stateis anp.random.Generatorand tuning is enabled.get_tuning_splitsforwardedrandom_stateraw toStratifiedKFold, whosecheck_random_staterejectsnp.random.Generator— even thoughGeneratoris a documentedrandom_statetype and works everywhere else infit(). Now converted to a static seed viainfer_random_state, matching whatfit()itself does.Behavior impact
None where the code previously worked:
labels = np.unique(y_true) = np.arange(n_classes)(y is label-encoded before the tuning call), so the explicitlabels=is bit-for-bit identical — verified, including the chosen temperature.Generatorconversion only touches theisinstance(random_state, np.random.Generator)case, which previously always crashed;int/RandomState/Noneare passed through unchanged.Tests
Three regression tests in
tests/test_inference_tuning.py, matching the file's unit-test style (no model needed). All three fail onmainand pass with this change; the 24 existing tests in the file still pass.🤖 Generated with Claude Code