Skip to content

Fix fit() crashes in tuning: rare class missing from holdout, Generator random_state - #1140

Open
LeoGrin wants to merge 2 commits into
mainfrom
fix-tuning-rare-class-and-generator
Open

Fix fit() crashes in tuning: rare class missing from holdout, Generator random_state#1140
LeoGrin wants to merge 2 commits into
mainfrom
fix-tuning-rare-class-and-generator

Conversation

@LeoGrin

@LeoGrin LeoGrin commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

Two crashes in the tuning_config path of TabPFNClassifier.fit(), both found by auditing main:

1. Temperature calibration / threshold tuning crashes when a rare class is absent from the tuning holdout.
find_optimal_temperature called log_loss(y_true, probas) without labels=. probas always has n_classes columns, but the holdout labels come from only the first tuning_n_folds folds of the StratifiedKFold, so a rare class can be entirely absent and sklearn raises ValueError: y_true and y_prob contain different number of classes. Repro: 25,002 samples with a 2-sample class and tuning_config={'calibrate_temperature': True} (auto-resolves to 1 of 3 folds).

Notably the failure is deterministic, not seed-dependent: StratifiedKFold's per-fold class allocation depends only on the class counts, shuffle/random_state only permute which samples fill each fold's quota. A scheduled retrain can start crashing because two rows of a new rare class were added.

The log_loss entry of METRIC_NAME_TO_OBJECTIVE (used by threshold tuning on binarized one-vs-rest labels) had the same missing-labels pattern and failed with y_true contains only one label; fixed with labels=[0, 1].

2. fit() crashes when random_state is a np.random.Generator and tuning is enabled.
get_tuning_splits forwarded random_state raw to StratifiedKFold, whose check_random_state rejects np.random.Generator — even though Generator is a documented random_state type and works everywhere else in fit(). Now converted to a static seed via infer_random_state, matching what fit() itself does.

Behavior impact

None where the code previously worked:

  • With every class present in the holdout, sklearn infers labels = np.unique(y_true) = np.arange(n_classes) (y is label-encoded before the tuning call), so the explicit labels= is bit-for-bit identical — verified, including the chosen temperature.
  • The Generator conversion only touches the isinstance(random_state, np.random.Generator) case, which previously always crashed; int/RandomState/None are passed through unchanged.

Tests

Three regression tests in tests/test_inference_tuning.py, matching the file's unit-test style (no model needed). All three fail on main and pass with this change; the 24 existing tests in the file still pass.

🤖 Generated with Claude Code

LeoGrin and others added 2 commits July 27, 2026 17:05
Temperature calibration and the log_loss threshold objective called
sklearn's log_loss without the labels argument, so fit() raised when a
rare class was absent from the tuning holdout (deterministic for a given
dataset shape, since StratifiedKFold's per-fold class allocation ignores
random_state). Pass the full label set explicitly; results are unchanged
whenever every class is present.

get_tuning_splits forwarded random_state raw to StratifiedKFold, which
rejects np.random.Generator despite it being a documented random_state
type. Convert it to a static seed via infer_random_state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LeoGrin
LeoGrin force-pushed the fix-tuning-rare-class-and-generator branch from fc08f4c to e3af60f Compare July 27, 2026 21:09
@LeoGrin
LeoGrin requested a review from bejaeger July 27, 2026 21:43
),
"log_loss": log_loss,
# y_true is binarized one-vs-rest and may contain a single label.
"log_loss": lambda y_true, y_pred: log_loss(y_true, y_pred, labels=[0, 1]),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if we deal with multiclass? should we take the labels from y_true unless it's a single label?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants