Skip to content

Choosing a hyperparameter-search strategy

hpo.provider assigns hyperparameter selection to exactly one owner. The three providers are alternatives, never layers on top of one another.

In short

  • none (the default) fits each roster model once at its effective defaults.
  • ml4_optuna runs one seeded TPE study per roster model and hands AutoGluon the winners.
  • autogluon uses AutoGluon's own search spaces and validation inside its fit budget.
  • With ml4_optuna, hpo.cv_folds=1 (Fast) scores one 80/20 holdout and 5 (Reliable) scores five folds; Reliable costs up to five times the search fits.

none: effective defaults

AutoGluon fits each requested ml4 roster entry once using the variant's required settings and the underlying learner defaults.

Choose this when training cost matters most, when the ensemble already supplies enough diversity, or when tuning evidence would be too small to be trustworthy. There is no search score and no winner's-curse contribution from comparing trials.

ml4 runs one seeded grouped-multivariate TPE study per roster model. Its spaces include family-specific regularisation, variant-only parameters, dataset-aware bounds, and the classification imbalance strategy. One completed winner per model is passed to AutoGluon as fixed parameters; AutoGluon then bags, stacks and ensembles those winners.

The validation choices are:

  • Fast — cv_folds=1: one policy-aware 80/20 inner holdout. Cheapest, but the selected parameters can exploit one unusually favourable split.
  • Reliable — cv_folds=5: five policy-aware folds. IID five-fold training uses 80% of the development rows per fit, matching the Fast training fraction, while averaging five held-out results.
  • Custom — any positive fold count. Two-fold CV is legal, but each fit trains on only half the rows and two observations provide little pruning evidence.

For non-temporal multi-fold studies, WilcoxonPruner compares each candidate with the incumbent on the same folds. Clearly inferior trials can stop early; competitive ones complete every fold. Pruning is a possible saving, not a promised one. Temporal rolling-origin studies always complete their folds because an early period is not safe evidence about a later regime.

Only completed trials can win. Search reports record both requested and realised fold counts, so a dataset that cannot support five honest folds cannot be presented as Reliable; the Ensemble search lane displays the downgrade explicitly.

AutoGluon supplies its own default search spaces, searcher and validation geometry. With bagging, a candidate is evaluated through AG's bag folds; without bagging it uses the policy-aware tuning holdout supplied by ml4. Candidate models are available to AutoGluon's weighted/stacked selection, and final disk retention still follows the fit shape's keep_only_best (on for a single-model run, off for cross-validation).

This mode does not use ml4's conditional spaces, space_tier, hpo.cv_folds, or searched imbalance strategy. Fold-local row balancing configured elsewhere in ml4 still applies as a fixed preprocessing decision. A family with no AG search space is fitted once at its effective default rather than copied num_trials times.

AG's num_trials is per searchable model and may apply again at stack levels. Its HPO time is part of AutoGluon's fit budget; ml4 does not reserve a separate pre-fit search allowance.

The result provenance records that fit-budget ownership, the validation geometry, the configured per-model trial cap, and the number of base candidates retained after AutoGluon's refit/retention policy. AutoGluon deliberately does not persist its potentially large per-trial HPO result mapping, so completed/failed/skipped trial counts are disclosed as unavailable rather than reconstructed from model-name suffixes.

Cancellation follows AutoGluon's public fit-callback boundary. A native search is one model-fit operation, so a stop request is observed before the next model search, after the current family search returns; AG exposes no supported per-trial cancellation hook for its local HPO executor. engine.time_limit supplies a nominal fit budget; native completion, refitting and later stages can exceed it. Use ml4_optuna when between-trial cancellation responsiveness is required.

Cost examples

For three boosters and 20 trials:

  • Fast ml4 Optuna: at most 3 × 20 × 1 = 60 search fits.
  • Reliable ml4 Optuna: at most 3 × 20 × 5 = 300 search fits before pruning.
  • AutoGluon native HPO: candidate cost depends on AG bag folds and stack levels; with eight bag folds, one level costs up to 3 × 20 × 8 = 480 child fits.

These counts exclude final bagging, stacking, weighted ensembling and refits.

What Reliable changes

Five-fold selection reduces overfitting to one validation split. It does not remove the winner's curse from comparing many configurations, and its best objective value is still a selection signal rather than an unbiased performance estimate. Keep the outer test untouched and quote that result for final model quality.

The reliable_tuned_ensemble preset is the bounded built-in example: three boosters, 20 trials each, five-fold adaptive selection, then AutoGluon bagging and weighted ensembling.

Next: Hyperparameter search · Configuration & presets