Skip to content

Hyperparameter search

With hpo.provider="ml4_optuna", ml4 runs a seeded, grouped-multivariate Optuna TPE study per roster model — per variant, so GBM:gbdt and GBM:dart are two separate searches — then hands each model's winning hyperparameters to AutoGluon as that model's roster entry. AutoGluon bags, stacks, ensembles and refits those winners exactly as it does on an untuned run.

In short

  • Search is off by default (hpo.provider="none"); set "ml4_optuna" or pick a tuned preset such as tuned_single or tuned_ensemble.
  • Each roster model gets its own seeded TPE study of hpo.num_trials trials (default 60); each trial is one fit scored on eval_metric over hpo.cv_folds inner splits (default 1).
  • With engine.time_limit set, the search takes at most half of it, and AutoGluon's fit gets what is left.
  • The best trial score is a selection signal, not a performance estimate: quote the leaderboard and the held-out test.
  • Below 100 training rows the search is skipped and every model trains at its defaults.
for each model in roster.models:

    Optuna TPE study  ──►  winning hyperparameters ─┐
      │  trial = fit on the inner split, score      │
      │          on eval_metric, report             │
      └── num_trials trials, seeded                 │
                                                    ▼
                        AutoGluon: bag → stack → weighted ensemble → refit_full

Because the search is ml4's own step in front of the engine, it has its own stage (hyperparameter_search, between fe_structure_decided and ag_fit_started). It is off by default, and a run that does not search ends that stage skipped with the reason disabled_by_config — so a progress UI can say "no hyperparameter search ran; each family fitted its stock defaults" rather than leaving a node blank. See the stage stream.

One search provider

hpo.provider is mutually exclusive: none fits effective defaults, ml4_optuna uses the spaces documented below, and autogluon delegates search, validation and candidate retention to AutoGluon. See Choosing an HPO strategy.

Turning it on

from parleml_ml4 import Ml4Config

cfg = Ml4Config()
cfg.hpo.provider = "ml4_optuna"  # default "none"
cfg.hpo.num_trials = 20    # trials per roster model (the default is 60)
cfg.hpo.cv_folds = 1       # inner splits scored per trial
cfg.roster.models = ["GBM:gbdt", "XGB:gbtree", "CAT:symmetric"]

Or pick a template that turns it on for you.

Seven presets enable the search. Everything else fits each roster model once at its stock defaults, which is faster and — on a well-featurised frame — usually within noise.

Preset Searches Trials Space tier Why
tuned_single ✅ 60 full The tuning counterpart to the untuned single-model presets: search the boosters, deploy the single best.
tuned_ensemble ✅ 60 full Search per roster model, then let AutoGluon bag the winners into a weighted ensemble.
reliable_tuned_ensemble ✅ 20 full tuned_ensemble with each trial scored over five policy-aware folds (hpo.cv_folds=5) rather than one: 20 trials per booster keep the search at no more than 300 fits before adaptive pruning.
tuned_variants ✅ 60 full The same, over the full six-model variant roster (GBM:gbdt GBM:dart GBM:rf CAT:symmetric CAT:lossguide XGB:gbtree) — six studies, one of which (GBM:dart) runs at the 0.5 budget brake. The most expensive template ml4 ships.
regulated_monotonic ✅ 60 full A single monotone booster ships alone, so it has no ensemble to lean on. The monotone contract is part of the recipe and stays fixed under the search; it does not narrow the space.
real_time ✅ 60 capacity-capped Latency is bounded by the model it can choose, not by a numeric budget: the tier caps the size knobs so the search cannot propose a tree that is expensive to score. The preset sets no serving-latency ceiling — set deployment.max_inference_seconds_per_row yourself if you have a hard per-row SLA.
large_dataset ✅ 15 full Non-bagged and validated on one holdout; 15 trials and a one-hour nominal search/model-selection budget limit the work. Model completion and refitting can add time.
every other preset ❌ — — Including high_accuracy: its breadth comes from roster.models, not from a search.

The capacity-capped tier is not a smaller budget

real_time searches the same number of trials. What the tier narrows is the size knobs — the ones that buy accuracy by spending inference time — and it narrows them for every family that has one, not only the boosters real_time itself ships:

Family Capped
GBM / XGB / CAT max_depth / depth (and num_leaves, which is derived from depth)
RF / XT max_depth, and n_estimators — a forest evaluates every tree for every scored row
rulefit max_rules and tree_size — the retained rules, and the conditions in each
ebm interactions — an EBM costs one lookup per term, and this is the only knob that moves the term count
LR / glm / scorecard nothing: a linear model's scoring cost does not depend on its regularisation

Regularisation and the imbalance treatment stay at full width everywhere: they are free at scoring time.

What a trial is

One trial is one fit. ml4 carves an inner validation slice from the training rows, fits that roster model on the tuning-train part with the trial's hyperparameters, scores it on the run's eval_metric, and returns that score to the sampler. Nothing about the trial touches AutoGluon.

Three properties follow, and all three matter:

  • The split honours your policy. The inner slice is carved with the same grouped / temporal / stratified rule as every other ml4 split (see Data splitting) — never a fresh train_test_split.
  • The tuning slice is disjoint from the post-fit holdout — where there is one. The search runs after the holdout policy has reserved its slice, so on every branch that reserves one (the default refit_full=True, and every single-model run) hyperparameter selection, calibration, the decision threshold and the baseline gate are not all fitted on the same rows. On the bagged-OOF branch (refit_full=False) nothing is reserved and the post-fit layers read an OOF spanning every row, so the tuning slices sit inside them. ml4 does not carve a third slice there: those rows already carry a larger optimism from the weighted ensemble's in-sample blend selection, and reserving twice would shrink both estimates to remove the smaller effect. Of the shipped presets only high_accuracy takes that branch, and it does not enable the search.
  • The trial fits in the deployed label space. A TabularPredictor never hands a model the raw label — it fits AutoGluon's LabelCleaner first, mapping classes to 0..K-1. The tuner fits those same model classes standalone, so it reproduces that step itself. Without it, string labels would reach LightGBM raw (pandas dtypes must be int, float or bool on every trial) and numeric ones would quietly fit the fold-local target encoders against a label space the real fit never uses.

The positive class is part of the label space

For binary, reproducing the mapping is not enough — it has to be the same mapping. positive_class is what decides which of your two labels becomes internal class 1, and the tuner passes the run's configured value to the cleaner, exactly as fit_predictor passes it to TabularPredictor.

That agreement is load-bearing. Left to guess, AutoGluon picks the greater label (True, 1, or the one that sorts second) — while the app defaults positive_class to the minority class on every binary launch. On a fraud / legit target the two rules disagree, and nothing raises, because each side is internally consistent on its own. What breaks is everything keyed on class 1 by number:

  • The objective. A class-asymmetric metric — f1, average_precision, precision, recall — is scored at sklearn's pos_label=1, so a search under the flip ranks trials on the opposite class's number, which on an imbalanced target is high, nearly flat, and close to signal-free. (roc_auc, accuracy, logloss and mcc are symmetric under the flip and are unaffected by it — including the binary default, logloss.)
  • scale_pos_weight. It weights internal class 1, so under the flip it would be selected meaning "up-weight the majority" and deployed meaning "up-weight the minority" — worse than not searching at all, whose default is 1.

Multiclass is unaffected: AutoGluon's multiclass cleaner takes no positive_class and both sides sort identically. Regression's cleaner is a no-op. A positive_class your target does not contain does not fail the search — the search never fails a fit for a search reason — and is reported by the post-fit check instead.

What is searched, per model

Ranges are ml4's own spaces. Where a bound differs from AutoGluon's own space the difference is deliberate and the reason is in the notes column.

One study per roster model, not per family. GBM:gbdt, GBM:dart and GBM:rf get three separate studies, because they are three different searches: DART's dropout knobs mean nothing to a plain booster, and rf's bagging bounds are constraints rather than preferences (LightGBM refuses random-forest mode unless bagging_freq >= 1 and bagging_fraction < 1). A single study spanning them would spend the sampler's warm-up learning that two thirds of its draws are inert. Study files are named per spec, so hpo_gbm_dart.db sits beside hpo_gbm_gbdt.db and the Leaderboard tab can tell them apart.

GBM:extra and GBM:arith (AutoGluon's Prep options) have no space: the first is a fixed hyperparameter set that AutoGluon's own portfolio search produced, the second changes the model's features rather than its hyperparameters. A roster that names them still runs under hpo.provider="ml4_optuna" — they train once at their stock values and the search report lists them as unsearched, the same as any model without a space. A roster made only of such models has nothing to search, so the search is declined and the fit proceeds untuned.

Details: the search space for each family

The variant tables below list only what each variant adds to its family's shared block:

Variant Adds Omits
GBM:dart drop_rate, skip_drop, max_drop, xgboost_dart_mode, uniform_drop —
GBM:rf extra_trees; wider column sampling learning_rate — rf averages rather than shrinks, so it is measurably dead (identical predictions at 0.01 and 0.5)
XGB:dart sample_type, normalize_type, rate_drop, skip_drop, one_drop —
CAT:depthwise min_data_in_leaf —
CAT:lossguide min_data_in_leaf, max_leaves —

Both DART variants carry a 0.5 trial-budget multiplier: DART has no usable early stopping, so every trial pays its full round budget where a plain booster stops as soon as validation turns over.

Parameter Range Notes
learning_rate [0.01, 0.2], log
max_depth [2, 10] Capped further on the capacity tier.
num_leaves [min(20, cap), cap], cap = min(512, 2**max_depth) Conditional on the same trial's depth. LightGBM silently clamps a num_leaves above 2**max_depth: the dimension is wasted and the value Optuna records is not the value LightGBM used, so the search stops being reproducible. The 512 ceiling (not 4095) keeps the winner servable.
min_data_in_leaf [4, 100] Floor 4 rules out single-row leaves.
lambda_l1, lambda_l2 [1e-8, 3.0], log AutoGluon's space has no L1/L2 at all.
feature_fraction [0.6, 1.0]
feature_fraction_bynode [0.6, 1.0] Per-node column sampling.
bagging_fraction [0.6, 1.0]
bagging_freq [0, 5] 0 disables row bagging.
min_gain_to_split [1e-6, 1.0], log Log, not uniform on [0, 5] — the large tail is irrelevant and uniform sampling spent most trials in it.
cat_l2, cat_smooth [1e-8, 100.0] Only when categoricals reach the estimator. LightGBM's native categorical split is an unregularised in-fold target encoding; these two are its only regularisers, and ml4 routinely hands the booster raw high-cardinality categoricals.
feature_pre_filter False (constant) Required, not tuned: LightGBM otherwise caches a dataset filtered for the first trial's min_data_in_leaf and reuses it for every later trial.
scale_pos_weight see imbalance Binary only, on the native_class_weight branch.
Parameter Range Notes
learning_rate [1e-3, 0.5], log Wider than AutoGluon's [5e-3, 0.2] at both ends.
max_depth [2, 12] AutoGluon stops at 10.
min_child_weight [1, 300] AutoGluon stops at 5. This is the main regularisation lever on wide or noisy frames, and a ceiling of 5 puts its useful range out of reach.
colsample_bytree [0.3, 1.0] AutoGluon floors at 0.5.
grow_policy depthwise | lossguide
scale_pos_weight see imbalance Binary only.

XGBoost's native-categorical knobs (max_cat_to_onehot, max_cat_threshold) and tree_method are not searched. They only act when the booster is handed real category-dtype columns, and ml4 never sets enable_categorical — AutoGluon's wrapper one-hot-encodes instead, so all three would be dimensions that cannot change a fit.

gamma and subsample are deliberately not searched — see What is deliberately not searched.

Parameter Range Notes
learning_rate [0.01, 0.3], log
depth [2, 12] AutoGluon uses 5–8.
l2_leaf_reg [1e-5, 50], log The single biggest gap versus AutoGluon, whose space is [1, 5] uniform — about one decade, and it under-explores the small-regulariser tail where most well-tuned CatBoosts live.
random_strength [1e-3, 20.0], log Split-score noise. Log-scaled around CatBoost's default of 1.0: uniform sampling gave the default a ~2% draw probability and put the median trial at 25x it, i.e. most of the budget in a regime CatBoost's own authors would call degraded.
rsm [0.6, 1.0] CPU only. CatBoost ignores rsm on GPU and warns on every iteration, so the dimension is stripped there rather than sampled into a warning storm.
border_count {32, 64, 128, 254} Quantisation granularity. A ladder rather than a uniform int, so CatBoost's CPU default of 254 is actually reachable — uniform [5, 255] put the median trial at 130, binning far coarser than stock.
one_hot_max_size [20, 200] Below this cardinality CatBoost one-hots instead of building CTRs.
leaf_estimation_iterations [1, 10] Newton steps per leaf. Reaches CatBoost's default of 1, which a floor of 3 excluded outright.
min_data_in_leaf [5, 35] Depthwise / Lossguide only. The one CatBoost default (1) this space deliberately does not reach: a single-row leaf memorises individual training rows. The reason is symmetric — LightGBM's floor of 4 above encodes the same policy, and its own default of 20 already satisfies it — so this is a modelling policy, not a handicap on CatBoost.
bootstrap_type Bernoulli | Bayesian | MVS (classification only) The gating is real: MVS aborts on regression. CatBoost's GPU-only Poisson is never offered, for the same reason rsm is safe to tune: a trial fit is a CPU fit. Not because the tuner pins it — it passes no resource kwargs at all, so AutoGluon falls through to CatBoostModel._get_default_resources, which is num_gpus = 0. The consequence is elsewhere: because the winners are CPU-searched, a tuned roster containing CAT refuses execution.device="cuda" outright rather than deploying them on a GPU. See GPU acceleration.
subsample / bagging_temperature [0.6, 1.0] / [0.0, 10.0] Mutually exclusive, conditional on bootstrap_type. Bayesian takes the temperature; everything else takes the subsample.
auto_class_weights see imbalance Binary and multiclass.

AutoGluon ships no search space at all for Random Forest or Extra Trees, so without ml4's space an HPO run would train num_trials identical forests and report the best of a set of copies.

Parameter Range Notes
n_estimators [10, 500] Genuine capacity for a bagged learner — this is the one round count ml4 does tune. Capped at 150 on the capacity tier: a forest evaluates every tree for every scored row, so this is its dominant per-row cost.
max_depth [3, 20] Capped at 10 on the capacity tier.
min_samples_split [2, 20]
min_samples_leaf [1, 10]
max_features sqrt | log2 | None | 0.5 | 0.8
min_impurity_decrease [0.0, 0.1]
bootstrap True | False (RF) · False | True (XT) Order matters: TPE's warm-up draws the first value more often, and Extra Trees conventionally does not bootstrap.
max_samples None | 0.5 | 0.7 | 0.9 Both families, and only when the same trial drew bootstrap=True — scikit-learn raises otherwise.
class_weight see imbalance balanced_subsample, which recomputes per bootstrap sample.

oob_score is deliberately absent: it costs 30–50 % of the fit time and ml4 scores on the validation slice, so oob_score_ would be computed and then ignored.

ml4's LR is AutoGluon's LinearModel — LogisticRegression for classification, Ridge / Lasso for regression.

Parameter Range Notes
C [1e-2, 1e5], log AutoGluon searches [0.1, 1e3] uniform, which is a defect in its own space: C spans decades. The ceiling is high on purpose — ridge and lasso on noisy or wide frames regularly need values well above 1, and a low ceiling strands the search below the optimum.
penalty L1 | L2 Regression only. It picks the estimator (Ridge vs Lasso) there; classification always builds LogisticRegression and AutoGluon strips the key before the fit, so sampling it would double the space for byte-identical models. Uppercase — AG keys on the uppercase strings and raises otherwise.
class_weight see imbalance

Searched on regulated runs (one model, chosen by roster.regulated_kind).

glm — classification: C [1e-4, 100] log · l1_ratio [0.0, 1.0] (0 = ridge, 1 = lasso; tuning the ratio directly rather than a penalty categorical) · solver="saga" constant (the only solver covering the whole continuum and multinomial) · max_iter [200, 3000] · class_weight.

The searched solver differs from the default — deliberately

An untuned regulated GLM fits with lbfgs, because it is deterministic and a regulated model's coefficients are the deliverable: the same data under two seeds must give the same audit (Reproducibility). lbfgs supports L2 only, so the search — which sweeps the whole l1_ratio continuum — pins saga for its trials.

The trade is one-directional and costs nothing already held: saga is stochastic, so a tuned regulated fit is not seed-invariant — but the search is itself seeded, so its winner moves with the seed regardless. If you need the bit-reproducible audit, run the regulated preset without hyperparameter search.

glm — regression: alpha [1e-4, 100] log · max_iter [200, 3000]. The GLM family (gaussian / poisson / gamma / tweedie) and its link are data-dependent and resolved at plan time, so neither they nor the distribution-specific knobs that go with them (power, l1_ratio) are searched.

ebm: max_bins [128, 512] · learning_rate [1e-3, 0.3] log · max_rounds [100, 1000] · min_samples_leaf [2, 50] · max_leaves [2, 5] · inner_bags [0, 4] · outer_bags [4, 16] · smoothing_rounds [0, 1000] · interactions in ratio form (0 / 0.5x / 1x / 2x) so it scales with the feature count. EBM trials are expensive — give this one a modest budget. On the capacity tier interactions narrows to 0 / 0.5x: an EBM predicts by summing one lookup per term, so the interaction count is its only per-row cost (max_rounds and outer_bags are folded into the fitted score tables and cost nothing at scoring time).

rulefit: n_estimators [50, 400] · tree_size [2, 8] · max_rules [20, 200] · sample_fract [0.3, 1.0] · memory_par [0.0, 1.0] · include_linear True | False. tree_size and max_rules control interpretability, not only accuracy — a 200-rule model is not a rule list an auditor reads — and they are also what a RuleFit costs per scored row, so the capacity tier narrows them to [2, 4] and [20, 60]. n_estimators stays at full width there: it generates candidate rules that the sparse linear stage then prunes to max_rules, so it is training cost, not latency.

scorecard: the glm space, because it is a logistic regression — what makes it a scorecard is the weight-of-evidence binning and the points-table reporting, neither of which is a hyperparameter. Its regularisation strength still is one, so it is searched rather than left at a default. (How the WOE bins are built is an open question, but a search space is neither the cause of it nor the fix for it.)

C is only identified when the metric is scale-sensitive

Ranking metrics — AUC, Gini, AUC-PR — are invariant to a global rescale of the log-odds, so under them a tiny C scores identically to a large one and the search can collapse every coefficient towards zero (odds ratios ≈ 1.0). The model ranks fine and the points table is meaningless. On a regulated run whose deliverable is the coefficients, tune against logloss or Brier, not AUC.

Foundation challenger (tabicl): not searched. AutoGluon's wrapper exposes no tunable surface at all — no search space and no ensemble-size knob — so there is nothing to search. It stays a fixed challenger and, like every family without a space, is skipped rather than given a budget to spend on identical configurations.

NN_TORCH / FASTAI: not searched. AutoGluon's own spaces for the two neural families are rich and correct, and ml4 has nothing better to add. They train at their stock defaults on a tuned run, exactly as on an untuned one.

The imbalance axis

On classification, every searchable family also searches one imbalance categorical — none / balanced_sample_weight / native_class_weight. It is one axis rather than several independent knobs on purpose, and the reasoning is in Imbalanced data. On regression the axis is inert and is not sampled at all.

Per-fold resampling is not an arm of it. It is a config-level decision (balancing.enabled) that reaches the fold-local models as a whole ResampleSpec, fixed for the run, so the tuner has no way to vary it per trial — offering it as a fourth choice would report a decision the run never made.

The glass-box families narrow it further. glm and scorecard are offered class_weight only, and rulefit is offered nothing at all, because GlassBoxModel drops a row-weight vector on the deployed path — its _fit never reaches the point that turns the searched balanced_sample_weight into weights. Offering the arm there would weight the trial and not the model that ships. glm / scorecard lose nothing by it: class_weight="balanced" is the same correction by a route the estimator honours. rulefit genuinely has no imbalance lever — imodels takes no sample_weight at all.

engine.sample_weight narrows the axis — and can close it entirely

engine.sample_weight is a TabularPredictor constructor argument: it is fixed for the whole run and AutoGluon applies it to every model fit, so a per-model search cannot vary it. The tuner therefore fits every trial under whatever it resolves to — the trial and the deployment are the same fit — and removes the branches it has made meaningless:

engine.sample_weight What the axis offers
None (default) the full axis — the search owns the class prior
"balance_weight" refused — the pairing is a config error (see below)
a weight column none / native_class_weight — no second row-weight vector

With "balance_weight" the prior is already corrected, so scale_pos_weight and class_weight would multiply a second full correction on top of it (CatBoost's auto_class_weights silently resolves to 1.0 instead, since it reads summed object weights AutoGluon has already equalised). With any weighting set, balanced_sample_weight cannot reach the estimator at all — there is one sample_weight slot per fit and the model refuses to overwrite AutoGluon's vector.

hpo.provider=\"ml4_optuna\" + engine.sample_weight=\"balance_weight\" is refused

Not a warning — a ValueError, before a single model trains.

balance_weight is a TabularPredictor constructor kwarg: AutoGluon materialises it once and hands the same vector to every model, so a per-model search cannot vary it. The searched imbalance axis therefore collapses to ("none",) for every model — you asked for the treatment to be measured and would get a run that measured nothing.

Either escape is legitimate, so ml4 refuses rather than picking one for you:

  • clear engine.sample_weight — the search owns the imbalance decision, chosen per dataset on this run's own metric; or
  • choose hpo.provider = "none" or "autogluon" — do not ask ml4 to search that axis.

Scoped deliberately. A named weight column is a business quantity (exposure, revenue, sampling design), not a class-prior correction, so it stays legal alongside the search and native_class_weight remains searchable on top of it. And the rule is classification-only: AutoGluon's set_predefined_weights early-returns for regression, where the pairing is inert.

The wizard shows the same rule at the point of choice — the value is greyed out with the reason — generated from the very table the engine validates against.

Pruning: only a finished trial can win

With cv_folds >= 2 the objective reports each individual fold score, and Optuna's WilcoxonPruner compares those paired outcomes with the incumbent. Clearly inferior configurations can stop before every fold has been paid for. Optuna gives a pruned trial a value — its last intermediate report — so the tuner ranks completed trials only. Rolling-origin temporal folds are never pruned: early periods cannot safely eliminate a configuration that may fit the most recent regime.

The honest limitation: the selection signal is optimistic

The study fits on one inner split — or cv_folds folds of the training rows when you ask for more — not inside AutoGluon's bagged loop. Stated plainly:

  • Each hyperparameter set is scored on a single validation slice, so its score carries that slice's variance. Part of what the search selects is the slice, not the model.
  • The best trial's value is therefore an optimistic estimate of the model's quality. Read the leaderboard and the post-fit holdout for a number you can quote; the trial score is a selection signal, not a performance estimate.
  • The optimism is confined to hyperparameter selection, and — on every branch that reserves a holdout — the tuning slice is disjoint from the rows the baseline gate and calibration use. See the bagged-OOF caveat under What a trial is.
  • The trial score is also in-slice: a booster is early-stopped on the very rows it is then scored on, because the round count is deliberately not searched. This mirrors what AutoGluon's own bagging does to every child model, so the trial measures the procedure it is selecting for — but it is one more reason to read the trial score as a ranking signal only.

Raising cv_folds above 1 buys a less noisy signal for a linear increase in wall-clock — each trial then costs cv_folds fits instead of one.

For native AutoGluon search, choose hpo.provider="autogluon"; its trials use the configured AG bagging/holdout geometry rather than hpo.cv_folds.

The budget

Knob Default Meaning
hpo.provider none none, ml4_optuna, or autogluon; exactly one search owner.
hpo.num_trials 60 Trials per roster model (before the cost brakes below).
hpo.cv_folds 1 Inner splits scored per trial. 1 = a single carved slice.
hpo.patience_trials 20 Stop a family's study once this many completed trials in a row fail to improve its best (None searches the whole budget). Failed and pruned trials neither count nor reset it.
engine.time_limit None on a bare config Nominal search/model-selection budget. For ml4_optuna, half caps the search, split across searchable models; elapsed search time is deducted from AutoGluon's fit allowance. Native completion can overrun, and preparation, refit_full, evaluation and explanations add time. Glass-box regulated fits have no AutoGluon fit limit.

A study also stops itself at the ceiling. Once a completed trial reaches the metric's best possible value — the AutoGluon scorer's optimum: an AUC or average precision of 1, a log-loss or RMSE of 0 — no later trial can win, so the study ends there. A user-defined metric has no such stop: its optimum is a convention ml4 assigns by direction, and a profit-style metric routinely exceeds it. Both early stops are reported (StudyResult.stopped_early, and a line in the run log), and neither counts as stopped_by_budget.

A perfect inner score is a finding about the data

When any family reaches the ceiling, the run raises one warning naming them: separating held-out rows perfectly usually means the label is a deterministic rule of some features (a policy the label encodes) or a feature carries the label (a leak). Check both before the number is believed.

Details: one budget, two clocks

The search runs before TabularPredictor.fit, and AutoGluon starts its own clock from zero — so search time would be spent on top of the fit's budget unless something took it off again. Two mechanisms do that: SearchClock caps the whole search at half the stated limit, and translate_config subtracts what the search cost from the time_limit it passes to fit(...).

For engine.time_limit = 3600, a search that takes 412 seconds leaves 3188 seconds for AutoGluon's model-selection fit. Both numbers are logged. This avoids allocating a fresh full hour after the search; it does not guarantee that the experiment finishes within an hour. Native completion can overrun and refit_full is not bounded by that fit limit; preparation and later evaluation stages also add time.

A pathological search can still overrun its own half, because Optuna checks its timeout only between trials. The fit's budget is therefore floored at the larger of 25% of the stated limit and ten seconds, capped by the original limit, rather than allowed to reach zero: AutoGluon treats time_limit <= 0 as TimeLimitExceeded and trains nothing at all, so an unfloored subtraction would turn a slow search into a failed run.

A budget truncates the search — and says so

Within a family the limit reaches two levels — the study's own timeout, and each trial fit's time_limit — because Optuna checks its timeout only between trials, so a single long trial would otherwise overrun the whole slice. A family that finishes under its slice donates the rest to the ones after it.

A truncated family logs how many of the configured trials actually ran, and a trial whose fit is cut short scores lower than it would have — so read a budgeted search as "fewer, and slightly pessimistic, trials", not as a smaller-but-equivalent one.

Below 100 training rows the search is skipped entirely and every family trains at its defaults: ranking configurations on a slice carved out of a handful of rows is worse than not searching at all.

A trial costs one fit. The search bill is therefore

num_trials × cv_folds × |searchable roster models|      fits

plus AutoGluon's own training of the winners (which is bagged, so it is num_bag_folds fits per model on top). Note that the trial count is not multiplied by the bag folds — that is the direct consequence of tuning outside the bagged loop.

Four entries are given fewer trials than num_trials, because a trial costs them far more than it costs a plain booster:

Model Multiplier Why
ebm 0.34 An EBM trial is many booster fits, with inner and outer bagging.
rulefit 0.5 A rule ensemble plus a sparse linear model over it — two models per trial.
GBM:dart 0.5 DART has no usable early stopping, so every trial pays its full round budget.
XGB:dart 0.5 Same trade.

The reduction is a fixed per-model multiplier, floored at 10 trials so the TPE sampler still gets past its random warm-up, and never raised above what you configured.

A bare Ml4Config has no time_limit

Most built-in presets set a model-selection budget; the two glass-box regulated presets leave the fit unbounded. tuned_single, regulated_monotonic and real_time set one hour, tuned_ensemble and reliable_tuned_ensemble two, and tuned_variants three. A config built by hand has no time limit, though patience, pruning and optimum-score stops can still shorten its search. Set engine.time_limit before raising num_trials. See Preset time budgets.

Reproducibility and failures

  • The sampler is seeded. execution.seed reaches the TPE sampler, and the shared study runner requires a seed rather than accepting None — an unseeded TPESampler makes the whole fit irreproducible.
  • Failed trials are counted and reported on the run, never swallowed. A family that loses trials raises a run warning — "LightGBM: 36 of 60 search trials failed", saying its winner is the best of the ones that completed — and a study where every trial failed says so explicitly and falls back to the family's defaults instead of pretending the defaults won. Silence here is how a systematically broken search space hides for months.
  • A loaded host does not decide the search. Every AutoGluon booster early-stops a fit when the host has under 512 MB free, which on a shared server measures the other processes: trials needing a few megabytes would be cut to a handful of rounds, or — for LightGBM on a metric without a native name — crash outright. The search therefore runs with AutoGluon's per-process budget (AG_MEMORY_LIMIT_IN_GB, set to the machine's memory unless you exported your own), so the guards judge the trial's own footprint, and Ml4FoldLocalGBM registers only its stopping metric, so the guard stops a fit cleanly rather than raising.
  • The winner deploys what the trial fitted. The configuration handed to AutoGluon is not read off study.best_params — that is what Optuna recorded, and a space may also return values it never suggested. It is rebuilt by replaying the winning draws through the same space, so constants, derived values and conditional branches all survive. This matters most for the regulated glm / scorecard: their space pins solver="saga" as a constant (the model's own default, lbfgs, supports L2 only), and without it every winner with l1_ratio > 0 would fail with "Solver lbfgs supports only 'l2' or None penalties".
  • Progress is observable, and the search is interruptible. Each finished trial emits an optuna_trial event, so the live progress strip shows the search the same way it shows the Learning-to-Rank tuner — and the run's cancel token is polled at the same point. Without both, a tuned preset spends its longest phase silent and deaf to a cancel: AutoGluon's fit callback cannot see the search, because the search finishes before the fit begins.

The studies are per-trial detail. Separately, every run records a short search report — what was searched, how much of the requested budget each family actually spent, and, crucially, which families were not searched at all. It rides in ensemble_composition.json under search and is drawn as a lane in front of stacking level 1 on the product's Ensemble tab.

It exists because the architecture diagram could not otherwise answer "where did these hyperparameters come from?" in either state: a searched run and a stock-defaults run would draw the identical picture.

Five states, each a different statement about the roster:

State What it means
provider: none The shipped default. Every family fitted its effective defaults — a complete answer, not an absence.
provider: ml4_optuna, refusal set You requested ml4 search and it could not run: too few training rows, no tuning folds could be carved, or no roster model declares a space.
provider: autogluon Search runs inside AutoGluon's fit and follows its validation geometry.
a family with has_space: false That family declares no search space (the neural nets, the foundation challenger). No trials were asked for — which is not the same as none having run.
a family with best_value: null Its study ran and no trial produced a usable score, so it fell back to its defaults. Distinct from the row above, and the diagram draws them the same way for a reason: both ended on stock defaults.

stopped_by_budget is the one number that keeps a winner honest. A family that completed 28 of 60 trials because its slice of engine.time_limit ran out produced the best of 28, and reading it as the best of 60 overstates the search.

The report carries best_params_digest, not the winning parameters: those are already in the roster and on the leaderboard, and a second copy is how the two come to disagree. The digest lets a reader prove they match while carrying nothing that can drift.

A run resumed after a crash does not re-run its search, so search is null there — not recorded, which is deliberately distinguishable from no search ran.

Persisted studies, the leaderboard, and resuming one

Every study a fit runs is written to SQLite inside the predictor directory, one file per roster model — the spec, lower-cased, with : as _:

<predictor path>/optuna/hpo_gbm_gbdt.db
<predictor path>/optuna/hpo_gbm_dart.db
<predictor path>/optuna/hpo_cat_symmetric.db

path is the only location fit_pipeline is given, so the studies go in a subdirectory of it rather than next to it — a library must not drop files into a directory the caller never named.

That is what makes a search inspectable rather than a number in a log: the studies carry full per-trial parameters, values and timings, and the product's leaderboard reads them back out. In a backend run the predictor lives at <outputs_dir>/deploy/predictor, so the studies land at <outputs_dir>/deploy/predictor/optuna/ — which is one of the two locations the leaderboard globs (the other is <outputs_dir>/optuna/, where the Learning-to-Rank search writes). A tuned supervised run therefore shows both boards: one row per trained model from the AutoGluon model board, plus one row per search trial. The model board still supplies the run's best model — a search trial is a probe, not something you can deploy.

Pass no path to the fit and the study runs in memory instead — the search is identical, it just leaves nothing behind.

A fit always starts its studies fresh: any DB left by a previous run at the same path is deleted first, so trial numbering starts at 0 and the leaderboard cannot show another run's configurations as this one's.

Continuing a study

Continuing one is a supported entry point, through the shared study runner rather than through fit:

from parleml_ml4.tuning import run_study

result = run_study(
    my_objective,
    direction="maximize",
    n_trials=20,            # 20 MORE trials, not a target total
    seed=42,
    study_name="hpo_gbm_gbdt",  # the study inside the DB
    storage_dir="outputs/optuna",
    resume=True,
)

What resume=True changes, and nothing else does:

  • n_trials becomes a budget of additional trials. A study with 40 trials on disk and n_trials=20 ends with 60.
  • The winner is the best of the whole study, not the best of the continuation. A resumed run that beats nothing it already had must not report a worse configuration than the one on disk — so the stored best_trial is adopted as the incumbent before the first new trial runs. (n_failed is the exception and stays scoped to this call, since the earlier run's failures were already reported when they happened.)
  • An unusable DB is discarded, not raised on. A study file is written incrementally during a long fit, so a killed process (OOM, cancel, machine sleep) routinely leaves a half-written SQLite file behind. Opening it fails deep inside SQLAlchemy with a message that says nothing about hyperparameter search. It is probed, and if it cannot be opened it is removed and the study starts clean; if it cannot even be removed, the search downgrades to in-memory. The worst case is a repeated search, never a failed run.

Resume the same question

A study carries the draws a search made, not the space it drew them from. Continuing one under a different hpo.space_tier, a different task or an edited space is a different search: Optuna will reject values that fall outside the new space's choices, and ml4 refuses to reconstruct a winner it cannot replay through the current space — the family falls back to its stock defaults and says so. There is no fingerprint check on a study DB; keep the config that produced it.

What is deliberately not searched

These are negative results, not omissions. Re-adding them has been tried and measured.

Not searched Why
Booster round counts (n_estimators, iterations) The tuner early-stops its own fits, and AutoGluon re-derives rounds with its own early stopping for the final fit. Tuning a number that gets overwritten burns a dimension. RF/XT n_estimators is the exception — for a forest it is genuine capacity, not a stopping point.
XGBoost gamma, subsample AutoGluon measured both and removed them: "made search worse." Its own comment invites refining the ranges first, which is exactly what a log scale would do — so this is revisitable, but it is not free knowledge and it is not re-added on assertion.
RF / XT oob_score 30–50 % fit-time penalty for a number ml4 never reads.
CatBoost fold_permutation_block, model_shrink_rate, feature_border_type No leaderboard signal, and model_shrink_rate = 0.0 silently undoes the learning-rate tuning — a knob that can invalidate another knob's search. GreedyLogSum is competitive everywhere.
monotone_constraints Not a hyperparameter. It is part of the recipe and stays fixed under the search; see Constraints.
Device, thread count, seed Resolved by execution and threaded to AutoGluon by the translator. A search that set them would fight the resource plan and oversubscribe the machine.
objective, loss_function, eval_metric Resolved from eval_metric, including custom scorers. Tuning the objective is tuning the question.
The booster variant itself (boosting_type / grow_policy / booster) Not a search axis inside a study — it is the roster entry. GBM:dart, GBM:rf, CAT:depthwise, CAT:lossguide and XGB:dart are separate specs with their own spaces and their own studies (see What is searched, per model), so the variant is chosen by you and the ensemble, not by a sampler that would spend two thirds of its warm-up on inert draws. The crash traps are handled inside each space rather than dodged: LightGBM's rf mode requires bagging_freq >= 1 and bagging_fraction < 1, so those are bounds of the GBM:rf space rather than preferences.
XGB:gblinear Not shipped at all — no tree structure means no monotone constraint and no native tree interpretation, so it would carve an exception through two subsystems to add a linear model the LR family already provides.

Watch the metric you are amplifying

The default metric is logloss on both classification tasks — threshold-free, so the search ranks trials on the quality of their probabilities and not at a cut ml4 replaces afterwards. A TPE search exploits a mis-specified objective far harder than random draws do, so this matters more here than anywhere else in the product.

If you name a label metric instead (f1, f1_macro, accuracy, the F-beta family), understand what you are buying: every trial is graded at a fixed 0.5 / argmax cut while ml4 tunes the decision threshold afterwards, so the search optimises a decision rule the run does not ship. Do it when argmax (or a 0.5 cut) is genuinely your product. If your positives are rare and you want the ranking rather than the labels, prefer eval_metric = EvalMetric.average_precision.

f1 and average_precision are also class-asymmetric — computed for internal class 1 only — which is why the search and the deployed fit have to agree on which label that is. See the positive class is part of the label space.

See also

Next: Choosing an HPO strategy · Imbalanced data