The training workflow¶

This page opens up AutoSupervised.fit and walks through what happens inside it. To train a model, start with Supervised learning. Read on to understand the machinery: recipe planning, the two feature-engineering loci, the AutoGluon fit, how validation is carved, the post-fit assembly, and the honesty gate that decides whether the model learned anything at all.

In short

A fit runs in three parts: plan the feature recipe, fit the AutoGluon engine, then assemble the post-fit layers (calibration, threshold, intervals, baseline gate).
Target-aware feature engineering re-fits inside every fold, so it never sees the rows it scores.
The post-fit layers are fitted on bagged out-of-fold predictions only when the bag itself is deployed; otherwise on a reserved holdout (the default).
The baseline honesty gate compares the model with the strongest feature-blind predictor. Read its verdict and n_rows, not just the score.
Pass fit(events=...) to receive the stage stream, and fit(cancel=...) to stop a run.

At a glance, a fit runs in three parts:

plan recipe  →  fit engine (AutoGluon)  →  assemble post-fit layers

Recipe planning¶

Before any model trains, a RecipePlanner inspects the training data and the config and produces a feature recipe: the concrete, ordered list of feature-engineering steps for this dataset. Planning is data-driven: which encoders fire, which interactions are worth trying, and which features survive selection are decided from the data, not hardcoded. The recipe is fingerprinted and carried on the fitted pipeline for audit.

The recipe distinguishes two feature-engineering loci, and the distinction is what makes ml4's FE leak-safe:

Locus	When it is fit	Sees the target?
Global	Once, before folds	No target leakage: distributional (second-order) statistics only.
Fold-local	Re-fit inside every bag fold	Target-aware encoders (target encoding, CV cat×num twins) fit only on that fold's training rows.

Target-aware transforms live at the fold-local locus so they never see the rows they will later score. Label-free transforms (clustering, text/embedding bases) are fit once at the global locus. This is a brief tour; the full transformer catalogue and leakage typing live in Transformers and Feature engineering.

The AutoGluon engine fit¶

The engine fit compiles the config + recipe into a fitted AutoGluon TabularPredictor that runs both loci: the global feature generator is passed as AutoGluon's feature_generator, and the fold-local models re-fit the target-aware FE chain per bag fold.

The roster and the pipeline shape¶

The roster selects which models to train, per spec, so a booster names the variant it trains. The pipeline shape (whether the run cross-validates, over how many folds, and how deep it stacks) is engine.training_policy with num_bag_folds / num_stack_levels, resolved by parleml_ml4.config.resolve_fit_shape and sent to AutoGluon lever by lever.

ml4 names no AutoGluon preset in the fit. AutoGluon's presets= ladder varies two things. One is its model portfolio, which never applies, because ml4 always hands AutoGluon its own recipe-carrying models. The other is a handful of fit levers the fit shape already sets explicitly: folds, stacking, the stacking probe, refit and the time limit. The model portfolio is roster.models, the deploy shape is refit_full / set_best_to_refit_full / save_bag_folds, and the pipeline shape is the fit shape above. engine has no preset field; a stored config that carries one is refused at load with a message naming the key.

Spending more compute

AutoGluon's best_quality / high_quality / extreme_quality would change nothing that ml4 does not already set itself. To spend more compute, widen roster.models, turn on hpo, raise engine.num_bag_folds, or raise the feature budget. Those are the levers that move results. The foundation_ensemble preset sets TabArena's shape (8 folds, no stacking layer) with them.

The engine section carries the rest: time_limit, bagging folds and stacking levels. The search lives in hpo.

Stock defaults vs HPO¶

By default each model in the roster is fitted once at its stock AutoGluon defaults, with no per-run search. Hyperparameter optimisation is an explicit opt-in:

cfg.hpo.provider = "ml4_optuna"
cfg.hpo.num_trials = 40         # default 60; a Training time limit, if set, ends it early

With HPO on, ml4 searches: a seeded Optuna TPE study per roster model, run before TabularPredictor.fit, whose winning hyperparameters are handed to AutoGluon as that model's roster entry. The recipe itself is never tuned.

Details: which models have a search space

The search runs per model, not per family. Every booster variant has its own space and therefore its own study, because DART's dropout knobs mean nothing to GBM:rf and one shared study would spend its warm-up learning that. Spaces exist for every booster variant plus RF / XT / LR and the regulated kinds (glm / scorecard / ebm / rulefit). NN_TORCH, FASTAI and the foundation challenger have none. ml4 fits them once at their stock defaults rather than give them a budget to spend on identical configurations.

The tuned_single, tuned_ensemble and tuned_variants presets bundle this so you don't have to set the levers by hand.

The search provider determines the strategy

hpo.provider="ml4_optuna" uses seeded TPE over ml4's conditional spaces; there is no separate searcher knob. hpo.provider="autogluon" instead uses AutoGluon's native spaces, searcher and validation geometry. none fits effective defaults. These providers are mutually exclusive; see Choosing an HPO strategy and Hyperparameter search.

Bagged OOF vs a reserved holdout¶

The rule is about the deployed model, not about bagging: the post-fit layers must be fitted on data scored by the model that actually serves.

Bagged out-of-fold (OOF): only when the bag itself is deployed, that is, bagging is on and refit_full=False (for example the high_accuracy preset). Every training row then has a prediction from a model that never saw it, and no data is set aside.
Reserved holdout: every other case, which includes the default. refit_full defaults to True, so the deployed model is a refit trained on all rows and its OOF would be self-inclusive; a single-model run (regulated, temporal, or bagging disabled) has no OOF at all. ml4 then reserves an engine.validation_holdout_frac slice before fitting and scores it with the deployed model.

The choice is resolved once by a HoldoutPolicy, in a way that respects the split policy: group and time isolation carry through to the reserved holdout. The fitted pipeline's fit_rows then names, by position in the frame handed to the fit, the rows each side of the choice took, so a caller can keep later evaluation off the rows the model learned from. A classification pipeline's out_of_sample holds the probabilities a model gave rows it never trained on: every trained row's out-of-fold prediction on a bagged run, plus the reserved holdout. A label audit reads those, because a model cannot dispute a label it has learned.

Details: out-of-fold is a per-row property, not an unconditional one

On the OOF branch each row is scored by a fold child that never trained on it, so the prediction is honest. AutoGluon's greedy weighted ensemble, however, is often the strongest model: it is the default on a cross-validation run over more than one model, which always builds it. That model's own OOF is its prediction on the matrix its blend weights were hill-climbed on (AutoGluon declares can_get_oof_from_train=True for it). The blend's weights and size were therefore chosen on the same rows the post-fit layers are then fitted on.

The optimism is one shared selection parameter across all rows, not per-row leakage, which is why ml4 keeps the OOF rather than carving a holdout: calibration, thresholds and conformal residuals are variance-dominated, and trading N rows for N/10 costs far more than the bias it removes. ml4 publishes the qualification instead: measures.json carries holdout.ensemble_weights_selected_in_sample alongside holdout.basis, and it is false whenever the deployed strongest model is a plain learner (including after the ensemble advantage gate downgrades one).

One canonical partition

The split policy resolves to a single partition used by AutoGluon bagging, the fold-local FE re-fit, and the post-fit holdout alike. There is never a second, inconsistent split hiding somewhere in the stack.

Plan the recipe on planning_frame, never on the raw training frame

Recipe planning is target-aware: instrumental feature selection, tree-guided interaction mining and the monotone probe all fit against y. Planning on the whole frame therefore lets the reserved holdout's labels help decide which features and interactions exist, and that holdout is precisely where calibration, the decision threshold and the conformal residuals are later fitted. parleml_ml4.runner.planning_frame re-derives the same deterministic carve fit_pipeline will make and returns only the fitting rows (the frame unchanged on the bagged-OOF branch, where nothing is reserved):

from parleml_ml4.runner import fit_pipeline, planning_frame

planning = planning_frame(cfg, train_data)
recipe = RecipePlanner().plan(planning.drop(columns=[target]), planning[target], cfg)
pipeline = fit_pipeline(cfg, recipe, train_data)   # full frame; re-carves internally

Every path that plans a recipe goes through it, including the recommender's per-(class, subcase) sub-model trainer, whose light budget sets the train/validation policy, so it reserves a holdout on every sub-model. It also applies the multiclass rare-class rule first (a class under engine.label_count_threshold is excluded), exactly as fit_pipeline does, so the recipe is planned on the classes the fit trains on.

Post-fit assembly¶

Once the engine has trained, ml4 scores the fitting set once (bagged OOF, or the reserved holdout) with the deployed model and fits every post-fit layer off that single pass:

Target-transform inversion (regression): the model trained on a shape-corrected target; predictions are inverted back to original units.
Calibration (classification): a frame-level calibrator is fit only if it wins a bake-off against the raw probabilities.
Decision threshold (classification): a binary threshold, or per-class multiclass thresholds, tuned for the run's metric on the probabilities the model will actually serve. Multiclass bars need a hard-label metric (f1_macro, …); under a probability metric such as the default logloss none are fitted and predict is plain argmax on those probabilities.
Prediction intervals (regression): CQR or split-conformal bands, when enabled. See Prediction intervals.
Baseline honesty gate: the skill-free-floor check described below.

Steps 2, 3 and 5 are one transaction, not three steps that happen to run in sequence: the threshold is fitted on whichever probabilities calibration decided should serve, and the gate grades the decision both produced. parleml_ml4.delivery.decision_layer owns that ordering so no caller can get it wrong. Because all three are a function of that single scoring pass and nothing else, all three can be re-decided later without retraining. See The decision layer.

The trained models become durable at path right after the engine fit, so a crash in these post-fit layers is recoverable: resume_pipeline reloads the models and re-derives the deterministic setup (the fitting set and the y-transform) without re-training AutoGluon. A recovered run produces a bundle with a decision basis, identical to the one the uninterrupted fit would have written. Without that, recovery would silently downgrade the artifact into one that can never be re-decided.

What the calibration bake-off measures — and what it does not¶

A calibration map ships only when it wins a 5-fold out-of-fold bake-off on the fitting set, scored as log-loss against the raw probabilities and by more than calibration.calibration_advantage_margin. Every part of that arrangement is a deliberate trade, and each trade costs something:

the gate is log-loss whatever eval_metric says;
it cannot see the argmax, so it also reports an inversion rate;
auto is task-dependent (binary keeps the one-vs-rest family, multiclass resolves to temperature scaling);
the map that ships is a full-set refit of the one that was measured, so the decision threshold and the baseline gate are both fitted on in-sample-calibrated numbers. The artifact discloses this in in_sample_calibrated_layers rather than leaving you to infer it.

All five trades, the four calibration.outcome values, and how to read the measures.json block are in Probability calibration.

The baseline honesty gate¶

A leaderboard can hide silent failures: a metric-direction bug, a model with no real signal, or target leakage that makes results look implausibly good. ml4 adds an explicit baseline honesty gate: it compares the deployed model against the strongest predictor that uses none of the features, on the fitting set, scored with the run's own eval metric in AutoGluon's higher-is-better score space, so it is direction-aware for free.

The floor is knowable before you train anything¶

The gate is two things, and only one of them needs a model:

the floor: what a feature-blind predictor scores on this target. That is a function of the label distribution alone, so skill_free_floor(y, task=…, metric=…) answers it with no fit, no split and no model;
the verdict: "your model barely beats it". That compares a strongest model, so it is necessarily post-fit.

Ask for the floor before you launch and you know what the run has to clear:

from parleml_ml4.delivery.baseline_gate import skill_free_floor

floor = skill_free_floor(train_df["y"], task=cfg.ml_task, metric=cfg.eval_metric)
print(f"a predictor that ignores every feature scores {floor.score:.3f} ({floor.strategy})")

Which skill-free predictor is strongest is a property of the metric, not the task, which is why every candidate is scored rather than one being picked by rule. On a 70/10/10/10 four-class target the best constant scores 0.175 precision_macro while a chance-level draw scores 0.248. Quote the constant and a model at 0.20 looks like it learned something, when it is in fact below guessing.

It reuses the gate's own candidate set and winner rule, so the number shown up front and the number the gate grades against afterwards cannot disagree.

The gate emits one of these verdicts:

Verdict	Meaning	Gate
`ok`	A real, plausible improvement over the baseline.	Passes.
`no_signal`	Beats the baseline by less than the configured margin, or by a margin a paired bootstrap of the graded rows cannot confirm (then flagged `low_confidence`).	Fails.
`worse_than_constant`	Scores below the baseline: a direction bug or genuinely no signal.	Fails.
`suspected_leakage`	Removes almost all of the baseline's error (implausibly good).	Passes, flagged as a warning.
`skipped`	The metric could not be scored on the fitting set (e.g. AUC on a single-class holdout), no skill-free candidate was scorable at all, or the baseline is already perfect so there is no error left to remove.	Not a failure.

The floor is searched, not guessed: every skill-free predictor is scored and the strongest one wins, so there is no hand-maintained (task, metric) table. The field covers both kinds of "no skill":

constants: mean / median (regression), each observed class as a label (hard-label metric), the class prior (probability metric);
chance-level draws from the target's own distribution: stratified and uniform (hard-label metric), a shuffled target (regression), drawn from a stream derived from execution.seed, so the floor is reproducible without replaying the draws the run itself made from that seed.

Both matter: a constant is a weak floor on metrics like precision_macro or f1_macro, where a coin flip outscores the best constant, and it is no floor at all for a correlation metric (pearson / spearman), which cannot score a zero-variance prediction. The reported strategy names the winner. By default a no_signal or worse_than_constant verdict is logged; set baseline.fail_on_no_signal to make it raise and abort the run.

How many rows the verdict rests on¶

Every verdict ships with the number of rows it was graded on. baseline_gate.n_rows in measures.json is the size of the fitting set the gate scored, and baseline_gate.low_confidence is true in two cases, which baseline_gate.low_confidence_reason tells apart: "few_rows" below 30 rows (the same floor, on the same rows, that flags a conformal interval's coverage claim) and "unconfirmed_win" on a win the paired bootstrap below cannot confirm. "few_rows" wins when both apply; the reason is null on a confident verdict.

That is not an exotic case. engine.refit_full is True by default, so every run reserves engine.validation_holdout_frac (0.1) of its training frame for the post-fit layers; after the launch wizard's reserved test set, a 200-row dataset grades this gate on 16 rows. At that size the decision-threshold sweep has already refused to tune and the calibration bake-off has already declined to fit, so the gate would be grading an untuned decision rule on noise.

Enough rows is not the whole test. A win that clears the margin must also survive a paired bootstrap of the graded rows: the strongest model and the floor are rescored on the same resamples, and the one-sided 95% lower bound of the improvement (baseline_gate.improvement_lower) must be above zero. On the point estimate alone, a pure-noise strongest model clears the 1% margin on about 40% of 200-row fitting sets. A win the bootstrap cannot confirm is graded no_signal with low_confidence: true and low_confidence_reason: "unconfirmed_win" (unless the rows are also too few), whatever n_rows says: nothing was demonstrated either way.

A low_confidence verdict is published and logged like any other, but baseline.fail_on_no_signal does not abort on it: a good model must not be deleted because sixteen held-out rows went the wrong way. Widen the evidence rather than the margin: more data, a larger validation_holdout_frac, or refit_full=False on a bagged run (which puts the gate on the full out-of-fold pass instead of a reserved slice).

The strongest model side is held to the same standard as the floor. The threshold is re-selected fold-wise before the strongest model is scored (the served threshold is untouched; only the gate's arithmetic changes). baseline_gate.strongest_basis in measures.json says which basis produced the number.

Details: why the threshold is re-selected, and the strongest_basis values

ml4 tunes the binary operating point on the fitting set, and the gate scores the model on that same set. Scoring it at that same threshold would report a statistic already maximised over its own nuisance parameter and compare it against unmaximised baselines. On the default binary metric that is not merely noisy but one-sided: improvement >= 0 would hold by construction, and worse_than_constant would be unreachable for any model on any data.

`strongest_basis`	Meaning
`out_of_fold`	The operating point was re-selected per fold. The score is honest.
`in_sample`	The parameter was selected on the very rows scored, so the score is an upper bound. Two ways to get here: a binary fitting set too small to re-select (a class under 5 rows, or a fold whose own sweep refuses to tune), so the served cutoff was reused and a warning is logged; or any multiclass run whose tuned decision layer serves, where the K one-vs-rest bars and the keep/discard comparison that retained them all came from these rows. No fold-wise re-selection exists for that composite rule: the bars come from independent one-vs-rest sweeps while the rule consuming them is joint, so an honest re-selection means re-fitting the whole layer per fold.
`untuned`	No decision parameter was selected on these rows at all: regression, a probability metric, a threshold that never tuned, or a multiclass run whose decision layer lost its keep/discard comparison and serves plain argmax. Nothing to be optimistic about.

Two limits, stated plainly

On pearson / spearman the shuffled draw is the only candidate that can be scored, and one shuffle is as noisy an estimate of zero correlation as a zero-skill model is. Measured over 10 seeds, the gate catches a pure-noise strongest model about half the time (a strong model was graded ok 10/10). Treat a passing correlation run as a weak signal, not a clearance. And a degenerate (constant) target is always skipped: a correlation metric can score nothing at all there, while an error metric's constant floor is already exact, so there is no error left to remove either way.

cfg.baseline.enable_constant_baseline = True      # default
cfg.baseline.fail_on_no_signal = True             # turn the warning into a hard failure

Read the verdict, not just the score

The baseline verdict lands in the run's measures and on pipeline.baseline. A high leaderboard score with a suspected_leakage verdict is a signal to go hunting for a leaking feature before you trust the model.

The stage stream¶

Throughout the fit, ml4 writes the run's stages to an parleml_ml4.telemetry.EventEmitter you pass as fit(events=...). It uses the same stage_start / stage_end vocabulary every ml4 workload writes, and the one parleml_ml4.telemetry.build_manifest_from_events reads back:

preflight → holdout_reserved → fe_structure_decided → hyperparameter_search
          → ag_fit_started → ag_fit_done → post_fit_layers → registered

Each stage is named for the milestone that ends it and spans the work since the one before, so ag_fit_done is the AutoGluon fit itself. Its stage_end records how it ended: ok, skipped, failed (with error), cancelled, or interrupted (the process was stopped). A raise ends the stage it happened in before it propagates. fit_pipeline emits every stage but registered, which belongs to whoever persists the pipeline (AutoSupervised.fit spans its MLflow logging with it; the product's worker, the bundle and sidecar writes), and it never starts or closes a run: wrap the fit in parleml_ml4.telemetry.RunTelemetry.run for run_start / run_end and a manifest.json.

A stage that deliberately did not run¶

Some stages are optional, and hyperparameter_search is off by default (hpo.provider). Such a stage still has its stage_start / stage_end pair; the end carries a different status:

{"kind": "stage_end",
 "payload": {"stage": "hyperparameter_search",
             "status": "skipped",
             "elapsed_s": 0.0,
             "skip_reason": "disabled_by_config",
             "note": "No hyperparameter search ran; each family fitted its stock defaults."}}

It stays in the ordered sequence, so the stages after it are not reported as unreachable and a later failure is still attributed to the right stage. A stage switched off before it started is a zero-length pair (elapsed_s is 0.0), because a stage that did not run must not be charged for whatever did run in the gap before it. One that started and then declined (an enabled search that tuned nothing, insufficient_data) is charged the time it took.

skip_reason is a closed vocabulary (parleml_ml4.telemetry.SkipReason); note is the one human sentence specific to this run. A consumer branches on the reason and displays the note.

Details: why a skipped stage still emits a pair

Emitting nothing would be worse: absence is indistinguishable from "not reached yet", so a progress UI that pre-renders the full stage list would leave the node sitting at pending for the life of the run, which reads as still-running or crashed. The reason is a closed vocabulary because "you switched this off", "this task has no such stage" and "it ran and refused for want of data" are three different statements about a model, and a bare skipped renders them identically.

Recipe planning happens before the stream starts

AutoSupervised.fit composes the planning frame and runs RecipePlanner().plan before the first stage. A raise from planning (the commonest is an unbindable monotone constraint) therefore propagates with no stage and no event at all. A run wrapped in RunTelemetry.run still records it, as a run_end with status="failed"; without one, treat the raised exception, not the stream, as the authority on whether a fit finished.

Three kinds ride the same stream but are not stages. The eight above are a fixed, ordered contract; these fire an unpredictable number of times or not at all:

Kind	When	Payload
`model_fit`	once per model AutoGluon starts, inside the `ag_fit_done` stage	`model`, `completed` (models finished so far), `stack_name`, `level`, `time_limit`
`model_scored`	once per model AutoGluon registers, i.e. has finished and scored it	`model`, `family`, `label`, `score_val`, `fit_time_s`, `eval_metric`, `higher_is_better`, `stack_level`, `stack_name`, `n_bags`, `fit_order`
`warning`	the run could not deliver something it was asked for (below)	`title`, `message`, `stage`

A warning is a fact the run worked around: a feature column that is entirely empty, a class excluded as too rare, a declared fit setting the fit shape overrides, a knn imputation downgraded to median, roster members that ignore the row weights, features that shipped degraded, a decision threshold that could not be tuned, a baseline verdict to act on. Each is logged once and emitted once, tagged with the stage it belongs to; the product's Notifications feed is built from exactly these events.

model_scored is the pair to model_fit: a start with no score, then a completion with one. It lets a consumer show a live leaderboard. Every field mirrors a deploy/leaderboard.json row, but that sidecar is only written at persist time.

Details: reading model_fit and model_scored

completed counts models AutoGluon has registered so far, which is what the table above says. The model name, by contrast, is announced before that model is fitted, so a model that then fails or is skipped is named in the stream but never counted.

score_val is AutoGluon's validation score, already sign-corrected (loss metrics arrive negative), and None for a model AutoGluon could not score. Show that as "not scored", never as a zero.
higher_is_better describes the metric, and is read off the scorer's _sign. AutoGluon's own Scorer.greater_is_better is True even for rmse, because it reports the sign-corrected convention rather than the metric. None when the direction cannot be read.
label falls back to AutoGluon's model name when the model's class is not in ml4's family table (a glass-box variant is only distinguishable on the instance, and nothing is loaded from disk here). The sidecar at the end of the run carries the exact label.

Stopping a run¶

fit(..., cancel=token) takes a parleml_ml4.telemetry.CancellationToken. The token is a sentinel path; touching it asks the fit to stop at the next model boundary. Models trained so far stay persisted, the stage the stop landed in ends cancelled (never failed, so a stopped run and a broken one never read alike), and the fit raises CancelledError. RunTelemetry.run ends the run with run_end status cancelled and the error.

from parleml_ml4.telemetry import CancellationToken

token = CancellationToken(outputs_dir)
pipeline = parleml_ml4.fit(config, df, cancel=token)   # another process touches token.sentinel_path

Details: sentinel lifetime and runs that die

A run never clears the sentinel when it starts. A stop requested before the run's first check (while a worker is still importing, say) is a request for that run, and its first check honours it. The run that honours the sentinel removes it, so the next run in the same directory starts clean. A run that ends any other way leaves a sentinel it never read in place.

A run whose process dies before it can close its log (a hard kill, out of memory, a crash in native code) does not read as running forever either. A reader that knows the writer has exited (build_manifest_from_events(..., writer_exited="failed" | "interrupted")) reports the run and every stage still open with that status.

Where to go next¶

The models and metrics the engine trains: Algorithms & metrics.
How validation partitions are formed: Data splitting & validation.
The two FE loci in depth: Transformers and Feature engineering.
Regulated and monotone runs: Constraints & regulated models.

Next: Inference · Deployment & export