Reproducibility & seeds¶

One knob — execution.seed (default 42) — drives every randomness source in the fit, including the trained models themselves. This page says exactly what it reaches, what no seed can buy, and the stability study the seed makes possible.

In short

execution.seed (default 42) seeds the trained models, AutoGluon's bag folds, the hyperparameter search, feature selection and resampling.
The split resolver has its own seed: engine.split_policy.seed (default 0).
A seed reproduces a run only on the same machine, build and data; thread count, GPU execution and engine.time_limit all add drift.
Refit at several seeds before you trust a small gap between two models.

from parleml_ml4 import Ml4Config

cfg = Ml4Config()
cfg.execution.seed = 42       # the whole run's randomness

Details: a rebuilt decision layer changes the artifact without changing the run

Calibration, the decision threshold and the baseline gate can be re-decided after a fit without retraining. That breaks the usual assumption that "same run id, same numbers": the run did not change, and what it serves did.

Two things make it answerable rather than a hole. refit_decision_layer is deterministic given the basis — the same configuration over the same persisted frame reproduces the same layer, exactly, which is why the runner fits from that frame in the first place. And every derived artifact carries decision_layer_revision, so "which layer produced this number?" has an answer even when the model file is identical.

What the seed reaches¶

Surface	What the seed decides
AutoGluon's fold split	Which rows land in each bag fold (AutoGluon's learner `random_state`; a regulated run pins it to `0`)
Trained models	Row and feature subsampling, tie-breaking, weight init — every family in the roster
Bagged folds	The bag's own fold shuffling (`model_random_seed`), not just its child estimators
HPO	The Optuna TPE sampler (Hyperparameter search)
Feature selection	The instrumental rankers that score features before the real fit
Target-encoding inner folds	Which training rows share each leak-safe encoder fit
Per-fold resampling	Over/under-sampling and outlier removal (Imbalanced data)
Baseline gate	The chance-level floor's draws, from a stream derived from the seed so the floor reproduces without replaying the run's own draws (Training workflow)
Temporal lag dropout	Which engineered lag columns the regulariser nulls, likewise on a derived stream (Temporal encoders)
Prediction intervals	The CQR quantile companions fitted for the bands (Prediction intervals)

The split resolver's outer-test carve and its stratified or grouped K-fold shuffle use the split's own seed, engine.split_policy.seed (default 0), not execution.seed. Hold it fixed to compare two configs on identical partitions; vary it to test whether a score survives a different partition.

How the models are seeded¶

ml4 writes the run seed into every roster entry — one per model spec, so all three LightGBM variants are seeded independently — under whatever name that family calls it. Without it, every AutoGluon-trained learner would fall back to AutoGluon's own constant (AbstractModel.default_random_seed = 0), and runs at different seeds would train byte-identical learners.

Details: the seed parameter per family

The name is read off the model class (AbstractModel.seed_name) rather than a hand-written table, so a family whose upstream name changes cannot silently stop being seeded, and a family that exposes no seed_name is left alone rather than handed a key it would reject:

Family	Parameter
`GBM` (LightGBM), `XGB`	`seed`
`CAT` (CatBoost)	`random_seed`
`RF`, `XT`, `LR`, the glass-box GLM / scorecard / RuleFit, EBM	`random_state`
`NN_TORCH`	`seed_value`
`FASTAI`	`random_seed`
Any bagged model	`model_random_seed` (via `ag_args_ensemble`)

Five spellings for one concept is exactly why the name is read off the class rather than listed in ml4.

Two details are load-bearing rather than incidental:

The bag is seeded as well as its children. The bagged wrapper does its own fold shuffling; seeding only the estimators would leave the fold assignment unseeded, the run would still not reproduce, and the failure would read as model noise. AutoGluon gives fold i the seed model_random_seed + i, so the folds stay different from each other while the bag as a whole reproduces.
The value is clamped to the int32 ceiling. LightGBM stores its seed as int32 and CatBoost validates random_seed as size_t; both crash the native layer on an overflowing value, and derived seeds routinely exceed it.

A family that exposes no seed name is left alone — there is nothing to seed, and inventing a parameter name would fail the fit rather than fix the randomness.

Measuring whether an edge is real¶

Seeded models make a stability study possible. Refit the same config at several seeds and compare the spread of the strongest model's score against the gap it claims over the runner-up: if the gap is inside the spread, the "winner" is fold noise.

import statistics
from parleml_ml4 import AutoSupervised, Ml4Config

scores = []
for seed in (1, 2, 3, 4, 5):
    cfg = Ml4Config()
    cfg.data.target_column = "y"
    cfg.execution.seed = seed
    cfg.engine.time_limit = None          # a wall-clock cap makes the roster itself vary
    model = AutoSupervised(cfg).fit(train_df)
    report = model.evaluate(X_test, y_test)
    scores.append(report["test"]["score"])

print(statistics.mean(scores), statistics.pstdev(scores))

The reported confidence interval is not part of the spread

evaluate's bootstrap CI resamples the test rows from its own fixed seed, so it is stable across runs and measures a different uncertainty: how much the score would move on a different test sample. The seed spread above measures how much it moves on a different fit. Both matter, and a strongest model whose edge survives neither is not a strongest model.

Vary the seed before you believe a leaderboard gap

ml4's ensemble sidecar reports lift_vs_single — how much the weighted ensemble beat the best standalone model by. A lift smaller than the seed spread measured here is noise, which is exactly what the ensemble advantage margin exists to refuse to ship.

What a seed cannot buy¶

A fixed seed makes a run reproducible on the same machine, build and data. It does not make it reproducible across these:

Source of drift	Why the seed does not cover it	What to do instead
Thread count	Float reductions are summed in completion order, so a different thread count changes the last bits	Pin `execution.n_jobs` and `OMP_NUM_THREADS`
GPU execution	CUDA atomics reduce non-deterministically	Use `execution.device = "cpu"` when bit-exactness matters (GPU)
`engine.time_limit`	A wall-clock budget decides how many models finish, and that varies with machine load	Leave `time_limit = None` for a reproducible run
Library / AutoGluon version	Different code trains a different model	Pin the versions; the bundle's `manifest.json` records the ml4 that wrote it

Regulated runs are the strictest case

regulated_glassbox and regulated_scorecard force a single, non-bagged estimator with no AutoGluon fit time limit. regulated_monotonic instead searches boosters under a one-hour model-selection budget, so its outcome can vary with host load. See Constraints & regulated models. How much the seed changes even so is family-dependent, and the guarantee should be stated that way: an lbfgs-solved GLM converges to the same coefficients whatever the seed, while a booster's row and feature subsampling does not. Because the seed reaches every model, "identical across two seeds" is a claim a reviewer can test.

The regulated GLM is lbfgs-solved by default (solver="lbfgs"). Two consequences: (1) the guarantee covers the untuned path only. Enabling hyperparameter search switches the solver to saga, a stochastic solver, because the search sweeps the elastic-net continuum that lbfgs cannot fit (Hyperparameter search), so a tuned regulated run is not seed-invariant. (2) EBM, RuleFit and scorecard have their own stochastic components; only the GLM is claimed bit-exact.

Next: GPU acceleration · Hyperparameter search