Skip to content

Algorithms & metrics

ml4 trains its models on AutoGluon's engine, and gives you two levers over which models: the roster section (the UI-facing model selection) and the models argument to fit. This page covers the full model roster, how selection works, the foundation-model challenger, and the complete evaluation-metric catalogue.

In short

  • A run trains exactly the models you name: the models argument to fit, else roster.models, else one LightGBM (("GBM:gbdt",)).
  • Boosters are named with their variant ("GBM:gbdt", "CAT:symmetric"); every model gets the same fold-local feature engineering.
  • Only LightGBM, XGBoost and CAT:symmetric accept monotonic constraints.
  • Leave eval_metric unset to get logloss for classification and rmse for regression. The decision layer still tunes the served cut afterwards.
  • beta* and user_defined_metric build a custom AutoGluon scorer; ml4 threads the run config through evaluation for you.

The model roster

Stock AutoGluon families

Non-regulated runs draw from these stock AutoGluon families. A roster entry is a model spec. A gradient booster ships several structural variants and each is named with the variant it trains; a family with one shipped variant is named bare.

Spec Model Notes
GBM:gbdt LightGBM, gradient boosting The default.
GBM:dart LightGBM, DART Drops trees between rounds. Often generalises better; always costs more, because it cannot early-stop.
GBM:rf LightGBM, random-forest mode Bagged trees, no boosting. Weak alone; earns its slot by disagreeing with the boosters. learning_rate does not apply.
GBM:extra LightGBM, extra-trees settings AutoGluon's extreme-preset settings (random split thresholds, 7 leaves, low learning rate, strong regularisation). Same features as the others; best as a second model beside GBM:gbdt. Not searched. See AutoGluon's Prep options.
GBM:arith LightGBM + arithmetic features (experimental) Gradient boosting that also sees up to ~2,000 generated sums, differences, products and ratios of the run's columns. Wins clearly on some datasets and loses clearly on others; the columns are not in the feature recipe; cannot carry a monotonic constraint. Not searched. See AutoGluon's Prep options.
CAT:symmetric CatBoost, oblivious trees CatBoost's own default. Fastest to score, most regularised, the only CatBoost variant that can carry a monotonic constraint.
CAT:depthwise CatBoost, depth-wise growth More expressive. No monotonic constraints.
CAT:lossguide CatBoost, leaf-wise growth Most expressive CatBoost. No monotonic constraints.
XGB:gbtree XGBoost, gradient boosting
XGB:dart XGBoost, DART Same trade as LightGBM's DART.
RF Random Forest
XT Extra Trees
LR Linear model
NN_TORCH PyTorch neural network Ignores engine.sample_weight.
FASTAI fast.ai neural network Ignores engine.sample_weight.

Why variants and not only families. A bagged forest and a dropout-boosted tree make different mistakes than a plain gradient booster, and the greedy weighted ensemble is a selection procedure — it can only exploit disagreement it is offered. Seed replicas do not supply that, which is why ml4 leaves those to AutoGluon's bagging (num_bag_folds / num_bag_sets) and spends roster slots on structural variety instead.

What it costs. Every entry is one more trained model, and one more Optuna study when hpo.provider="ml4_optuna" is selected. The tuned_variants preset is the widest roster ml4 ships; tuned_ensemble searches the same three families at one variant each.

Monotonic constraints are per variant, not per family — see Monotonic constraints below.

Every one of them trains as a recipe-carrying custom model, so the choice of model does not change which feature engineering a run gets. Picking Random Forest or a neural net costs you nothing in FE terms.

Fold-local feature engineering applies to all of them

ml4 wraps each stock family in a custom AutoGluon model that re-fits the target-aware feature-engineering chain (target encoding → numeric twins) inside every bag fold, so those features never leak across the fold boundary. The same wrapper carries the per-fold resampling — outlier removal and class rebalancing. The opt-in foundation challenger is the one deliberate exception: it is zero-shot and rides the global feature generator.

Details: how the wrapper covers every family

engine.models.registry.FOLD_LOCAL_MODELS is checked at import time to cover STOCK_FAMILIES exactly — an explicit raise, not an assert, so python -O cannot strip it — and a family therefore cannot quietly fall back to a recipe-less passthrough.

Two mechanics make this work across families that consume their input very differently:

  • Added columns are registered. The boosters take whatever columns are in the frame, but LinearModel and both neural nets select features by metadata — so a target-encoded column their _feature_metadata had never heard of would be silently dropped. The chain runs on AutoGluon's per-model feature-generator channel, whose output schema AutoGluon records in _feature_metadata itself.
  • ml4's orchestration params never reach the estimator. ml4_recipe, ml4_family and ml4_resample are lifted out of self.params in _init_params rather than filtered in _get_model_params — that method is a convention, not a chokepoint, and LinearModel._fit builds its kwargs straight off self.params (LogisticRegression() got an unexpected keyword argument 'ml4_recipe').

What this buys the non-boosters

Linear models see categoricals as one-hot columns, and the neural nets as learned embeddings. The fold-local numeric twins arrive alongside that native representation, adding a smoothed, cross-validated target signal that a one-hot column cannot carry and an embedding has to learn from scratch.

Monotonic constraints: boosters only

The one real behavioural line through the table is which entries accept a per-feature monotone direction — roster.MONOTONE_MODELS: every LightGBM and XGBoost variant, and CAT:symmetric. roster.MONOTONE_CAPABLE_FAMILIES is derived from it (GBM, XGB, CAT) and is the coarser question "does this family have a monotone story at all". See Constraints & regulated models.

Two different cases follow from that, and ml4 treats them differently:

  • No direction to take. RF, XT, LR and the neural nets are outside MONOTONE_CAPABLE_FAMILIES. Selecting one alongside a constrained booster is a legitimate roster: it trains unconstrained and the ensemble blends the two.
  • Refuses a direction its siblings honour. CAT:depthwise and CAT:lossguide are inside MONOTONE_CAPABLE_FAMILIES but absent from MONOTONE_MODELS, because CatBoost rejects monotone constraints on nonsymmetric trees — on CPU as well as GPU (Monotone constraints are unsupported for nonsymmetric trees). A run with active constraints that names either one is refused before a single model trains, naming the roster and the constraint, rather than dying deep inside AutoGluon's fold loop. Use CAT:symmetric, or drop the variant.

It is not "the tree families"

RF and XT are tree ensembles too. What separates the three booster families is that they accept a direction, not that they are trees.

CatBoost honours monotone constraints on CPU only, so a monotone-CatBoost run is forced onto CPU automatically (num_gpus=0, logged at WARNING once per run) — a separate mechanism from the variant refusal above, which is about the tree shape rather than the device.

The glass-box quartet

For regulated, fully-transparent runs, ml4 ships four glass-box models whose native artifact is the deliverable an auditor reads:

Kind Model Audit artifact
glm scikit-learn linear (logistic / ridge) Coefficients
scorecard Weight-of-evidence logistic + a points table The points table (with its base score)
rulefit imodels RuleFit Human-readable rules
ebm Explainable Boosting Machine (adopted, audit-wrapped) Per-feature shape functions

A regulated run selects exactly one of these. See Constraints & regulated models for the full regulated story and the task-support matrix.

Every artifact in that column reaches a reader. Three of them are read off the persisted estimator by MLI's native adapters. The points table cannot be — it lives on ml4's AutoGluon wrapper, one level above the LogisticRegression, whose coefficients are per standard deviation of an engineered column — so it travels its own route: FittedPipeline.scorecard_points() on the published contract, a scorecard.json sidecar in the saved bundle, and the product's Explain → Glass-box tab, which renders it instead of the estimator's coefficients.

The foundation challenger

ml4 can add a tabular foundation model as an ensemble challenger — a zero-shot pre-trained tabular model that errs differently from the boosters, so it decorrelates the ensemble. It is a challenger by design: never the sole strongest model, excluded from the default portfolio, and memory/infer-limit gated by the resolver.

Key Model Availability
tabicl TabICL v2 (default, the one catalogued model) Installed-only

Turn it on with the composer. On an ordinary roster the challenger is required: a missing package or a failed challenger fit stops the run.

from parleml_ml4.config import presets

cfg = presets.with_foundation_challenger(cfg)                 # TabICL

See Foundation models for GPU notes and the challenger mechanics.

Selecting the roster

Selection follows a clear precedence. In order:

  1. models argument to fit, if given.
  2. config.roster.models, if non-empty.
  3. Falls back to roster.DEFAULT_MODELS — ("GBM:gbdt",), a single plain LightGBM.
# Via the roster section (UI-facing)
cfg.roster.models = ["GBM:gbdt", "GBM:dart", "XGB:gbtree", "CAT:symmetric", "LR"]

# Or per-fit
model = AutoSupervised(cfg).fit(train_df, models=("GBM:gbdt", "CAT:symmetric"))

An entry that is not a StockModel spec is rejected at config time — "GBM" on its own is not a roster entry, because the booster has to say which variant it trains.

That is the whole rule: a run trains the models named here, and nothing else picks models for it. A regulated run ignores models entirely and trains the single transparent model instead.

ml4 does not use AutoGluon's zeroshot portfolio

AutoGluon can select a curated portfolio of pre-tuned configurations (hyperparameters="zeroshot"). ml4 does not, on any path: every fitted model must carry the feature recipe for the two-locus FE path, which a stock portfolio entry cannot do, so ml4 always builds the roster itself. If no models are named, the default is one LightGBM, not a portfolio.

A regulated roster needs a train/validation policy

roster.regulated=True trains one deterministic model with no bagging and no weighted ensemble — the fit shape derives those levers from the roster. It cannot run under engine.training_policy="cross_validation": that combination is refused when the config is built, so declare "train_validation" (or "chronological_train_validation" on a temporal split). See Constraints.

The evaluation-metric catalogue

The eval_metric (an EvalMetric enum member) drives both optimisation and reporting. Leave it None to accept the task default: binary → logloss, multiclass → logloss, regression → rmse. An explicit metric is validated against the task and rejected on mismatch.

Details: why the classification default is threshold-free

AutoGluon optimises, early-stops, ranks the leaderboard and hill-climbs its ensemble weights against eval_metric. A label metric — f1, f1_macro, accuracy, precision_*, recall_*, the F-beta family — is scored by calling predict, which is a fixed 0.5 cut on binary and a plain argmax on multiclass. ml4 does not serve that cut: it fits a tuned operating point after the fit (Imbalanced data). Under a label metric, the model AutoGluon selects is therefore not judged at the cut ml4 ships: a candidate with better probabilities and a worse default cut can lose a comparison it would win at the cut that actually ships.

logloss is the only metric in the catalogue that is both threshold-free and defined on binary and multiclass (auc and average_precision are binary-only), and it is a strictly proper scoring rule, so it selects for exactly what the calibration and threshold layers stacked on top of the strongest model consume — and it is the same rule the calibration bake-off itself is scored on, so selection and calibration cannot disagree about "better". On binary the decision layer still maps logloss to the F1 objective for the served cut; on multiclass it fits no per-class bars under logloss, and predict is plain argmax on the served probabilities.

One thing can break that agreement, and it says so. Re-deciding the decision layer after the fit (the decision layer) leaves the strongest model selected under one calibration and serving another. Re-running selection would cost exactly what a cheap refit exists to avoid, so the strongest model is frozen and the discrepancy is reported instead: decision_provenance.selection_consistent is False from the first behaviour-changing refit onward.

Naming a label metric explicitly is fully supported and means what it says — select at the default cut — which is the right request when argmax (or a 0.5 cut) is what your product actually deploys.

Metric validity and presentation have separate owners:

  • Runtime modules own accepted and emitted metric IDs (EvalMetric, forecast metrics, ranking keys, clustering scores, and association rank fields).
  • parleml_ml4.contracts.metric_catalog is the single owner of shared labels, concise explanations, aliases, direction, and display format.
  • describe_metric_catalog(context) joins those two layers and fails if a runtime-owned metric has no presentation entry. The API exposes that join at GET /api/describe/metric-catalog; web and embedded apps resolve labels from this payload and must not add local metric dictionaries.

To add a metric, first add it to the runtime vocabulary that computes or accepts it, then add one MetricPresentation entry and aliases where needed. Add workflow-specific chart-reading help beside the chart; do not put chart instructions into the reusable metric definition.

Metric Notes
auc Area under the ROC curve.
average_precision Area under the precision–recall curve.
logloss Lower is better.
f1 sklearn average="binary".
beta F-beta at beta_threshold; builds a custom scorer.
accuracy
Metric Notes
logloss Lower is better.
accuracy
f1_macro / f1_weighted Explicit averaging — no hidden default.
beta_macro / beta_weighted Averaged F-beta; custom scorer.
precision_macro / precision_weighted
recall_macro / recall_weighted

Plain f1 / beta are binary-only (they use average="binary"); multiclass callers pick an explicit averaging variant so the objective is unambiguous and leaderboard-consistent.

precision_* and recall_* are graded on what you asked for, but the decision layer is not tuned for it. There are three per-class sweeps and no standalone precision or recall sweep, so precision_macro, precision_weighted and recall_macro all tune their divisors on F1 instead. The layer is kept only when it beats plain argmax under your metric, so it cannot make the run worse — but see predict vs predict_proba for the full substitution table before reading a per-class operating point as tuned for precision or recall.

There is no *_micro variant, and that is deliberate. On a single-label multiclass prediction every row contributes exactly one predicted label and exactly one true label, so the micro sums are Σtp = correct and Σpred = Σtrue = n, and F_β = (1+β²)·Σtp / (β²·Σtrue + Σpred) reduces to correct/n — accuracy, identically, for every β. Micro-precision and micro-recall collapse the same way. So f1_micro, precision_micro, recall_micro and beta_micro would be four extra names for accuracy. They would make the metric list look like a choice where there is none, and each would route to a different decision-layer objective, so a run graded on accuracy would serve labels tuned for per-class F1 or F-β. Pick accuracy when you want that number.

Metric Notes
rmse Lower is better (default).
mse Lower is better.
mae Lower is better; minimised by the median.
mape Lower is better; minimised by the median.
r2 Coefficient of determination.
spearman Rank correlation.
pearson Linear correlation.

Native vs custom scorers

The beta* metrics and any user_defined_metric build a custom AutoGluon Scorer; every other metric maps to a native AutoGluon metric string. The distinction is invisible in use — both optimise and report the same way — but it means a beta or user-defined run needs the run config threaded through evaluation, which ml4 does for you.

User-defined metrics

Supply any scalar-returning metric by reference:

from parleml_ml4.contracts.metrics import EvalMetric

cfg.eval_metric = EvalMetric.user_defined_metric
cfg.user_defined_metric_ref = "myproject.metrics:profit_score"
cfg.user_defined_metric_direction = "maximize"     # or "minimize"

The callable must accept (y_true, y_predicted, sample_weight=None) and return a scalar. user_defined_metric applies to all three tasks; user_defined_metric_ref and user_defined_metric_direction are both required when you use it.

The metric drives the decision threshold too

For classification, ml4 tunes the decision threshold to the run's metric — so choosing f1 versus beta versus accuracy changes not only which model wins but where the class boundary is set.

Where to go next

Next: Configuration & presets · Hyperparameter search