The decision layer (classification)¶

Three things are fitted after AutoGluon returns, from the deployed model's own out-of-sample output: probability calibration, the decision threshold, and the constant-baseline check.

In short

Every classification fit tries to ship a calibration map over the model's raw predict_proba. It ships only when it beats raw probabilities on out-of-fold log-loss by calibration_advantage_margin (0.005 nats by default).
calibration_method="auto" picks one-vs-rest (isotonic or Platt) on binary targets and temperature on multiclass targets, because temperature never changes the predicted class.
Raw and calibrated probabilities are always both available: predict_proba(X, calibrated=False) returns the model's own output.
measures.json → calibration.outcome tells you why a run serves raw probabilities: declined, not_measured, disabled or refit_degenerate.
Calibration, threshold and baseline check are one unit. model.recalibrate() re-decides all three in seconds, without retraining.

This page is mostly about calibration, the layer you reach for most. The three layers are one unit, not three steps:

calibration decides which probabilities serve;
the decision threshold is fitted on those. If you recalibrate without refitting it, the operating point is stale by construction;
the baseline check grades the decision those two produced.

So there is no "recalibrate only" operation at any layer, deliberately. All three are a function of one frozen frame, so all three can be re-decided in seconds without retraining anything. See Changing your mind, without retraining and the decision-layer reference.

The calibration map is a post-hoc probability→probability transform over the deployed model's raw predict_proba frame. It is fitted on the same fitting set as the decision threshold, gated on an out-of-fold bake-off, and shipped only when it wins by a stated margin.

ml4 owns this layer itself. AutoGluon's own calibrate=True is pinned off (engine/kwargs.py), so raw and calibrated probabilities are always both available. See Why AutoGluon's own calibration stays off.

proba = model.predict_proba(X)                    # calibrated, when a map shipped
raw   = model.predict_proba(X, calibrated=False)  # always the model's own output
model.is_calibration_active                        # True when a map serves

What calibration is for — and what it cannot fix¶

A probability is calibrated when it means what it says: of the rows scored 0.30, about 30% are actually positive. Model scores frequently are not. Boosted trees on a small or imbalanced frame are typically over-confident. Class weighting distorts them on purpose: it trains the model on a reweighted world, not yours, and roughly doubles raw Brier (see Imbalanced data).

Calibration repairs the score axis, and only the score axis.

It fixes	It does not fix
Systematic over- or under-confidence.	Discrimination. Calibration is a (near-)monotone map, so it moves AUC by essentially nothing. A model that cannot separate the classes will, after calibration, say so honestly — that is the whole improvement.
The probability distortion introduced by class weighting or resampling.	Missing signal, a wrong feature set, or leakage. No transform of the output can add information the model did not have.
Probabilities that are unusable as an input to an expected-value calculation, a price, or a downstream model.	Distribution shift. The map is fitted on the fitting set's regime. If production drifts, a calibrated model drifts with it — arguably faster, because the map is one more thing fitted to the old regime.
—	The operating point. Where to cut is the decision-threshold layer's job, fitted after calibration on whichever probabilities actually serve. Calibrating and then thresholding at 0.5 is not "tuned".

Log-loss and Brier are the only numbers that answer this question

Both are strictly proper scoring rules: they are minimised by the true probabilities, so they reward calibration and discrimination together and cannot be gamed by a confident guess. Accuracy, F1 and AUC are all blind to calibration: AUC is invariant under any strictly monotone rescaling of the score, and the label metrics only ever see which side of a cut a row landed on. ml4 deliberately does not report ECE: it is binning-dependent, and two defensible binnings of the same frame disagree.

The two families ml4 ships¶

One-vs-rest (OneVsRestCalibration) fits a 1-D calibrator per column of the probability frame — each class against the rest — and then renormalises the row so it sums to 1. The per-class primitive is isotonic (a non-parametric step function) or platt (an unpenalised logistic on logit(p) with Platt's Bayesian target smoothing).

Temperature (TemperatureCalibration) fits one scalar over the whole logit vector: softmax(log(p) / T). T > 1 flattens an over-confident row, T < 1 sharpens an under-confident one. The same divisor hits every logit, so the transform is monotone in the row as a whole.

That distinction is the substance of the choice:

One-vs-rest fits each class an independent curve and then renormalises. No step of that is monotone in the row, so two classes can swap order. The layer can then change the predicted class while claiming only to repair its probability.
Temperature cannot do that, by construction. It pays for that with a single parameter: it rescales confidence, but it cannot reshape a mis-shaped reliability curve the way per-class isotonic can.

Neither family is uniformly better, which is why both stay and the bake-off decides per dataset.

Details: how large the relabelling risk is, and why Platt is genuine Platt

The relabelling risk is not nominal. On a 3-class fixture in ml4's own test suite — built to exhibit the failure, with one class's column inflated on every row — one-vs-rest improved out-of-fold log-loss by roughly 0.9 nats while relabelling about 80% of rows. Temperature on that same frame relabelled exactly 0%, which is the property rather than the fixture.

platt is genuine Platt scaling: a logistic on logit(p) with Platt's target smoothing. It is deliberately not a stock LogisticRegression on the raw probability, which shrinks the calibration slope hardest in exactly the small/imbalanced regime that selects Platt.

Method comparison¶

Details: flexibility, argmax safety, properness and reversibility per method

	Flexibility	Monotone in the row (argmax preserved)	Multiclass properness	Reversible
`isotonic` (one-vs-rest)	Highest — any monotone reliability curve, non-parametric. Over-fits on thin or imbalanced fitting sets, which is what `platt` is the fallback for.	No. Per-class monotone, not row-monotone; the renormalisation is not monotone either.	No. K independent 1-D fits, then a renormalisation no fit accounted for. There is no joint likelihood.	Yes — post-hoc layer, raw frame always available.
`platt` (one-vs-rest)	Low — a two-parameter sigmoid on `logit(p)`. Strictly monotone, so it does not even create AUC ties.	No. Two different sigmoids on two columns can still swap their order.	No — same structure as isotonic.	Yes.
`temperature` (frame-level)	Lowest — one scalar. Rescales confidence only.	Yes, exactly. One divisor over every logit.	Yes. One fit against the joint softmax likelihood; the output is a distribution by construction, with no renormalisation step.	Yes.
AutoGluon `calibrate=True`	Same as `temperature` — it is temperature scaling.	Yes.	Yes.	No. It writes `temperature_scalar` into the model and saves it, so the model's own `predict_proba` is scaled thereafter and there is no raw frame to go back to.

Why AutoGluon's own calibration stays off¶

fit["calibrate"] = False and fit["calibrate_decision_threshold"] = False are pinned in parleml_ml4.engine.kwargs, unconditionally. The objection is not to the method: ml4 fits the same temperature scaling itself, as an exact NLL minimisation in scipy that needs neither AutoGluon nor torch. The objection is to trainer.calibrate_model(), the hook that installs it. It mutates the model, and its accept test is in-sample.

Details: what AutoGluon's calibration hook would break

It mutates the model. The scalar is stored on the model and saved, so the model's stored output becomes the calibrated output. Four things in this stack would then quietly change meaning at once, with nothing in any artifact saying so:

predict_proba(calibrated=False) — there would be no raw frame left to return.
the scores_raw sidecar the backend persists, and the Calibration-vs-Raw tab built on it, which would compare a frame with itself.
MLI: every attribution and every partial-dependence curve would be explaining a temperature-scaled output while reporting the model's name.
Batch Predict, whose output would change with no config change to point at.

The layer is also invisible to measures.json: nothing carries the scalar, the gate's verdict, or the fact that a layer exists.

And its accept test is in-sample. calibrate_model tunes T on the validation slice (or the bagged OOF), then scores scaled-versus-unscaled on those same rows and keeps the scalar if the score improved. Fitting and grading a parameter on one set of rows is exactly the comparison ml4's bake-off cross-fits to avoid.

Keeping calibration a post-hoc layer over the raw frame costs one extra transform at predict time. It buys both frames always available, one place that decides, and an audit record of the decision.

Choosing a method¶

cfg.calibration.calibration_method = "auto"           # default
cfg.calibration.calibration_advantage_margin = 0.005  # nats of out-of-fold log-loss
cfg.calibration.apply_at_predict_time = True          # default

auto is task-dependent:

Task	`auto` resolves to	Why
Binary	One-vs-rest — `isotonic` when the rarest class has 1000 rows or more, `platt` below it.	The argmax risk is not a binary risk: the served label comes from a tuned `BinaryThreshold` fitted on whichever probabilities win the bake-off, not from comparing two columns. So binary can take the flexible family and keep the repair power isotonic offers.
Multiclass	`temperature`.	Here the served label really does come from comparing columns, and a one-vs-rest map can reorder them. Argmax safety is worth more than reshaping power on a task where the layer is otherwise free to relabel rows silently.

isotonic / platt / temperature force the family on either task. Whatever you choose is only the challenger. The bake-off still has to approve it, and a forced method that loses means raw probabilities serve.

The rarest-class floor is inert under temperature

The isotonic-versus-Platt floor of 1000 rows selects between per-class curves, and temperature fits none. On a temperature run nothing consults it, and measures.json reports one fitted scalar instead of a family per class. Setting temperature explicitly on a binary target is therefore a deliberate trade of the floor's small-sample protection for argmax safety; the run log says so at configure time.

argmax-preserving is a statement about the calibration layer, not about the served label

Temperature never relabels a row by itself. The decision layer downstream is still fitted on whichever frame serves, so a run that ships a temperature map and a run that ships nothing can still disagree about a label. The difference is in the tuned thresholds, which were selected on differently scaled probabilities. test.calibration.argmax_inversion_rate measures the calibration layer; threshold_objective and thresholds describe the layer after it.

Where the map is fitted — the three cases¶

Calibration is fitted on the run's fitting set, which the HoldoutPolicy resolves once and shares with the decision threshold, the conformal residuals and the baseline gate (see Training workflow). Inside that set, the bake-off cross-fits under the run's own split policy. A GROUPED policy gets grouped folds; TEMPORAL_HOLDOUT gets a single chronological cut (case 3 below).

Details: why the bake-off follows the run's split policy

A stratified shuffle would leak on grouped and temporal data, and the leak is not symmetric between the arms. Isotonic is a step function and can memorise a near-duplicate row; a two-parameter Platt curve cannot. So the gate would over-select isotonic on exactly the data whose declared isolation says near-duplicates exist. Grouped folds are still a partition: only the geometry changes, and the comparison still spans the whole fitting set. TEMPORAL_HOLDOUT cannot have that, which is case 3 below.

Which case applies follows from the fit shape:

Fit shape	Served model	Fitting set
Bagged, `engine.refit_full = False`	the bagged ensemble	bagged OOF (case 1)
Bagged, `engine.refit_full = True` (the default)	the refit model	reserved holdout (case 2)
Not bagged (train/validation, regulated, temporal)	a single model	reserved holdout (cases 2 and 3)

1. Bagged / cross-validated fit¶

This case applies when the run bags and serves the bagged ensemble (engine.refit_full = False). Every row carries an out-of-fold prediction from a fold child that never trained on it, so the map is fitted on the whole training frame with honest scores, and no separate holdout is carved. This is the best of the three cases. The bake-off's folds are a partition, so n_rows_measured == n_rows.

One shared optimism, disclosed

When AutoGluon's greedy weighted ensemble is the strongest model, its OOF is its prediction on the matrix its blend weights were hill-climbed on. The blend's weights were therefore chosen on the same rows the calibrator is fitted on. That is one shared selection parameter across all rows, not per-row leakage, and it is published: holdout.ensemble_weights_selected_in_sample.

2. Train–validation (a single-model run)¶

A regulated preset, a temporal policy, or a train/validation training_policy leaves no OOF at all. The default bagged run with engine.refit_full = True serves a model retrained on every training row, so its OOF is not out-of-sample for it either. In both cases ml4 reserves an engine.validation_holdout_frac slice before fitting and scores it with the deployed model. The calibration set is genuinely unseen, but small, which is where the refusals below bite. The default 10% of a 2,000-row frame is 200 rows, and a rarest class under 10 of them means nothing is measured at all.

3. Temporal¶

A temporal run is case 2 twice over: it has no OOF, so the fitting set is a reserved holdout, and the bake-off inside that set cannot cross-fit either. Under TEMPORAL_HOLDOUT the bake-off is a single chronological cut: the same engine.validation_holdout_frac the outer split uses, 10% by default. n_splits is therefore 1, not 5, and n_rows_measured is roughly a tenth of n_rows.

Details: why temporal folds cannot partition, and what a one-cut gate costs

Time-ordered folds cannot partition: the earliest block has no past to train on, so it can be in no test fold. Its out-of-fold value would stay at the initialising zero, which the gate would then score both arms against. So only the later slice is scored.

A gate decided on a tenth of the rows is a much weaker gate than one decided on all of them, and n_rows alone cannot tell you which you are reading. That is what n_rows_measured is for. It is also where the refusal thresholds bite first: the cut has to leave at least 10 measured rows and both classes present, so a small temporal fitting set produces not_measured where an IID one of the same size would have produced a verdict.

Two caveats specific to temporal runs

Calibration drifts, and this layer has no drift term. Miscalibration is the first thing to go when the population moves. A map fitted on last quarter's regime is a fitted-to-the-past object and decays faster than the model's ranking does. Monitor held-out log-loss and Brier over time (test.calibration, below) and treat a widening raw-versus-calibrated gap as a refit trigger, not as evidence the layer is working.

The shipped map is refit on the whole fitting set, not on the most recent slice. The bake-off measures on the later cut, but the map that ships is refit over every row available, including the oldest. That is the right default for variance (a calibration curve estimated on a recency slice is noisy), and the wrong default if your regime genuinely changed inside the training window. There is no knob for it: narrow the training window itself if the older rows describe a different world.

What the gate measures — and what it does not¶

A map ships only when it wins a 5-fold out-of-fold bake-off on the fitting set, scored as log-loss, by more than calibration_advantage_margin:

raw_log_loss - calibrated_log_loss  >  calibration_advantage_margin   # nats, absolute

The same factory builds the challenger for every fold and for the shipped refit, so the gate never measures one family and installs another.

The margin is 0.005 nats and it is absolute. 0.0 ships on any strict improvement.
The gate is log-loss whatever eval_metric says. Hard-label metrics are unaffected either way; a pure ranking metric can lose about 1e-3 AUC under isotonic.
Log-loss cannot see the argmax, so the gate reports argmax_inversion_rate beside its verdict.

Details: why the margin is absolute, why the gate is log-loss, and what the inversion rate catches

The margin. A bare strict-improvement test (cal_ll < raw_ll) lets an edge of 1e-9 — comfortably inside the fold-to-fold noise of the very estimate that produced it — install a map on every served prediction for the life of the model. The layer's own report would then read as evidence that calibration had helped. This gate always measures one quantity in one unit, so the bar can be stated directly. That is the one place it diverges from engine.ensemble_advantage_margin, which is relative because it compares two models on a metric whose scale it does not know. Set the margin to 0.0 to accept any strict improvement on purpose. Values must lie in [0, 1).

The gate metric. Log-loss is what "calibrated" means, and the decision layer is fitted on whichever flavour wins, so every hard-label metric is unaffected either way. The one objective the choice can cost is a pure ranking metric: isotonic is a step function, so it maps distinct raw probabilities onto identical calibrated ones and roc_auc cannot break those ties. The effect is one-signed and small (measured at roughly 1e-3 AUC). ml4 does not skip calibration under a rank metric, because that buys the AUC back at about 2× the Brier score. Platt and temperature cost exactly zero here, being strictly monotone.

The argmax. Log-loss is an average over rows, and a map can lower it while relabelling individual rows — precisely the one-vs-rest failure mode. Nothing in the gate's objective would notice, so the gate reports argmax_inversion_rate alongside its verdict: the share of measured rows whose predicted class the map moved, out of fold. It costs one argmax over two matrices the gate already holds. It is structurally 0.0 for temperature, which is what makes temperature the multiclass default. A non-zero value there means something is wrong, not merely that the calibrator was aggressive.

"Not calibrated" is several different statements¶

calibration.active answers only are the served probabilities calibrated?, so every way of answering "no" collapses into the same false. calibration.outcome says which one:

`outcome`	What happened	`active`
`fitted`	The bake-off ran, the map cleared the margin, and a full-set refit ships.	`true`
`declined`	It ran and did not clear the margin — raw probabilities serve. Check `gate_declined_by_margin`: `true` means the map did beat raw, by less than you were willing to pay for, which is a different statement from "calibration does not help here".	`false`
`not_measured`	Too few rows to bake off honestly: fewer than 10 in the rarest class, fewer than 20 in total, only one class present — or folds that held out fewer than 10 rows. Nothing was measured; raw is the safe default.	`false`
`disabled`	`calibration.apply_at_predict_time = False`. The question was never asked and nothing was inspected (`n_rows`, `rarest_class_rows` and `n_splits` are all `null`).	`false`
`refit_degenerate`	The bake-off ran and won — and the full-set refit then produced no usable map (both families absorb a degenerate fit rather than crashing the run, e.g. a temperature whose optimiser returns nothing). Raw probabilities serve. This is the one raw-serving outcome that is a failure rather than a decision: the `gate_log_loss_gain` beside it is real and belongs to the cross-fitted arm, not to anything the run ships.	`false`

The block exists only on classification runs. A regression run has no class probabilities to map, so measures.json carries no calibration key at all.

Details: why the fold count does not shrink on a thin fitting set

Wherever the folds are a partition the count is a fixed 5 (temporal is the 1 above), and it does not adapt downward on a thin fitting set. The obvious softening — max(2, min(5, rarest)), so a thin set still gets some bake-off — is deliberately not used. A 2-fold comparison over a rarest class of two rows does not measure whether calibration helps; it measures which way those two rows fell.

The cost of being wrong is asymmetric. A wrongly declined map serves the model's own probabilities. A wrongly accepted one ships an isotonic step function fitted on a handful of rows into predict, the decision layer and the baseline gate at once. So a not_measured on a small frame is a refusal to measure, never a measurement that calibration would not help.

Three consequences worth knowing before you read the numbers¶

On a binary auto run whose rarest class has 1,000 to 1,249 rows, the folds validate Platt while the shipped refit is isotonic. resolved / resolved_per_class say what shipped.
A class with no rows in the fitting set is served uncalibrated under one-vs-rest, and is named in a WARNING.
The threshold, multiclass divisors and baseline gate are scored on in-sample-calibrated probabilities. in_sample_calibrated_layers lists them.

Details: the three consequences in full

Within the one-vs-rest family, the bake-off validates the method, not the fitted map. auto on a binary target is re-resolved for every calibrator fitted, against the rarest-class floor of 1000, and each fold trains on 4/5 of the rows. So on a fitting set whose rarest class lands in [1000, 1250) every fold validates Platt while the full-set refit that deploys is isotonic. The gate's verdict is then about a family the run does not serve. This is deliberate: pre-resolving the family off the full set would score a calibrator fitted on more rows than any fold has, trading a narrow family mismatch for a systematically optimistic gate. What it costs is observability, and that is paid instead: resolved / resolved_per_class publish the family each class actually got. Temperature has nothing to re-resolve (a scalar fitted on 4/5 of the rows is the same estimator as one fitted on all of them), and neither does an explicit isotonic or platt.

A class with no rows in the fitting set is served uncalibrated — one-vs-rest only. One calibrator is fitted per column of the frame, which is the model's class_labels, not the classes present in y. A class that never appears has no positive/negative contrast, resolves to identity, and passes its column through raw. The row is then renormalised across a mix of corrected and uncorrected columns. Every such class is named in a WARNING and in resolved_per_class. ml4 does not shrink that column toward a prior: that would permanently silence a class the model genuinely learned whenever the absence is only sampling luck, which is most of the cases that reach this branch. Temperature has no analogue: log(0) = -inf and exp(-inf) = 0, so a deleted class's identically-zero column comes out of the transform at exactly zero with no special case at all.

Everything downstream of the calibrator sees in-sample-calibrated numbers. The cross-fitted matrix is used for the decision and then discarded. What ships is a full-set refit, and the decision threshold, the multiclass divisors and the baseline gate are all fitted and scored on that refit's output: probabilities produced by a map fitted on those very rows. Serving the cross-fitted matrix instead would score those layers on probabilities the deployed model never emits, trading an in-sample estimate for a train/serve skew. The artifact names the affected layers rather than leaving you to infer them: in_sample_calibrated_layers lists decision_thresholds and baseline_gate when a map ships, and is empty when none does.

Reading the numbers¶

`measures.json` → `calibration` — the fit-time decision¶

A multiclass run where auto resolved to temperature and the map shipped. The model was over-confident, so the fitted T > 1 flattens it:

"calibration": {
  "method": "auto",
  "resolved": "temperature",
  "active": true,
  "temperature": 1.454,
  "outcome": "fitted",
  "n_rows": 8000,
  "n_rows_measured": 8000,
  "rarest_class_rows": 1528,
  "n_splits": 5,
  "gate_metric": "log_loss",
  "gate_log_loss_raw": 0.7787,
  "gate_log_loss_calibrated": 0.7505,
  "gate_log_loss_gain": 0.0282,
  "gate_advantage_margin": 0.005,
  "gate_declined_by_margin": false,
  "argmax_inversion_rate": 0.0,
  "in_sample_calibrated_layers": ["decision_thresholds", "baseline_gate"]
}

Details: every field of the calibration block

Field	Reading
`method`	What was asked for — often literally `"auto"`. `null` when nothing was fitted, because the block reports the map, and a declined run has none; the request is in your config, not the artifact.
`resolved`	What was actually fitted: `isotonic` / `platt` / `temperature` / `identity`. `null` when the per-class techniques genuinely disagree, and then `resolved_per_class` carries the detail instead of flattening a real mix into one misleading name.
`temperature`	Temperature family only: the fitted scalar. `null` with `resolved: "identity"` means the fit degenerated and the map serves the input unchanged — deliberately not a silent `T = 1.0`, which is numerically the identity but reads as "these probabilities were calibrated".
`active`	Are the served probabilities calibrated? Nothing more.
`outcome`	Which of the four "no" answers you are looking at (table above).
`n_rows` / `n_rows_measured`	What the gate was offered, and what it actually scored. Equal for a k-fold; roughly a tenth of `n_rows` under `TEMPORAL_HOLDOUT`.
`rarest_class_rows`	With `n_rows`, the pair the refusal is decided on.
`n_splits`	Folds the bake-off actually used: `5` for a partition, `1` for the temporal chronological cut.
`gate_metric`	Always `"log_loss"`, named rather than implied, so an `auc` run cannot be misread.
`gate_log_loss_raw` / `_calibrated` / `_gain`	The comparison itself. Positive gain means the map helped; a close call and a landslide are distinguishable.
`gate_advantage_margin` / `gate_declined_by_margin`	The bar and the one verdict the numbers alone read as a contradiction — a positive gain beside a run serving raw probabilities.
`argmax_inversion_rate`	The share of measured rows the map relabelled, out of fold. `0.0` is structural for temperature.
`in_sample_calibrated_layers`	Which other numbers in this artifact carry the refit caveat. Empty when no map ships.

A one-vs-rest binary run that was measured and declined by the margin:

"calibration": {
  "method": null,
  "resolved": null,
  "active": false,
  "outcome": "declined",
  "n_rows": 1200,
  "n_rows_measured": 1200,
  "rarest_class_rows": 143,
  "n_splits": 5,
  "gate_metric": "log_loss",
  "gate_log_loss_raw": 0.3310,
  "gate_log_loss_calibrated": 0.3288,
  "gate_log_loss_gain": 0.0022,
  "gate_advantage_margin": 0.005,
  "gate_declined_by_margin": true,
  "argmax_inversion_rate": 0.0308,
  "in_sample_calibrated_layers": []
}

Read that as: the calibrator was genuinely better, by 0.0022 nats — under half the bar — and it would have relabelled 3.1% of rows to get there. Serving raw is the right call, and gate_declined_by_margin tells you it was a judgement rather than a failure.

`measures.json` → `baseline_gate` — the verdict, and whether it is conclusive¶

The constant-baseline gate grades the decision the two layers above produced, on the same fitting set. Beside its verdict it says how many rows it graded and — when the verdict is not a finding about the model — why:

"baseline_gate": {
  "verdict": "no_signal",
  "passed": false,
  "metric": "f1",
  "strategy": "stratified",
  "strongest_score": 0.5412,
  "strongest_basis": "out_of_fold",
  "baseline_score": 0.5031,
  "improvement": 0.0767,
  "improvement_lower": -0.0214,
  "margin": 0.01,
  "n_rows": 412,
  "low_confidence_reason": "unconfirmed_win",
  "low_confidence": true
}

`low_confidence_reason`	Reading
`few_rows`	Fewer than 30 rows were graded — a verdict about the holdout, not the model.
`unconfirmed_win`	The point estimate cleared the margin, but a paired bootstrap of the graded rows does not confirm it (`improvement_lower` is its lower bound), so the verdict is `no_signal`: unproven rather than weak.
`null`	The verdict is conclusive (`low_confidence` is `false`).

few_rows wins when both apply: on too few rows an unconfirmed bootstrap is a symptom of the row count. baseline.fail_on_no_signal never aborts a run on an inconclusive verdict, and the run's warning says which reason it was.

`evaluate()` → `test.calibration` — the held-out confirmation¶

The gate decides on the fitting set. evaluate re-measures the same question on rows nothing was fitted to, for binary and multiclass. Here is the same run as above, on its test set:

"test": {
  "calibration": {
    "serves_calibrated": true,
    "log_loss_raw": 0.7864,
    "brier_raw": 0.4671,
    "log_loss_calibrated": 0.7621,
    "brier_calibrated": 0.4592,
    "argmax_inversion_rate": 0.0
  }
}

log_loss_* is in nats and directly comparable to the gate's own gate_log_loss_*: the same rule, on honest rows. A held-out gain much smaller than the fit-time one is the signal that the map over-fitted its fitting set.
brier_* is sklearn's brier_score_loss convention (parleml_ml4.delivery.scoring.brier_score). For a binary run it is the squared error of the positive-class probability, in [0, 1]. For multiclass it is the squared error between the one-hot target and the probability row summed over classes, in [0, 2]. It is the same number the application's Performance tab reports for the same model.
serves_calibrated is what makes the nulls readable. When no map serves, the served probabilities are the raw ones, so log_loss_raw / brier_raw still describe exactly what the model emits and the other three are null. A declined map cannot be re-measured here even in principle: the gate returned no object to apply.
argmax_inversion_rate is the held-out twin of the gate's.

Details: where these numbers appear in MLflow and the application

Under MLflow the block is flattened to test_calibration_* (test_calibration_log_loss_raw, …), with serves_calibrated logged as 1/0 so a raw-versus-calibrated pair in a compare-runs table is never read without it. The application backend hoists the same numbers into the held-out metric map as calibration_* (the block is a nested dict, and a flat float sweep would drop it silently). The Summary tab's calibration KPI tile shows the Brier of the arm the model actually serves: calibration_brier_calibrated when a map ships, calibration_brier_raw otherwise.

Turning it off¶

cfg.calibration.apply_at_predict_time = False

Raw probabilities then serve and the decision threshold is fitted on them. Use this for a shadow or A/B comparison against the uncalibrated baseline. Expect distorted probabilities if the run also uses class weighting, and judge it on ranking metrics rather than Brier or log-loss.

Temperature and the airgapped export¶

export_python_pipeline promises an ExportedPipeline that imports no AutoGluon, and neither calibration family breaks that promise. A TemperatureCalibration serves as a numpy/scipy softmax and a one-vs-rest map as scikit-learn, and neither needs AutoGluon or torch to load or to predict. A multiclass run that resolves calibration_method="auto" to temperature therefore exports airgapped as it is. See Deployment & export.

Changing your mind, without retraining¶

Calibration is not a step that happens once and is then sealed into the model. It, the decision threshold and the baseline gate are all functions of one frozen frame — the strongest model's out-of-sample output on the fitting set — which the bundle keeps as decision_basis/. So the three can be re-decided in seconds:

model = parleml_ml4.AutoSupervised(cfg)
model.fit(train_df)
model.recalibrate(method="temperature").save("out/bundle")

All three are refitted together, always. A threshold left over from the previous calibrator is an operating point for probabilities the model does not serve any more. The strongest model is not re-selected, and the artifact says so. A run that shipped raw probabilities can be calibrated later this way, too.

The full API, the refusal reasons, and why the preview reports out-of-fold numbers rather than held-out ones: the decision layer.

What a re-decided layer invalidates¶

The model does not change; what it predicts does. So everything derived from its predictions describes the previous predictions, and each is handled differently on purpose:

Artifact	What happens
The served probabilities and the decision threshold	Replaced. That is the operation.
The held-out calibration evidence	Recomputed from the stored raw scores — the strongest model is frozen, so nothing is re-scored through the model.
The headline outer-test score	Kept and stamped with the revision it was measured at. ml4 holds no test rows, so it cannot be recomputed; dropping it would empty the surface, and leaving it silently would present the previous layer's number as current.
Explanations (SHAP, PDP, rules)	Marked as explaining the previous revision. Recomputing them is minutes of work, so it stays a decision rather than a side effect.
The airgapped export	Carries `decision_layer_revision`. It is the one derived artifact that can leave the system entirely, holding its own copies of all three layers — re-export after a change.
A registered model	Untouched. Registration snapshots the artifact, so a production model never changes because someone adjusted the experiment it came from.

Why the strongest model stays frozen¶

Model selection ranks the leaderboard on the calibration bake-off's own criterion, so selection and calibration cannot disagree about better. Re-deciding the layer breaks that: the strongest model was chosen under one layer and serves another.

Re-running selection would cost exactly what a three-second change exists to avoid. So ml4 freezes the strongest model and stamps the discrepancy. decision_provenance.selection_consistent is False from the first behaviour-changing refit onward, and the surfaces that show a leaderboard say so.

When the whole layer is switched off¶

calibration.apply_at_predict_time = False is a supported configuration, not a degraded one:

raw model probabilities serve, and the decision threshold is fitted on those;
calibration_report.outcome is disabled, which is a different statement from declined (measured and lost) and from not_measured (too thin to test). Every surface branches on which;
fit, save, load, predict, predict_proba, export, batch scoring and the model card all work unchanged;
the basis is still captured, so you can switch calibration on later without retraining.

The same is true of baseline.enable_constant_baseline and of deployment.persist_decision_basis. Turning off the last one costs only the ability to re-decide the layer at all, never anything a prediction depends on.

Where to go next¶

Where in the fit this layer sits, and what else is fitted on the same rows: Training workflow.
The layer that turns calibrated probabilities into labels: Imbalanced data & outliers and Inference & fast scoring.
Why logloss is the default selection metric: Algorithms & metrics.
Every knob on this page: CalibrationConfig in the configuration parameters reference.

Next: Imbalanced data & outliers · Decision layer reference