Signal check (zero-shot)¶

The signal check is a standalone, dependency-light probe that answers one question fast: is there learnable signal in (X, y)? It fits a tabular foundation model zero-shot on a context slice, scores a disjoint eval slice, and compares the result to a prior or majority baseline. There is no full AutoGluon run and no feature engineering pipeline.

In short

Call foundation_signal_check(X, y) before a full experiment. It runs TabICL zero-shot and returns a verdict in seconds on CPU for a few thousand rows.
The verdict is none, weak, moderate, strong or suspicious. It grades how far the score beats a baseline, not the raw score.
suspicious means a near-perfect score: look for a leaked column in top_features before you trust it.
Use include_drivers=False when you only need the verdict; the drivers are the main cost.
Time series have their own check: see Forecasting: Signal check.

Agent access

Agents reach the probe through the MCP tool probe_dataset (kind signal; compute scope, cached). It is the recommended check before proposing any modelling. See the ParleML MCP docs.

Forecasting has its own signal check

For time series, the parallel probe is forecast_signal_check: a zero-shot Chronos backtest scored as MASE against a naive baseline. This page covers the tabular (TabICL) probe.

Unlike this one, it always answers. A panel whose timestamps have no regular spacing is scored on an ordinal axis (observation order standing in for the calendar) rather than refused. If your "time series" turns out to be an event log or an entity table sorted by a date, that check says so.

Run it before a full experiment to decide whether a dataset is worth training on, to catch a leaky column early, or to rank candidate targets.

from parleml_ml4.foundation_signal import foundation_signal_check

result = foundation_signal_check(X, y)     # TabICL by default
print(result.verdict, result.metric, result.primary_score, result.lift)

What it does¶

foundation_signal_check(X, y, *, task=None, engine=None, context_rows=4000, eval_rows=1000, seed=42, top_k=10):

Splits the data into a context slice (up to context_rows=4000) and a disjoint eval slice (up to eval_rows=1000), stratified for classification.
Screens columns into a numeric matrix and records a per-column ledger. Timestamps are cast to epoch seconds, constant columns are excluded, the remaining non-numeric columns are coded as sorted ordinals, and NaNs are median-imputed. The screen is fitted on the context slice alone and applied to the eval slice, so the held-out rows never inform the probe's own inputs.
Fits the in-context foundation model (default TabICL) on the context slice and scores the eval slice.
Compares the primary metric to a baseline and returns a coarse verdict plus the top signal drivers.

task is inferred from y when omitted, and engine defaults to the registry default (TabICL). The probe needs at least 50 rows (_MIN_ROWS); fewer raises ValueError. If the chosen engine's package is not installed, it raises PreflightError.

Details: why the column screen sorts, and why it casts timestamps

Every foundation engine's front end is a chain of strictly monotone per-feature maps (standardise, then a quantile or power transform). So a feature's entire usable content is its rank order. The screen respects that in two ways:

A datetime64 column is not numeric as far as pandas is concerned, so it would otherwise take the categorical path. It is cast to epoch seconds instead: a real scale. That also stops a high-cardinality timestamp from collapsing to a single "unseen" code on the eval slice.
The remaining non-numeric columns are coded with factorize(sort=True). pandas defaults to first-appearance order, and the context slice arrives shuffled. The default would number an ordered column (an ISO date string, an ordered categorical) by a random permutation of its true order, turning a real trend into noise. Sorted codes cost nothing on a genuinely nominal column. When values cannot be ordered at all, the ledger says so.

A degenerate slice is refused, not graded

For classification the probe raises ValueError when either slice holds a single class. At the default slice sizes this happens once the minority rate drops below roughly 0.04%.

Details: why a one-class slice raises

roc_auc_score returns nan on a one-class y_true rather than raising. A nan primary score would fall through every verdict cutoff to none. "Could not evaluate" would then read as "no learnable signal", the strongest discouragement this probe can give. Stratified allocation puts zero minority rows in the eval slice once the minority rate drops below roughly 0.04%, so this is not an exotic input.

Details: how the task is inferred

A numeric target is treated as regression only when it carries a fractional part, or when its distinct-value count is high both in absolute terms and relative to the row count. The relative half keeps integer-coded categoricals (product codes, ICD groups, segment ids) on the classification path. A 25-value code column over 6,000 rows has plenty of distinct values but very few per row, and grading it with R² over arbitrary code integers is meaningless.

That relative test depends on scale, so a numeric target with more than 50 distinct values reads as regression whatever the row count. Otherwise an ordinary integer measurement (unit counts, ages, durations) on a large frame would stay under the fraction and be probed as a 70-class problem. The ceiling applies to numeric targets only; a string target with more levels is still classification. Pass task= to override.

Details: wide multiclass targets

TabICL's shipped checkpoint supports 10 classes natively. Above that the probe drops its key/value context cache (a pure speedup) and runs slower rather than refusing, so a target with 11+ classes still gets a verdict.

One exception comes from the task inference above, not the engine. A target whose classes are integer codes can be read as a regression (by the distinct-value ceiling, or by the relative test on a small frame), and R² over arbitrary code integers reads as no signal. Any whole-number target inferred (not requested) as a regression carries a caveat naming the distinct-value count and telling you to pass task=. If you already know the column holds codes, pass it up front.

The verdicts¶

The verdict grades the skill: the share of the headroom above the baseline that the probe captured, (primary_score − baseline_score) / (1 − baseline_score). 0 is no better than the baseline and 1 is perfect. Cutoffs apply highest first. A near-perfect score is treated as a leakage smell, not a win.

Verdict	Classification skill	Binary ROC-AUC equivalent	Regression (R²)
`suspicious`	`>= 0.98`: implausibly good, likely leakage	`>= 0.99`	`>= 0.999`
`strong`	`>= 0.60`	`>= 0.80`	`>= 0.50`
`moderate`	`>= 0.30`	`>= 0.65`	`>= 0.20`
`weak`	`>= 0.10`	`>= 0.55`	`>= 0.05`
`none`	below `weak`	below `0.55`	below `weak`

For binary classification the baseline is a ROC-AUC of 0.5, so the skill cutoffs equal the ROC-AUC values in the second column. For multiclass the baseline is the majority-class accuracy, so a model that only predicts the majority class scores a skill of 0 and reads none. For regression the baseline R² is 0, so the skill is the R² itself.

suspicious means investigate, not celebrate

A suspicious verdict flags a near-perfect score. That is almost always a leaked identifier, a target proxy, or train/test contamination. Inspect the top drivers (below) before trusting it.

Inputs¶

Argument	Default	Meaning
`task`	inferred from `y`	`BINARY` / `MULTICLASS` / `REGRESSION`
`engine`	registry default (TabICL)	which foundation model to probe with
`context_rows`	`4000`	max rows the model fits on (the in-context "training" set)
`eval_rows`	`1000`	max rows in the disjoint scoring slice
`seed`	`42`	split and permutation-importance seed
`top_k`	`10`	number of top drivers to return
`include_drivers`	`True`	compute the permutation-importance drivers (the probe's dominant cost); `False` returns the verdict and metrics only, with `top_features` empty
`n_estimators`	`None`	override the in-context ensemble size for engines that expose it (such as TabICL); `None` keeps the fast-probe default
`rows_total`	`None`	rows in the population `(X, y)` was drawn from, reported as `n_total`. Only a caller that row-caps its own load knows it; omit it when you pass the whole frame. A value below `len(X)` raises
`positive_class`	`None`	binary only: which label `metrics["f1"]` and the returned eval predictions describe. `None` falls back to the fitted model's sorted `classes_[1]`, which on `{"churn", "stay"}` is the majority class `"stay"`. The verdict is unaffected (ROC-AUC and accuracy do not depend on it), but the reported F1 and the downstream PR and calibration charts are drawn for this class. Pass `Ml4Config.positive_class`.

Outputs — `SignalCheckResult`¶

result.verdict          # suspicious | strong | moderate | weak | none
result.task             # the task probed (inferred or passed)
result.metric           # primary metric name: roc_auc | accuracy | r2
result.primary_score    # the primary metric value
result.baseline_score   # prior / majority / mean baseline
result.lift             # primary_score - baseline_score
result.top_features     # [(column, importance), ...] permutation-importance drivers
result.decisions        # per-column screening ledger: (column, action, reason)
result.metrics          # every computed metric (superset of the primary)
result.caveats          # e.g. "converted 1 timestamp column(s) to epoch seconds"
result.n_context, result.n_eval, result.n_total, result.engine

Primary metric follows the task: ROC-AUC (binary), accuracy (multiclass), R² (regression). The metrics dict carries the rest (accuracy and F1 for classification; MAE and RMSE for regression).
Lift is how far the model beats the baseline: a ROC-AUC of 0.5, the majority-class accuracy, or an R² of 0.0.
Drivers are the top permutation-importance features on the eval slice (bounded to keep the foundation predict cost in check).
Screening ledger (decisions) records kept, factorized or excluded, with a reason per column. You can see exactly what the probe consumed, including which columns were cast from timestamps and whether a coded column kept its ordering.
n_total is the size of the population sampled from, not len(X). A caller that reads a bounded slice of a large snapshot passes rows_total, so the "rows used / rows total" caption does not quote its own cap back.

for col, action, reason in result.decisions:
    print(f"{col:20s} {action:11s} {reason}")

Cost & speed¶

The whole cost of the probe is foundation-model inference. The split and the "fit" are free (an in-context model only stores the context slice). The probe runs one predict for the eval metrics, plus one per feature for the permutation-importance drivers: about n_features + 1 predicts over the same context.

Because the context slice is identical across all those predicts, the probe uses a fast-probe config instead of the engine's stock inference defaults:

KV-cache reuse: the context key/value tensors are computed once and reused across every predict.
Reduced test-time ensemble: a couple of in-context members instead of the stock eight. A coarse verdict does not need full test-time augmentation.

Two arguments let you trade speed for detail:

include_drivers=False skips permutation importance entirely (the largest single cost) and returns only the verdict and metrics (top_features is empty). Use it when you only need "is there signal?" and not "which columns carry it".
n_estimators overrides the probe ensemble size for engines that expose it. Accuracy is essentially flat across ensemble sizes here, so the fast-probe default suits a coarse verdict; raise it only for extra stability.

Details: measured speed-up

On a 6,000-row, 14-feature multiclass set on CPU, the fast-probe config turns about 7 minutes into seconds with no material change to the verdict. The K/V cache is the dominant lever: permutation importance went from about 345 s to about 37 s on its own (≈9×). The reduced ensemble adds a further 1.5–2× with no measurable accuracy change (0.66 either way from 2 to 8 members).

The settings are engine-specific (parleml_ml4.engine.models.foundation.foundation_probe_kwargs). Engines without a tuned config run at their stock defaults. GPU is faster still: see GPU acceleration.

When to run it¶

Before a full experiment: a none verdict means the full AutoGluon run is unlikely to find signal, so you save the compute.
Leakage triage: a suspicious verdict plus a single dominant driver is a classic leak.
Target or feature-set comparison: it is cheap enough to run across candidate targets or feature subsets and rank them by lift.

Warm the engine first

The first probe pays the backbone load synchronously. Call parleml_ml4.foundation_signal.prewarm() at boot to warm the default estimator, so the first real check is hot. It is best-effort and never raises. On GPU the probe is substantially faster: see GPU acceleration.

The probe uses the same foundation catalogue as the full engine: see Tabular foundation models. For the result schema, see the signal-check reference.

Next: Tabular foundation models · Supervised learning