Signal check¶

A standalone, dependency-light zero-shot probe that answers "is there learnable signal here?" in seconds, before any full experiment. It fits the in-context tabular foundation model on a sample and compares it to a prior baseline. See the signal-check guide.

In short

foundation_signal_check(X, y) returns a SignalCheckResult; read verdict first.
The verdict grades skill over the baseline (suspicious, strong, moderate, weak, none), not the raw score.
top_features holds the permutation-importance drivers; decisions says how each column was screened.
It needs at least 50 rows, and raises PreflightError when the engine's package is not installed.

foundation_signal_check ¶

foundation_signal_check(
    X: DataFrame,
    y: "pd.Series | np.ndarray",
    *,
    task: Optional[Tasks] = None,
    engine: Optional[str] = None,
    context_rows: int = 4000,
    eval_rows: int = 1000,
    seed: int = 42,
    top_k: int = 10,
    include_drivers: bool = True,
    n_estimators: Optional[int] = None,
    positive_class: Any = None,
    rows_total: Optional[int] = None
) -> SignalCheckResult

Probe whether (X, y) carries learnable signal, zero-shot, with a foundation model.

task is inferred from y when omitted; engine defaults to the registry default (TabICL). Fits on up to context_rows and scores a disjoint slice of up to eval_rows (both stratified for classification). Raises ValueError on too few rows and PreflightError when the engine's package is not installed.

include_drivers (default True) computes the top signal drivers by permutation importance — the probe's dominant cost, since it re-scores the eval slice once per feature. Turn it off for just the verdict + metrics (top_features is then empty). n_estimators overrides the in-context ensemble size for engines that expose it (e.g. TabICL); None keeps the fast-probe default (see foundation_probe_kwargs). Larger is marginally more accurate and proportionally slower.

positive_class (binary only, the run's Ml4Config.positive_class) picks which label metrics["f1"] and the returned eval_predictions describe. Left None the probe falls back to the fitted model's sorted classes_[1], which on {"churn", "stay"} is the majority class — safe for the verdict (ROC-AUC and accuracy are invariant) but wrong for the reported F1 and for the PR / calibration charts drawn downstream. Raises ValueError if the label is not one of the target's classes.

rows_total is the row count of the population (X, y) was drawn from, reported as n_total. A caller that row-caps its own load (the backend reads a bounded slice of a multi-million-row snapshot) is the only one that knows it: len(X) is then the size of the cap, not of the dataset, and reporting it made a 5 M-row table read as 200 000 in the UI's "rows_used / rows_total" caption. Omit it when the whole frame was handed over.

SignalCheckResult `dataclass` ¶

SignalCheckResult(
    verdict: Verdict,
    task: Tasks,
    metric: str,
    primary_score: float,
    baseline_score: float,
    lift: float,
    n_context: int,
    n_eval: int,
    engine: str,
    top_features: List[Tuple[str, float]],
    caveats: List[str],
    metrics: Dict[str, float] = dict(),
    decisions: Tuple[ColumnDecision, ...] = (),
    n_total: int = 0,
    eval_predictions: Optional[EvalPredictions] = None,
)

The verdict of a zero-shot foundation-model signal probe.

Next: Signal check guide · Tabular foundation models

Signal check¶

foundation_signal_check ¶

SignalCheckResult dataclass ¶

SignalCheckResult `dataclass` ¶