Skip to content

Multiclass classification

The multiclass path is the same AutoSupervised API — only the task and metric change. Here we classify a support ticket into one of three queues.

1. A labelled frame with three classes

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
n = 3_000
priority = rng.integers(1, 4, n)
words = rng.integers(3, 60, n)
attachments = rng.poisson(0.4, n)

# Route by a noisy rule so there is signal to learn.
score = 0.9 * priority + 0.02 * words + 0.5 * attachments + rng.normal(0, 1, n)
queue = np.where(score < 2.5, "billing", np.where(score < 3.8, "technical", "account"))

df = pd.DataFrame({
    "priority": priority,
    "word_count": words,
    "attachments": attachments,
    "channel": rng.choice(["email", "chat", "phone"], n),
    "queue": queue,
})

# Hold rows back BEFORE fitting: ml4's own validation tunes the decision layer and
# calibration, so it cannot also be what reports the score. These rows never reach the fit.
cut = int(len(df) * 0.8)
train_df = df.iloc[:cut].reset_index(drop=True)
test_df = df.iloc[cut:].reset_index(drop=True)

2. Configure and fit

The task default is logloss — threshold-free, so AutoGluon selects on the quality of the probabilities rather than on labels cut at a plain argmax it will not serve. Naming f1_macro instead is an explicit choice to select on argmax labels with every class weighted equally, which is what you want when the classes are imbalanced, you care about the rare ones, and the argmax decision is the product.

from parleml_ml4 import AutoSupervised, Tasks
from parleml_ml4.config import presets
from parleml_ml4.contracts.metrics import EvalMetric

cfg = presets.quick_explore()
cfg.ml_task = Tasks.MULTICLASS_CLASSIFICATION
cfg.eval_metric = EvalMetric.f1_macro
cfg.data.target_column = "queue"

model = AutoSupervised(cfg).fit(train_df)

3. Predict — labels and the class-probability frame

X = test_df.drop(columns=["queue"])

labels = model.predict(X)              # pd.Series of class names
proba  = model.predict_proba(X)        # pd.DataFrame: one column per class

print(proba.head())                    # columns: account / billing / technical
print(proba.idxmax(axis=1).head())     # argmax class per row

predict applies per-class decision thresholds tuned to the metric, so the served label is not always the arg-max probability — a calibrated minority class can win at a lower threshold. Those thresholds are divisors (argmax_c P_c / thr_c), not cuts, and the layer only ships if it beats plain argmax on the fitting set under f1_macro; see Inference → predict vs predict_proba.

Multiclass probabilities are calibrated with temperature scaling

calibration_method="auto" resolves to temperature scaling on a multiclass target: one scalar over the whole logit vector, so the layer can rescale confidence but can never reorder two classes and change the predicted class. The one-vs-rest alternative ("isotonic" / "platt") is more flexible and can do exactly that — measurably, on some frames. Both are selectable and both must win the same bake-off; the trade is in Probability calibration.

4. Evaluate and save

Score the held-out rows. An in-sample number is not a weaker estimate of generalisation, it is a different quantity — and with target encoding or a tuned decision layer it can look excellent on a model that is worthless on new data.

import parleml_ml4

report = model.evaluate(X, test_df["queue"])
print(report)

# A reloaded bundle scores itself too — no façade and no config needed.

model.save("out/ticket-router")
reloaded = parleml_ml4.load_pipeline("out/ticket-router")
reloaded.evaluate(X, test_df["queue"])

Where to go next

Next: Regression · The decision layer