Evaluation

Holdout model evaluation — metrics, calibration, and the expected-cost lenses (binary business-value + multiclass K×K). See the Evaluation & trust guide.

In short

• compute_model_evaluation scores a live model on held-out splits; evaluate_from_predictions scores stored predictions.
• Each split keeps a raw (y_true, scores) sample of up to 200,000 rows; curves are re-derived from it on demand.
• Headline numbers use the full split. A capped sample can be class-enriched: weight it with raw_sample_weight.
• Multiclass thresholds are tuned on the first split only, so that split's per-class numbers are optimistic.

compute_model_evaluation

compute_model_evaluation

compute_model_evaluation(
    model: Any,
    *,
    task_type: TaskType,
    adapter: Any,
    splits: Mapping[str, SplitSpec],
    decision_threshold: (
        ResolvedThreshold | float | None
    ) = None,
    positive_class: Any = None,
    multiclass_threshold_metric: str = "f1",
    n_bootstrap: int = 0,
    ci_alpha: float = 0.05
) -> ModelEvaluationResult

Score holdout splits and compute evaluation artifacts.

Parameters:

Name	Type	Description	Default
model	Any	Fitted estimator used for scoring holdout rows.	required
task_type	TaskType	TaskType.BINARY, TaskType.MULTICLASS or TaskType.REGRESSION. MULTICLASS is dispatched to :func:compute_multiclass_evaluation; as a defensive fallback a BINARY task whose model.classes_ has > 2 entries is also treated as multiclass.	required
adapter	Any	Model adapter exposing get_predictions(model, X, task_type). Bypassed for multiclass classifiers so we can pull the full (n, K) proba matrix straight from model.predict_proba instead of the binary-positive-class slice the adapter returns.	required
splits	Mapping[str, SplitSpec]	{split_name: (X, y)} — e.g. {"valid": (X_valid, y_valid)}. Empty dict returns an empty :class:ModelEvaluationResult.	required
decision_threshold	ResolvedThreshold | float | None	Threshold for classification metrics at the profiler cut-off (see :func:evaluate_from_predictions). Ignored for multiclass splits (argmax already picks the predicted class).	None
positive_class	Any	The binary positive label; the model's classes_ order fixes it when None (see :func:evaluate_from_predictions).	None
multiclass_threshold_metric	str	F-beta spec for per-class OvR threshold tuning (multiclass-plan §3.3 leak-prevention plumbing). The first split in iteration order — typically "train" or "valid" — is used as the tuning source via :func:tune_per_class_thresholds; the resulting per-class operating points are then frozen and applied verbatim to every other split so the test metrics never peek at their own labels. Defaults to "f1" (matches the binary path's threshold="f1" default). Ignored when only one split is supplied (nothing to leak into).	'f1'

Split protocol / in-sample optimism

Pass the tuning split ("train" or "valid") first and the reporting split ("test") after it: per-class operating thresholds are tuned on the first split and frozen for the rest, so only the first split's own multiclass metrics are read at an in-sample-tuned threshold and are correspondingly optimistic. Every subsequent split is scored at those frozen thresholds and is leak-safe. If you pass a single split, its thresholds are tuned and reported on the same rows — treat those metrics as optimistic. The binary path's optimal_* fields are likewise tuned in-sample on each split (see :func:parleml_mli.evaluation.metrics.evaluate_classification_split).

evaluate_from_predictions

evaluate_from_predictions

evaluate_from_predictions(
    *,
    task_type: TaskType,
    splits: Mapping[str, PredictionSplitSpec],
    classes: Sequence[Any] | None = None,
    positive_class: Any = None,
    decision_threshold: (
        ResolvedThreshold | float | None
    ) = None,
    predicted_labels: Mapping[str, ndarray] | None = None,
    multiclass_threshold_metric: str = "f1",
    multiclass_tuning_split: str | None = None,
    n_bootstrap: int = 0,
    ci_alpha: float = 0.05
) -> ModelEvaluationResult

Build a :class:ModelEvaluationResult from precomputed scores.

This is the single dispatch over (y_true, scores) tuples shared by the model-scoring path (:func:compute_model_evaluation, which resolves the scores from a model first) and the bring-your-own / precomputed-holdout path (e.g. monitor_predictions's lite pipeline, or a deployment that persisted the holdout predictions at fit time). No model or feature matrix is required — only the targets and the model's scores.

Parameters:

Name	Type	Description	Default
task_type	TaskType	TaskType.BINARY or TaskType.REGRESSION. Multiclass is a property of the scores, not the enum: pass classes with more than two entries (or 2-D scores with more than two columns) and the split is routed to :func:compute_multiclass_evaluation.	required
splits	Mapping[str, PredictionSplitSpec]	{split_name: (y_true, scores)}. scores is the positive-class probability (binary, (n,) — a (n, 2) matrix is accepted and its positive-class column taken), the (n, K) probability matrix (multiclass), or the raw prediction (regression). Splits whose y_true or scores is None are skipped.	required
classes	Sequence[Any] | None	Class label ordering (e.g. model.classes_). Drives multiclass detection (len > 2; None infers it from the score shape) and the binary class order.	None
positive_class	Any	The binary positive label. The binary target is resolved once with :func:parleml_mli.core.target.resolve_target (positive_class, else classes[1], else the labels of the splits) and every split's y_true is binarised against it.	None
decision_threshold	ResolvedThreshold | float | None	Classification cut-off for the at-threshold metrics — a :class:~parleml_mli.core.threshold.ResolvedThreshold or a float. None reports at the model's own decision (0.5 on the positive-class probability). Ignored for multiclass (argmax) and regression.	None
multiclass_threshold_metric	str	F-beta spec for leak-safe per-class OvR threshold tuning — tuned on one split and frozen for the rest. See :func:compute_model_evaluation.	'f1'
predicted_labels	Mapping[str, ndarray] | None	Optional multiclass decisions by split, from the fitted model's decision layer. These drive confusion matrices and label metrics; the original probabilities still drive proper scores and calibration. Omit to use argmax.	None
multiclass_tuning_split	str | None	The split whose rows tune the per-class thresholds. None tunes on the first usable split. Name it when the display order puts the reporting split first (a caller that lists "test" before "validation" so readers open on the final read); a name that is not in splits is a caller bug and raises. The result keeps the caller's split order either way.	None
n_bootstrap	int	Number of bootstrap resamples for per-metric confidence intervals. 0 (default) disables them. Applied to the binary and regression splits via :func:bootstrap_metric_cis (multiclass macro metrics are left without CIs). Populates each split's metric_cis.	0
ci_alpha	float	Two-sided level for the bootstrap CIs (default 0.05 → 95%).	0.05

Split protocol / in-sample optimism

Multiclass per-class operating thresholds are tuned on one split — multiclass_tuning_split, else the first usable split — and frozen for every other split. That split's own metrics are read at an in-sample-tuned threshold and are optimistic; the frozen-threshold splits are leak-safe. With a single split, tuning and reporting share the same rows — the metrics are optimistic. multiclass_tuning_split on the result names the tuned split either way.

ModelEvaluationResult

ModelEvaluationResult dataclass

ModelEvaluationResult(
    task_type: str,
    splits: dict[
        str,
        "ClassificationSplitEvaluation | RegressionSplitEvaluation | MulticlassEvaluation",
    ] = dict(),
    multiclass_tuning_split: str | None = None,
)

Optional holdout evaluation attached to a profiling result.

The splits dict carries one evaluation per holdout split:

• :class:ClassificationSplitEvaluation (aka BinaryEvaluation) for binary classifiers.
• :class:RegressionSplitEvaluation for regression models.
• :class:parleml_mli.evaluation.MulticlassEvaluation for multiclass classifiers (len(model.classes_) > 2). Multiclass plan §3.3
• Phase 4: dispatched automatically based on the model — the viewer's Performance page renders the K×K matrix, per-class OvR ROC/PR/lift/KS, and per-class reliability diagrams off the active class dropdown.

The string task_type is one of "binary" / "regression" / "multiclass" (the last added in Phase 4) so the viewer can pick the right rendering branch without sniffing the split values.

preferred_split

preferred_split() -> (
    "tuple[str, ClassificationSplitEvaluation | RegressionSplitEvaluation | MulticlassEvaluation] | None"
)

(name, split) of the split a headline number should quote.

The first of :data:PREFERRED_SPLIT_ORDER the result carries (held-out test before validation before a generic holdout before out-of-fold), else the first split stored; None without splits. Every reader that quotes one split — the Trust tiles, the model card, the narrators — picks it here, so they all quote the same one.

Per-split evaluation types

The typed payloads stored in ModelEvaluationResult.splits — one per holdout split, dispatched by task. BinaryEvaluation is an alias of ClassificationSplitEvaluation.

ClassificationSplitEvaluation dataclass

ClassificationSplitEvaluation(
    split_name: str,
    n_samples: int,
    positive_rate: float,
    decision_threshold: float,
    accuracy: float,
    precision: float,
    recall: float,
    f1: float,
    f05: float,
    f2: float,
    roc_auc: float | None,
    pr_auc: float | None,
    log_loss: float | None,
    confusion_matrix: list[list[int]],
    optimal_f05: OptimalMetric,
    optimal_f1: OptimalMetric,
    optimal_f2: OptimalMetric,
    optimal_accuracy: OptimalMetric,
    ks_statistic: float,
    ks_threshold: float,
    raw_y_true: ndarray,
    raw_scores: ndarray,
    brier_score: float | None = None,
    expected_calibration_error: float | None = None,
    raw_sample_weight: ndarray | None = None,
    metric_cis: (
        dict[str, tuple[float, float]] | None
    ) = None,
)

Classification metrics for one data split, backed by the raw sample.

The split stores only (a) the raw (y_true, scores) sample — capped at :data:RAW_EVAL_SAMPLE_CAP rows — and (b) cheap scalars that are expensive or random to recompute. Every threshold-dependent array (threshold sweep, ROC/PR, ranking, calibration curve, score distribution) is derived on demand from the raw arrays via :mod:parleml_mli.evaluation.derive.

RegressionSplitEvaluation dataclass

RegressionSplitEvaluation(
    split_name: str,
    n_samples: int,
    mae: float,
    rmse: float,
    r2: float,
    mape: float | None,
    mean_residual: float,
    median_abs_error: float,
    raw_y_true: ndarray,
    raw_y_pred: ndarray,
    metric_cis: (
        dict[str, tuple[float, float]] | None
    ) = None,
    raw_sample_weight: ndarray | None = None,
)

Regression metrics for one data split, backed by the raw sample.

Stores headline scalars + the raw (y_true, y_pred) sample (capped at :data:RAW_EVAL_SAMPLE_CAP); actual-vs-predicted and residual plots are derived from the raw arrays via :mod:parleml_mli.evaluation.derive.

MulticlassEvaluation dataclass

MulticlassEvaluation(
    split_name: str,
    class_keys: tuple[str, ...],
    n_samples: int,
    accuracy: float,
    macro_f1: float,
    macro_precision: float,
    macro_recall: float,
    log_loss: float,
    brier_score: float,
    confusion_matrix: ndarray,
    y_true_idx: ndarray = (lambda: empty(0, dtype=int))(),
    per_class: dict[str, BinaryEvaluation] = dict(),
    macro_expected_calibration_error: float | None = None,
)

Holdout evaluation for a multiclass (K > 2) classifier on one split.

Attributes:

Name	Type	Description
split_name	str	Split identifier ("train" / "valid" / "test").
class_keys	tuple[str, ...]	Canonical class keys (string form of class labels), in model.classes_ order. Rows/columns of :attr:confusion_matrix and keys of :attr:per_class follow this order. Named class_keys — not classes — to avoid shadowing ProfilingResult.classes which holds the per-class profiles.
n_samples	int	Number of holdout rows scored.
accuracy, macro_f1, macro_precision, macro_recall		Macro-averaged metrics computed from the argmax predictions (sklearn.precision_recall_fscore_support(average="macro")), so they are consistent with accuracy / :attr:confusion_matrix and a stratified-random K-class model floors macro-F1 at 1/K (keeping the macro_f1 × K trust lift honest). No positive-class assumption. The per-class OvR operating points (with any tuned thresholds) live on :attr:per_class.
log_loss	float	Multinomial log-loss over every class (:func:parleml_ml4.delivery.scoring.proba_log_loss) — the number ml4 reports for the same model.
brier_score	float	The multiclass Brier score — the squared error between the one-hot target and the probability row, summed over classes and averaged over rows, in [0, 2] (:func:parleml_ml4.delivery.scoring.brier_score, sklearn's convention). The same number ml4's held-out calibration block reports for the same model.
confusion_matrix	ndarray	K × K matrix, rows = true class index, cols = predicted index, ordered by :attr:class_keys. Pure np.ndarray so the viewer can plot it directly.
per_class	dict[str, BinaryEvaluation]	{class_key: BinaryEvaluation} — one per-class OvR slice computed via :func:evaluate_classification_split. Each slice carries Brier / ECE / reliability curve, so per-class calibration diagrams come for free; a slice's brier_score is that class's one-vs-rest diagnostic on [0, 1], not a share of :attr:brier_score.
macro_expected_calibration_error	float | None	Mean ECE over the per-class slices — the headline for the viewer's macro strip.

proba property

proba: ndarray

Reconstruct the (m, K) probability sample from the OvR slices.

The single copy of the proba data lives in each per-class slice's raw_scores; stacking them in class_keys order (they share one subsample) rebuilds the matrix without a duplicate stored field.

confusion_matrix_df

confusion_matrix_df() -> DataFrame

Friendly DataFrame view of the K×K matrix, labelled with :attr:class_keys for easy display.

OptimalMetric dataclass

OptimalMetric(
    metric_name: str, value: float, threshold: float
)

Best value of a scalar metric and the threshold that achieved it.

The persisted raw sample is not always an unbiased draw

A split stores only the raw (y_true, scores) arrays plus cheap scalars; every curve (ROC, PR, reliability, lift/gains, threshold sweep, business value) is re-derived from them on demand. Two facts matter if you read those arrays yourself:

• Headline scalars are exact. roc_auc, brier_score, expected_calibration_error, the at-threshold metrics and the optimal_* fields are all computed on the full split before any capping.
• The arrays may be capped and class-stratified. Above RAW_EVAL_SAMPLE_CAP rows the sample is subsampled, and at rare-event prevalence it is deliberately enriched with minority rows so the re-derived curves are not positive-class noise. That enrichment makes the sample's base rate different from the split's.

raw_sample_weight is how the two are reconciled: None means the sample is an unbiased view (whole split, or a uniform draw) and every row counts once; otherwise it is a per-row population weight whose sum is n_samples.

Every prevalence-sensitive statistic must consume it — precision, the observed positive rate per reliability bin, the gains baseline, and any per-row expected value all shift with the base rate. parleml_mli.evaluation.derive.* and parleml_mli.evaluation.plots.* already do; a custom consumer should too:

split = result.model_evaluation_.splits["test"]
precision, recall = derive.pr_curve(
    split.raw_y_true, split.raw_scores, sample_weight=split.raw_sample_weight,
)

Rank-only statistics (ROC, KS) are invariant to a per-class-constant weight, but they accept it too, so no caller has to know which kind it is holding.

RawSample

Bases: NamedTuple

Row indices for a persisted raw sample plus their population weights.

Attributes:

Name	Type	Description
index	ndarray	Sorted row indices into the full split.
weight	ndarray | None	None when the sample is an unbiased view of the split — either it is the whole split, or it is a uniform draw (constant weights, which every normalised statistic we derive is invariant to). Otherwise a float64 vector aligned with :attr:index giving how many population rows each sampled row stands for, so weight.sum() recovers the split size.
A stratified draw (see	func:`subsample_index`) is *not* an unbiased view:
it deliberately over-samples the minority class. Every prevalence-sensitive
statistic re-derived from such a sample — the PR curve, the reliability
diagram, lift/gains, the score distribution, the business-value sweep — must
consume	attr:`weight` or it reports the model's behaviour at the wrong base
rate (mli review H1). Rank-only statistics (ROC) are invariant either way but
take the weight too, so no caller has to know which is which.

subsample_index

subsample_index(
    n: int,
    *,
    cap: int,
    rng: Generator,
    sample_index: ndarray | None = None,
    stratify_labels: ndarray | None = None
) -> RawSample

Rows (and their population weights) for the persisted raw sample.

Returns sample_index verbatim when the caller supplies one (so multiclass OvR slices and the parent y_true_idx all share identical rows); otherwise the full range when within cap, else a sorted cap-sized draw. Single home for the capping policy so every evaluator subsamples the same way.

When stratify_labels (a binary 0/1 vector aligned to n) is given and a draw is needed, the sample is class-stratified: every minority row is kept when the minority class comfortably fits, else the classes are sampled in proportion. An unstratified draw at a rare-event prevalence would keep only a prevalence-thinned handful of positives, so the re-derived ROC/PR/reliability/gains curves would carry positive-class noise the headline scalars (computed pre-cap) don't share (LOC-N8).

Stratifying trades that noise for bias: the sample's base rate is not the split's. The returned :attr:RawSample.weight is the inverse inclusion probability per class (n_class_total / n_class_kept) that undoes it, and is None for the draws that need no correction (mli review H1).

weighted_class_totals

weighted_class_totals(
    y_bin: ndarray, sample_weight: ndarray | None = None
) -> tuple[float, float]

(n_pos, n_neg) in population units, for the gains/lift baseline.

The ranking table's baseline is the split's base rate, so its class totals must be weighted whenever the raw sample is a stratified draw.

Multiclass operating points are tuned on one split

Per-class OvR thresholds are tuned on the first split in ModelEvaluationResult.splits and frozen for the rest — multiclass_tuning_split records which one that was. That split's own per_class metrics are read at an in-sample-tuned threshold and are therefore optimistic; every later split is leak-safe. fit_profiler always puts train first.

Calibration

compute_calibration

compute_calibration(
    y_true: ndarray,
    y_score: ndarray,
    *,
    n_bins: int = 10,
    strategy: CalibrationStrategy = "quantile",
    min_samples: int = DEFAULT_MIN_SAMPLES_FOR_CURVE,
    sample_weight: ndarray | None = None
) -> tuple[float, CalibrationCurve | None, float]

Convenience wrapper returning (brier, curve, ece) in one call.

Used by :func:parleml_mli.evaluation.metrics.evaluate_classification_split and by per-class OvR slices inside MulticlassEvaluation. y_true is the {0, 1} positive-class indicator and y_score the positive-class probability; the Brier score is :func:parleml_ml4.delivery.scoring.brier_score over the [1 - p, p] frame — the positive-class Brier, brier_score_loss(y_true, y_score). sample_weight is forwarded to both the Brier score and the curve, and the ECE then inherits the weighted bin counts. Labels outside {0, 1} and scores outside [0, 1] raise ValueError — ml4's Brier score would otherwise clip them into range and report a plausible number.

Business value

evaluate_business_value

evaluate_business_value(
    split: ClassificationSplitEvaluation,
    *,
    cost_fp: float = 0.0,
    cost_fn: float = 0.0,
    gain_tp: float = 0.0,
    gain_tn: float = 0.0,
    fbeta: float | None = None,
    capacity_top_k: int | None = None,
    capacity_fraction: float | None = None,
    objective: Objective = "expected_value"
) -> BusinessValueCurve

Cost-matrix / capacity / Fβ lens over an evaluated classification split.

Cheap (one threshold sweep over the raw sample, no model / dataset access) so the Diagnose page can call this on every slider drag. Derives the per-threshold confusion counts + capacity via :func:parleml_mli.evaluation.derive.threshold_sweep for the expected-value and capacity arithmetic, and reuses split.optimal_f{05,1,2} (or a small custom-β recompute) for the Fβ recommendation.

With a capacity (capacity_top_k / capacity_fraction) every recommendation respects it: the expected-value and Fβ optima are taken over the thresholds that flag at most that fraction, the free optima land in unconstrained (so the cost of the cap is visible), and at_capacity states precision / recall / lift / counts at the selected sampled threshold, which can underfill the budget. Ties stay together; this is not an exact top-k quota. If no sampled threshold respects an explicit cap, raise rather than exceed it.

Parameters:

Name	Type	Description	Default
split	ClassificationSplitEvaluation	The output of :func:parleml_mli.evaluation.metrics.evaluate_classification_split (or a per-class OvR slice from a multiclass evaluation). Must carry the raw (raw_y_true, raw_scores) sample — joblibs that predate the raw-sample refactor must be rebuilt via compute_model_evaluation.	required
cost_fp	float	Cost-matrix entries in per-row utility units (typically dollars). Default zeros render the EV curve uninformative; pass the actual costs for the expected_value objective to be meaningful.	0.0
cost_fn	float	Cost-matrix entries in per-row utility units (typically dollars). Default zeros render the EV curve uninformative; pass the actual costs for the expected_value objective to be meaningful.	0.0
gain_tp	float	Cost-matrix entries in per-row utility units (typically dollars). Default zeros render the EV curve uninformative; pass the actual costs for the expected_value objective to be meaningful.	0.0
gain_tn	float	Cost-matrix entries in per-row utility units (typically dollars). Default zeros render the EV curve uninformative; pass the actual costs for the expected_value objective to be meaningful.	0.0
fbeta	float | None	Which β to use for the fbeta recommendation. 0.5 / 1 / 2 reuse the existing split.optimal_f{05,1,2} argmax for free; any other β triggers a small on-the-fly recompute from the derived threshold sweep.	None
capacity_top_k	int | None	Mutually exclusive. capacity_top_k flags an absolute population cap (e.g. "we can only review 500 cases / week") that's translated to a fraction internally; capacity_fraction sets the fraction directly. None for both → the capacity recommendation is mapped to the τ that flags the base rate (the natural "predict the prevalence" baseline).	None
capacity_fraction	int | None	Mutually exclusive. capacity_top_k flags an absolute population cap (e.g. "we can only review 500 cases / week") that's translated to a fraction internally; capacity_fraction sets the fraction directly. None for both → the capacity recommendation is mapped to the τ that flags the base rate (the natural "predict the prevalence" baseline).	None
objective	Objective	Which τ from recommended should be flagged as recommended_objective. The full dict is always returned — this only chooses the headline pick and the confusion_at_recommended slice.	'expected_value'

Returns:

Type	Description
BusinessValueCurve	Dataclass carrying the full per-threshold curve plus the recommended τ for each objective. Backend projects scalar headlines into the FastAPI BusinessValueResponse.

BudgetPoint dataclass

BudgetPoint(
    threshold: float,
    flagged_fraction: float,
    flagged: int,
    tp: int,
    fp: int,
    fn: int,
    tn: int,
    precision: float,
    recall: float,
    lift: float,
)

What the selected capacity threshold yields (scores >= threshold, ties together). The sampled threshold may underfill the budget. Counts are in split units (the raw sample's weights rescale a capped sample); lift is precision over the split's positive rate.

Multiclass cost lens

The N×N analogue of the binary business-value lens: a full K×K cost matrix and the Bayes-optimal minimum-expected-cost decision rule (argmin_j Σ_i P(i|x)·C[i, j]) — no threshold to sweep.

evaluate_multiclass_cost

evaluate_multiclass_cost(
    evaluation: MulticlassEvaluation,
    *,
    cost_matrix: ndarray
) -> MulticlassCostResult

Apply the minimum-expected-cost decision rule to a multiclass split.

Parameters:

Name	Type	Description	Default
evaluation	MulticlassEvaluation	A published :class:MulticlassEvaluation (its proba / y_true_idx raw sample is the input).	required
cost_matrix	ndarray	(K, K) array, cost_matrix[i, j] = cost of predicting j when the true class is i (row = truth, col = prediction), in class_keys order.	required

Returns:

Type	Description
MulticlassCostResult

MulticlassCostResult dataclass

MulticlassCostResult(
    class_keys: tuple[str, ...],
    cost_matrix: ndarray,
    confusion_baseline: ndarray,
    confusion_cost_optimal: ndarray,
    total_cost_baseline: float,
    total_cost_optimal: float,
    per_row_cost_baseline: float,
    per_row_cost_optimal: float,
    n_evaluated: int,
    n_samples: int,
    is_sampled: bool,
)

Cost-optimal vs argmax decision over a multiclass split's raw sample.

All counts/totals are on the n_evaluated sampled rows; per_row_cost fields let callers scale to n_samples (the full split).

Label-noise scan

Confident-learning review queue of likely-mislabelled rows, with the estimated share of wrong labels (classification only). See the Label-noise scan guide section.

compute_label_noise

compute_label_noise(
    y_true: Sequence[Any] | ndarray | Series,
    oof_probabilities: ndarray,
    *,
    classes: Sequence[Any] | None = None,
    row_index: Sequence[int] | ndarray | None = None,
    method: Method = "auto",
    max_suspects: int = _DEFAULT_MAX_SUSPECTS,
    oof_source: str = "provided",
    threshold_rows: ndarray | None = None
) -> LabelNoiseResult

Confident-learning label-noise scan over out-of-fold probabilities.

Parameters:

Name	Type	Description	Default
y_true	Sequence[Any] | ndarray | Series	Given labels, one per row. When a :class:pandas.Series, its index is used as row_index unless row_index is passed explicitly.	required
oof_probabilities	ndarray	(n, K) out-of-fold class probabilities whose columns follow classes (a 1-D binary positive-class score is widened to [1-p, p]).	required
classes	Sequence[Any] | None	Class label per probability column. Defaults to sorted(unique(y)) — pass the model's classes_ when the column order differs.	None
method	Method	confident_joint (default via auto) uses per-class self-confidence thresholds and is robust to imbalance; self_confidence is a simpler threshold-free fallback (flag when the model's top class disagrees with the label).	'auto'
max_suspects	int	Cap on the surfaced suspects table (the rate is still over all rows).	_DEFAULT_MAX_SUSPECTS
threshold_rows	ndarray | None	Boolean mask of the rows whose probabilities set each class's confident_joint threshold (default: every row). Pass it when the rows were scored by different models: the thresholds then come from one model's scale and are applied to every row. A class with no row in the mask takes its threshold from all of its rows.	None

Notes

estimated_noise_rate is not n_flagged / n_total. A row the model is confident about no class for (confident_joint only) is evidence neither way, so it is left out of its class's share rather than counted as a clean row: per given class, the rate is the flagged share of its confidently predicted rows, and the overall rate weights those shares by how common each class is. That is the calibrated confident joint's off-diagonal mass, and it is at least the flagged share. Under self_confidence every row counts, so the two coincide.

build_label_noise_result

build_label_noise_result(
    y_true: Sequence[Any] | ndarray | Series,
    *,
    oof_probabilities: ndarray | None = None,
    estimator: "ClassifierMixin | None" = None,
    x: DataFrame | ndarray | None = None,
    classes: Sequence[Any] | None = None,
    row_index: Sequence[int] | ndarray | None = None,
    method: Method = "auto",
    max_suspects: int = _DEFAULT_MAX_SUSPECTS,
    cv_folds: int = 5,
    random_state: int = 0
) -> LabelNoiseResult

Build a :class:LabelNoiseResult, resolving out-of-fold probabilities.

Prefers caller-supplied oof_probabilities (e.g. persisted OOF from the engine). Falls back to sklearn.model_selection.cross_val_predict(estimator, x, y_true, method="predict_proba") for a bring-your-own sklearn estimator with its input frame x. Raises when neither is available.

LabelNoiseResult dataclass

LabelNoiseResult(
    suspects: DataFrame,
    estimated_noise_rate: float,
    per_class_noise: dict[Any, float],
    n_flagged: int,
    n_total: int,
    method: str,
    oof_source: str,
    classes: list[Any] = list(),
)

Likely-mislabelled training rows and the estimated noise rate.

A standalone evaluation artifact (like :class:~parleml_mli.evaluation.business_value.BusinessValueCurve): returned by :func:compute_label_noise / :func:build_label_noise_result and rendered by :func:parleml_mli.evaluation.plots.plot_label_noise — not a profiler stage.

Next

• Evaluation, trust & model cards
• Trust summary