AssociationMiner¶

Market-basket mining — frequent-itemset discovery (mlxtend hmine / fpgrowth), a one-sided Fisher exact test per rule with Benjamini–Yekutieli correction (valid under the arbitrary dependence an itemset lattice creates), minimum-improvement redundancy pruning against every proper sub-rule, and optional out-of-sample validation. See the association-rules guide.

In short

AssociationMiner turns baskets into ranked rules, configured by AssociationConfig.
Support is searched automatically to hit target_itemset_count (default 2,000); set min_support to pin it.
Every rule gets a Fisher p-value and a Benjamini–Yekutieli q-value. The q-value is not a calibrated false-discovery rate here.
Set holdout_fraction to measure the rules on baskets that did not select them.

AssociationMiner ¶

AssociationMiner(
    config: AssociationConfig,
    outputs_dir: Optional[str] = None,
)

Mine frequent itemsets + association rules with an honest quality ledger.

Parameters¶

config: The task configuration (columns / budget / constraints / quality). outputs_dir: When set, opts into the events / cancel / deploy discipline: fit writes events.ndjson + manifest.json and persists the result under <outputs_dir>/deploy/association/. None (default) keeps the run fully in-memory.

fit ¶

fit(df: DataFrame) -> AssociationResult

Run the four mining stages and return the result.

Result¶

AssociationResult `dataclass` ¶

AssociationResult(
    rules: DataFrame,
    itemsets: DataFrame,
    item_stats: DataFrame,
    n_baskets: int,
    n_items: int,
    resolved_min_support: float,
    support_search_trail: List[Tuple[float, int]],
    search_outcome: str,
    excluded_ubiquitous: Tuple[str, ...],
    excluded_by_config: Tuple[str, ...],
    engine: str,
    config: AssociationConfig,
    timings: Dict[str, float],
    rule_stats: Optional[RuleGenerationStats] = None,
    holdout: Optional[HoldoutEvaluation] = None,
    n_dropped_small_baskets: int = 0,
    n_blank_tokens_dropped: int = 0,
    n_blank_item_cells: int = 0,
    n_missing_entity_rows_dropped: int = 0,
    n_unparsed_timestamp_rows: int = 0,
    n_missing_basket_key_rows: int = 0,
    n_baskets_emptied_by_exclusion: int = 0,
    unmatched_exclude_items: Tuple[str, ...] = (),
    unmatched_focus_items: Tuple[str, ...] = (),
    basket_size_stats: Dict[str, float] = dict(),
    holdings: Optional[DataFrame] = None,
)

Rules + itemsets + the run's cleaning/search ledger.

n_baskets_mined `property` ¶

n_baskets_mined: int

The denominator every rule support is a share of.

Not always n_baskets. With a holdout, mining runs on the train split alone, so support is count / n_train while n_baskets still counts the whole population (see the holdout field). Anything reconstructing a basket count from a support must divide by this, or it overstates the evidence by 1 / (1 - holdout_fraction) — a third at the documented holdout_fraction=0.25.

item_stats is the deliberate exception: it describes the whole population and is paired with n_baskets.

lift_inflation_factor `property` ¶

lift_inflation_factor: float

How much retained empty baskets scale every reported lift.

n / (n - emptied). 1.0 when nothing was emptied. Published so the magnitude of a documented, deliberate modelling choice is visible instead of implicit.

save ¶

save(path: str | Path) -> None

Versioned joblib dump (zlib level 3), then its JSON format stamp.

Both writes are atomic, and the stamp goes last: a stamp beside the joblib says the joblib is complete.

load `classmethod` ¶

load(path: str | Path) -> 'AssociationResult'

Check the format stamp, then unpickle.

The stamp is read from its JSON sidecar before anything is deserialised, so an artifact whose stamp is missing, unreadable or of another format version is refused with an actionable ValueError and never unpickled.

rules_export_frame ¶

rules_export_frame() -> DataFrame

One flat row per rule — the assoc_rules fact shape.

Carries the same metric set as the CSV download: an analyst reading the published table and an analyst reading the export must be able to reach the same conclusion. pruned_by in particular is load-bearing — without it, WHERE significant returns redundancy-pruned rules as findings and no published column can tell them apart.

items_export_frame ¶

items_export_frame() -> DataFrame

One flat row per observed item — the assoc_items dim shape (raw names; SQL-safe slugging happens in the backend).

itemsets_export_frame ¶

itemsets_export_frame() -> DataFrame

One flat row per frequent itemset — the assoc_itemsets shape.

Combinations are a first-class deliverable beside the rules: the UI table, the workspace star schema, and the parquet download all read this view.

items carries the structured member list alongside the joined text. Without it, "does this combination contain item X" can only be answered by splitting itemset_text on the separator — which re-hardcodes the separator in the consumer and gives the wrong answer for any item whose own name contains it.

HoldoutEvaluation `dataclass` ¶

HoldoutEvaluation(
    n_baskets_train: int,
    n_baskets_holdout: int,
    n_rules_ranked: int,
    n_rules_held_up: int,
    n_rules_unmeasurable: int,
    median_lift_ratio: Optional[float],
    n_rules_refuted_absent_consequent: int,
    n_rules_in_lift_ratio: int,
)

What a holdout run proved, summarised over the ranked rules.

The one part of an association result that is not selection-conditioned: these baskets were withheld from mining, so they had no say in which rules exist. median_lift_ratio is the headline — the median of lift_holdout / lift across the ranked rules that could be measured. Near 1.0 means the shipped list generalises; well below 1.0 means the mining sample flattered it, which no multiple-testing correction would have caught.

held_up_share `property` ¶

held_up_share: Optional[float]

Share of measurable ranked rules that survived the holdout.

The denominator is every ranked rule the holdout could hold to account — including those it refuted outright — so a consequent that vanished from the withheld baskets cannot quietly raise the pass rate.

RuleGenerationStats `dataclass` ¶

RuleGenerationStats(
    n_candidates: int,
    n_after_shape: int,
    n_after_focus: int,
    n_after_lift: int,
    n_ranked: int,
    n_ranked_dropped_by_cap: int,
    n_dropped_by_cap_returned: int,
    n_flagged_total: int,
    n_flagged_returned: int,
    candidate_family_size: float,
    n_pruned_behind_sub_rule: int,
)

Honest counts for every stage of rule generation that removed rules.

The published rule frame is capped on both sides and filtered four times, so these are the only record of what was taken away. Lives here rather than in :mod:parleml_ml4.association.rules because it is part of the shape of a result — which keeps it, like everything else in this module, free of the mlxtend import.

Every field is a count of rules at one point in the pipeline, so the sequence reads as a funnel::

n_candidates -> n_after_shape -> n_after_focus -> n_after_lift
             -> n_ranked (+ n_ranked_dropped_by_cap, of which
                            n_dropped_by_cap_returned)
             +  n_flagged_total (of which n_flagged_returned)

n_pruned_behind_sub_rule is the part of n_flagged_total the superset gate flagged: rules with an antecedent item that did not raise the consequent's rate at equal basket breadth (min_superset_lift_gain).

stopped_at `classmethod` ¶

stopped_at(
    *,
    candidate_family_size: float,
    n_candidates: int = 0,
    n_after_shape: int = 0,
    n_after_focus: int = 0,
    n_after_lift: int = 0
) -> "RuleGenerationStats"

Stats for a run that produced no rules, keeping what is already known.

An empty result is exactly when the counts matter most — "mlxtend generated 40 000 candidates and the focus filter removed all of them" is actionable, all zeros is not.

Statistics¶

The multiple-testing layer is a standalone, scipy-only module — read its docstring before quoting a q-value, which is not a calibrated false-discovery rate as this pipeline uses it.

significance ¶

Multiple-testing layer for mined association rules.

Separated from :mod:parleml_ml4.association.rules because it is a distinct, directly testable concern with a much narrower dependency: everything here is plain numpy + scipy.stats over integer count arrays. Nothing touches mlxtend, and nothing needs a DataFrame — so these functions can be exercised against hand-computed values without mining anything.

Read this before trusting a q-value¶

:func:fisher_p_values is sound. The 2×2 table is recovered exactly from the support columns, and the one-sided test matches the positive-association claim the product makes.

:func:benjamini_yekutieli is not a calibrated false-discovery rate as this pipeline uses it, and no amount of care inside this module can make it one. The p-values handed to it come from rules that already survived the support, confidence and lift filters, and those filters select on the same co-occurrence counts the test measures — so the retained null p-values are stochastically smaller than uniform, and the family is orders of magnitude smaller than the one the search traversed. Both errors point the same way: q-values understate the true FDR, worst for rules sitting nearest the floors.

Two things mitigate that, and both live here:

Yekutieli, not Hochberg. BH controls FDR only under independence or positive regression dependence. Rules from an overlapping itemset lattice violate PRDS in both directions — {a,b}→c and {a,b,d}→c share a count (positive), while the 2^k-2 splits of one itemset are constrained by a common total (negative). BY is the member of the family that is valid under arbitrary dependence, at the cost of a ln(m) + γ factor.
A family-wide bound. :func:candidate_family_size counts every rule the configured search space contains, and :func:family_wide_significant applies Bonferroni against it. Crude, but it holds under any dependence and any selection, so a rule that passes is defensible without caveat.

Neither replaces validating rules on held-out baskets.

:func:improvement_p_values is the same exact test asked of a longer rule against its own shorter versions — does each antecedent item raise the consequent's rate among the baskets that already hold the rest? — which is what the redundancy pass in :mod:parleml_ml4.association.rules uses to drop rules whose extra item rides along. Those tests are corrected with :func:holm_adjusted (family-wise), not BY: an FDR correction lets a share of free riders through, and the ones it lets through are exactly those whose chance surplus lifts them to the top of the ranking.

candidate_family_size ¶

candidate_family_size(
    n_items: int,
    max_itemset_len: int,
    *,
    max_antecedent_len: Optional[int] = None,
    max_consequent_len: Optional[int] = None
) -> float

Every rule the configured search space contains.

sum_k C(n_items, k) × (admissible antecedent/consequent splits of a k-itemset). This is the multiple-testing family the search actually traverses — typically three to five orders of magnitude larger than the set of rules that survive to be tested, which is why the q-values alone are not a calibrated error rate.

Returned as float because the exact integer routinely exceeds what JSON consumers can carry.

basket_counts ¶

basket_counts(
    support: ArrayLike, n_baskets: int
) -> NDArray[int64]

Recover exact integer basket counts from support values.

Both mining engines report support as fl(count / n_baskets), so scaling back and rounding is lossless (see :data:_EXACT_COUNT_LIMIT). Doing it in one place keeps the reconstruction identical everywhere it is needed.

counts_from_supports ¶

counts_from_supports(
    antecedent_support: NDArray[float64],
    consequent_support: NDArray[float64],
    support: NDArray[float64],
    n_baskets: int,
) -> tuple[NDArray[int64], NDArray[int64], NDArray[int64]]

The exact integer basket counts (n_a, n_c, n_ac) of a rule table.

fisher_p_values ¶

fisher_p_values(
    n_a: NDArray[int64],
    n_c: NDArray[int64],
    n_ac: NDArray[int64],
    n_baskets: int,
) -> NDArray[float64]

One-sided Fisher exact p per rule, vectorised.

P(X >= n_ac) for X ~ Hypergeometric(population=n_baskets, successes=n_a, draws=n_c) — which is exactly what scipy.stats.fisher_exact(..., alternative="greater") computes internally, one Python call at a time. Calling the tail directly over arrays is the same number, without the per-rule interpreter overhead: this runs on every rule that reaches the test, which is bounded by the mined lattice rather than by max_rules.

The clip mirrors scipy's own min(p, 1.0) and is load-bearing — :func:benjamini_yekutieli rejects anything outside [0, 1].

Degenerate margins (an antecedent or consequent present in every basket) give a deterministic table and correctly yield p = 1.0.

improvement_p_values ¶

improvement_p_values(
    n_sub: NDArray[int64],
    n_with_item: NDArray[int64],
    n_sub_c: NDArray[int64],
    n_with_item_c: NDArray[int64],
) -> NDArray[float64]

One-sided p that an extra antecedent item raises the consequent's rate — Webb's test.

For a rule A→C and one item x ∈ A with Z = A \ {x}, the question is whether x adds anything to Z→C. The 2×2 table lives inside the n_sub baskets that contain Z: n_with_item of them also contain x (so contain A), n_sub_c contain C, and n_with_item_c contain both. Under the null that x is independent of C among Z's baskets, the overlap is hypergeometric, so this is Fisher's exact test of P(C | A) > P(C | Z, not x) — G. I. Webb, Discovering significant patterns, Machine Learning 68 (2007), the "productive rule" test.

Same tail as :func:fisher_p_values with the population narrowed from every basket to Z's baskets. An x that never leaves Z's side (n_with_item == n_sub) gives a deterministic table and p = 1: the longer rule then says exactly what Z→C says.

benjamini_yekutieli ¶

benjamini_yekutieli(
    p_values: NDArray[float64],
) -> NDArray[float64]

Adjusted p-values (q-values) valid under arbitrary dependence.

Thin, deliberate wrapper over :func:scipy.stats.false_discovery_control so the choice of "by" over "bh" lives at exactly one call site with the reason attached (see the module docstring).

holm_adjusted ¶

holm_adjusted(
    p_values: NDArray[float64],
) -> NDArray[float64]

Holm step-down adjusted p-values — family-wise error control, any dependence.

max_{j <= i} (m - j + 1) * p_(j) over the ascending p-values, capped at 1: uniformly more powerful than Bonferroni and as assumption-free. Written out rather than imported because the libraries that ship it (statsmodels) are not core dependencies, and it is three array operations.

family_wide_significant ¶

family_wide_significant(
    p_values: NDArray[float64],
    family_size: float,
    alpha: float,
) -> NDArray[bool_]

Bonferroni against the whole searched space: p × family_size <= alpha.

The assumption-free companion to the q-values — conservative, but valid under any dependence structure and any selection rule, so a rule that passes can be stated as a finding without caveat. All-False when the family size is unknown, which is the honest answer rather than an unearned pass.

Holdout validation¶

The only association statistic not measured on the baskets that selected the rule.

holdout ¶

Holdout validation — does a mined rule survive baskets it never saw?

Every other number this workload reports is in-sample. The rules were chosen using the same baskets their support, confidence and lift are measured on, and the significance layer inherits that: q-values are adjusted across rules that already passed a support / confidence / lift screen, so they understate the true false-discovery rate (see :mod:parleml_ml4.association.significance).

A holdout answers the question those statistics cannot. Baskets are split before mining; the support search, the lattice, the rules and every statistical gate are derived from the train side alone; then each surviving rule is re-measured on baskets that had no say in its selection. A rule whose lift collapses there was an artifact of the mined sample, and no amount of multiple-testing correction would have told you that. A rule that holds up is defensible regardless of the correction debate.

One documented exception. Transactionisation runs before the split, so the item vocabulary — exclude_items matching, min_basket_size, the max_item_support ubiquitous-item guard and the published item_support — is fitted on all baskets, holdout included. That is deliberate: cleaning is a data decision, not a mining decision, and a vocabulary that changed with the split would make the two sides incomparable. It is also a genuine (if second-order) leak — an item sitting either side of the ubiquity threshold can have that decision swung by the withheld baskets — and it is marginal-prevalence information only, which cannot manufacture a co-occurrence. Stated here rather than claimed away.

What this does and does not answer¶

The split is random over baskets, so it tests "is this rule an artifact of the sample I mined?" — the selection question. It is deliberately not a temporal split, which would answer a different question ("will this rule still hold next quarter?"). Conflating the two would let a drift check pass as a selection check.

Scoring is honest about what it cannot measure: a rule whose antecedent appears in too few holdout baskets gets NaN, never a confident number computed from two or three baskets.

BasketSplit `dataclass` ¶

BasketSplit(train: NDArray[int64], holdout: NDArray[int64])

Row positions of the mining set and the untouched evaluation set.

HoldoutScores `dataclass` ¶

HoldoutScores(
    support: NDArray[float64],
    confidence: NDArray[float64],
    lift: NDArray[float64],
    lift_breadth_adjusted: NDArray[float64],
    holds_up: NDArray[bool_],
    measured: NDArray[bool_],
)

Per-rule holdout metrics, positionally aligned with the rules frame.

split_baskets ¶

split_baskets(
    n_baskets: int,
    holdout_fraction: float,
    random_state: int,
) -> BasketSplit

Randomly partition basket row positions into train / holdout.

Deterministic for a given random_state so a rerun of the same data reproduces the same evaluation — a holdout number that moves on every run would be worse than none.

Raises :class:ValueError when either side would be too small to mean anything; silently shrinking to an unusable split would publish a validation column that is really just noise.

basket_rows ¶

basket_rows(
    onehot: DataFrame, rows: NDArray[int64]
) -> DataFrame

Row-slice the one-hot without losing the bool sparse dtype.

.iloc on a frame built by DataFrame.sparse.from_spmatrix silently degrades any column whose slice contains no True to Sparse[int64, 0]. mlxtend's valid_input_check then sees a non-bool frame and drops off its all-bool fast path onto the branch that re-densifies — precisely the cost the explicit dtype pin in :mod:parleml_ml4.association.frame exists to avoid. Splitting for a holdout must not quietly make mining slower.

score_rules_on_holdout ¶

score_rules_on_holdout(
    rules: DataFrame,
    onehot_holdout: DataFrame,
    *,
    min_lift: float = 1.0
) -> HoldoutScores

Re-measure each rule on the holdout baskets.

onehot_holdout must be a row slice of the same one-hot the rules were mined from, so the columns line up by name.

min_lift is the bar holds_up grades against, and it should be the run's own config.min_lift — the gate the rule cleared to be shipped. Grading against 1.0 instead asks only "is it better than independence?", which a rule selected at 1.10 and ranked by lift can pass with a holdout lift of 1.0004 while the product still narrates its train lift of 6.7×.

Antecedent and union masks are cached by itemset: rules from one lattice share sides heavily (every split of {a,b,c} reuses the same three columns), so the cache turns a per-rule scan into roughly a per-itemset one.

evaluate_holdout ¶

evaluate_holdout(
    scores: HoldoutScores,
    *,
    train_lift: NDArray[float64],
    ranked: NDArray[bool_],
    split: BasketSplit
) -> HoldoutEvaluation

Summarise the holdout over the ranked rules — the shipped list.

Flagged rules are scored too (the UI can show them), but the headline is about what the run actually recommends.

median_lift_ratio is the shrinkage number: the median of lift_holdout / lift. Near 1.0 means the ranked list generalises; well below means the mining sample flattered it. Rules with no computable holdout lift, or whose train lift is not a usable denominator, are excluded from the median rather than counted as either success or failure — note this is a narrower set than the one excluded from held_up_share, because a rule refuted by an absent consequent has no lift but is very much measured.

Because the ranked rules were selected and ordered by lift, regression to the mean pulls this ratio below 1.0 even when every rule is genuine, by an amount that grows with the lattice size and max_rules. Read it against a previous run of the same shape, not against 1.0 in the abstract.

Next: Association mining engines · Example: market basket

AssociationMiner¶