Association rules¶

AssociationMiner mines frequent itemsets and statistically sound association rules (market-basket analysis) from basket-shaped data. Where the recommender and LTR backends score per entity, association rules describe the catalogue globally — "customers who hold A and B also hold C" — as auditable, business-readable statements with support / confidence / lift and a significance verdict attached. Same data, a third lens.

In short

Pick the input shape with grain (long, event or basket) and start from ASSOCIATION_PRESETS["balanced"].
The shipped rules are the rows of result.rules with a non-null rank.
The support floor is searched for you; every rule is tested for significance and pruned against its shorter sub-rules.
q_value is not a calibrated false-discovery rate. To quote a rule, set holdout_fraction and read lift_holdout.
Read lift_breadth_adjusted beside lift: broad baskets inflate plain lift.

from parleml_ml4.association import AssociationMiner
from parleml_ml4.config.association import ASSOCIATION_PRESETS

cfg = ASSOCIATION_PRESETS["balanced"](
    grain="long",
    entity_col="customer_id",
    target_key_col="product",
    target_group_col="category",   # optional — display / network colouring only
)
result = AssociationMiner(cfg, outputs_dir="runs/assoc").fit(df)

ranked = result.rules[result.rules["rank"].notna()]     # the shipped rules
ranked[["antecedents", "consequents", "support", "confidence", "lift"]].head()

No extra needed

Mining runs on mlxtend, which is a core ml4 dependency — nothing extra to install. There is no predict: the deliverables are the rule and itemset tables.

Input grains¶

AssociationConfig.grain selects the input shape:

long (default)eventbasket

One row per (entity, item) holding — product holdings, subscriptions. The basket is all items held by an entity.

cfg = ASSOCIATION_PRESETS["balanced"](
    grain="long", entity_col="customer_id", target_key_col="product")

A transaction log (one row per purchase). Rows are grouped into baskets per entity, optionally windowed by a timestamp (basket_time_granularity, a pandas floor freq — e.g. "D" for one basket per customer-day) or an explicit basket_col.

cfg = ASSOCIATION_PRESETS["balanced"](
    grain="event", entity_col="customer_id", target_key_col="product",
    timestamp_col="ts", basket_time_granularity="D")

One row per basket with a delimiter-packed items column. Carries no entity id, so no opportunity-audience export.

cfg = ASSOCIATION_PRESETS["balanced"](
    grain="basket", items_col="items", item_delimiter=",")

The quality stack¶

Raw frequent-itemset mining produces thousands of redundant, spurious rules. The miner applies six defenses. All are on by default at the balanced tier except the holdout (defense 5), which is opt-in because it changes what the headline numbers mean:

Auto-tuned support floor. Rather than guess min_support, a bounded deterministic search targets target_itemset_count multi-item frequent itemsets — singletons never yield a rule, so they do not count (the probe trail and the reason the search stopped are recorded on the result). In auto mode every itemset is backed by at least 5 baskets; pinning min_support skips the search and that floor, so a low pin will report itemsets backed by one or two baskets (the run warns when it does). target_itemset_count is a floor-pushing target, not a cap — the independent upper guard is max_lattice_factor, which makes the search ascend if a probe overshoots.
Statistical significance. Each rule gets a one-sided Fisher's exact test (is the confidence genuinely above the consequent's base rate?), corrected across rules with Benjamini–Yekutieli FDR at significance_alpha (default 0.05). Yekutieli rather than Hochberg because rules from an overlapping itemset lattice are dependent in both directions, which voids Hochberg's assumption. Insignificant rules are retained but flagged (significant=False, rank=NA) so they never ship silently when significance=True — with it off, they are ranked and shipped, and only the flag records the test outcome.

q_value is not a calibrated FDR

It is adjusted across the rules that survived the support, confidence and lift filters — not across the rules the search considered. Those filters select on the same counts the test measures, so q_value understates the true false-discovery rate, most for rules near the floors. fwer_significant is the assumption-free companion: p_value × candidate_family_size ≤ alpha, a Bonferroni bound over the whole search space that holds under any dependence. Neither replaces validating rules on held-out baskets, which is what holdout_fraction (defense 5 below) provides.
Interpretability (minimum-improvement) pruning. A rule {A,B} → C must beat every proper sub-rule with the same consequent — A → C, B → C, and the empty antecedent {} → C (i.e. C's own base rate) — by min_improvement confidence, or it is flagged with the winning sub-rule's id in pruned_by (or "baseline" for the base-rate case). Checking only the immediate sub-antecedents would be strictly weaker: confidence is not monotone in the antecedent, so a 3-item rule can beat every 2-item parent while being far worse than a 1-item ancestor. With a fixed consequent a confidence gain is a proportional lift gain (lift = confidence / P(consequent)), so the longer rule ships only when its extra items genuinely add strength — in the style of the filter Meta describes for its production root-cause miner (Fast Dimensional Analysis, arXiv:1911.01225). min_improvement=0 does not disable pruning — the comparison is a strict <, so a rule whose confidence is below its best sub-rule is still flagged. Only a negative margin would be a true off switch, and that is not permitted.

Two further tests stop a passenger item from riding on a strong rule. With significance on, a longer rule must be productive: each extra item must significantly raise the consequent's rate. With or without significance, it must also beat its shorter rule at equal basket breadth by min_superset_lift_gain (default 0.10).

Details: the productive-rule and equal-breadth tests

A fixed margin is not enough on its own. On a strong rule a passenger item clears it by sampling noise — {diapers, yogurt} → baby_wipes at 0.63 against diapers → baby_wipes at 0.61 — and then outranks its own parent on lift. So, with significance on, a longer rule must also be productive (Webb, Discovering significant patterns, 2007): for each antecedent item, among the baskets holding the rest of the antecedent, the consequent must be significantly more frequent with that item than without it (one-sided Fisher, Holm-adjusted across every such test at significance_alpha). A rule that fails is flagged against the sub-rule whose extra item had the weakest evidence. Holm (family-wise) rather than an FDR correction on purpose: the free riders a looser correction lets through are exactly the ones whose chance surplus puts them at the top of the ranking.

Both tests are fooled by basket breadth — see defense 4 — so a third one asks the productive-rule question at equal breadth: for each antecedent item, among the baskets holding the rest of the antecedent and grouped by how many items they hold besides the consequent, the Mantel–Haenszel rate ratio of the consequent with the item against without it (Greenland & Robins 1985) must reach 1 + min_superset_lift_gain (default 0.10). A rule that falls short is flagged against the shorter rule (pruned_by), which carries the finding; rule_stats.n_pruned_behind_sub_rule counts them. When an item is never compared with its absence at equal breadth (every basket holding it is broader than every basket without it), breadth and the item cannot be told apart and the rule is kept. This gate runs with or without significance.
Breadth-adjusted lift. Lift compares a rule with independence over all baskets. When baskets differ in breadth — a customer holding six products against one holding two, a weekly shop against a top-up — broad baskets hold more of everything, so every pair of items co-occurs above chance, lift rises with breadth alone, and a longer rule always out-lifts the pair inside it. On customer holdings that is most of the lift: on one live cross-sell run, fifteen triples at lift 3.4 had a lift of about 1.05 among customers holding the same number of products.

Every rule therefore carries lift_breadth_adjusted (and lift_breadth_adjusted_holdout on a holdout run). Read values within a few percent of 1 as breadth, not association.

Details: how the breadth adjustment is computed

Baskets are grouped by their rest score — the number of mined items they hold besides the rule's consequent — and the rule's co-occurrences are divided by the Mantel–Haenszel expected count, Σ_j n_A,j · n_C,j / N_j (Mantel & Haenszel 1959). With one group it is exactly the lift. Under the pure-breadth (Rasch) model the rest score is a sufficient statistic for breadth, so the adjusted lift is 1 for any antecedent and any catalogue size; matching on such a score is the item-response test of association beyond a common trait (Holland & Thayer 1988; Rosenbaum 1984). Real breadth is not exactly Rasch, so read values within a few percent of 1 as breadth. The two nearby conditionings are biased: the basket size counts the consequent itself (true independence then reads as low as 0.67 on a seven-item catalogue), and the items outside both sides leave the antecedent to carry breadth (a pure breadth effect reads 1.2–1.4). See parleml_ml4.association.breadth.

parleml_ml4.association.breadth.summarise_breadth(result.rules) gives the median share of the ranked rules' lift above 1 that breadth explains; past half, the run emits a warning saying so — the list then describes broad baskets more than pairs of items. BreadthStrata(onehot).strata(A, C) returns the per-group counts behind one rule (the evidence drawer shows them).
Holdout validation (opt-in — holdout_fraction). Everything above is measured on the baskets that chose the rules, so none of it can tell you whether a rule is an artifact of the mined sample. Set holdout_fraction=0.25 and the miner withholds a random quarter of baskets before mining: the lattice, the rules and every gate come from the train side alone, and each surviving rule is then re-measured on baskets that had no say in its selection.

Each rule gains support_holdout / confidence_holdout / lift_holdout and a holds_up flag (holdout lift back above the run's own min_lift — the bar the rule cleared to be shipped, not the weaker "better than independence"), and the result carries a HoldoutEvaluation whose median_lift_ratio is the headline: the median of lift_holdout / lift over the ranked rules. Near 1.0 means the shipped list generalises; well below means the mining sample flattered it — which no amount of multiple-testing correction would have caught.

A rule whose antecedent is too rare in the holdout is scored NaN, not a confident number from three baskets; those are counted separately as unmeasurable and are neither confirmed nor refuted.

It changes what the headline numbers mean

With a holdout, support / confidence / lift describe the train baskets, not all of them. That is the correct trade — you cannot validate on data you mined — but it is why this is off by default rather than imposed on every run. The split is random over baskets, so it answers "is this rule an artifact of my sample?", not "will it hold next quarter?".
Ubiquitous-item guard. Items present in more than max_item_support (default 0.95) of baskets — "everyone holds the base plan" — flood the lattice with junk and are auto-excluded pre-mining, with a warning recorded on the result. focus_items are exempt; max_item_support=1.0 disables it.

Steering the output¶

exclude_items — removed before mining. Only the items leave; their baskets stay, so every other rule's support denominator is unaffected.
focus_items + focus_side ("any" / "antecedent" / "consequent") — keep only rules touching these items on the given side. Focus items also bypass the ubiquitous guard and dig the auto-support floor deeper for rare targets. exclude_items and focus_items may not overlap (a contradiction, raised at config time).
max_itemset_len — the longest itemset mined (native mlxtend max_len), capped at 10: it is the exponent of the whole run's cost.
max_lattice_factor — the mined lattice may not exceed max_lattice_factor × target_itemset_count. Checked on every probe including the first, and the search ascends when one overshoots. This, not target_itemset_count, is the upper guard.
max_consequent_len (default 1) / max_antecedent_len — how many items each side of a rule may hold. mlxtend enumerates every antecedent/consequent split of each frequent itemset (2^k − 2 per k-itemset), so these bound the bulk of the candidates. Single-item consequents are the default because they are what makes a rule actionable ("add this one item"); set max_consequent_len=None for the unrestricted behaviour.
min_confidence / min_lift — floors applied before ranking.
rank_by ("lift" (default) / "confidence" / "leverage" / "conviction" / "zhangs_metric") + max_rules — the final rank-and-cap.
one_rule_per_itemset (default True) — each item set takes at most one ranked slot. A → B and B → A are one co-occurrence read in two directions, and lift is symmetric, so without this every finding filled two slots of a short list (and every split of a larger set — {a,b} → c, {a,c} → b — one more). The best rule of the set by rank_by ranks (ties go to the higher confidence, the more reliable reading); the other directions stay in the result, unranked, with pruned_by naming that rule. False ranks every direction.
max_rules caps each returned block separately, so the frame is bounded either way: the ranked rules; the runners-up — rules that passed every gate but ranked beyond the cap, kept unranked, best first, with pruned_by = "max_rules" so the rules a short list left out can still be read; and the flagged (redundant or insignificant) rows. result.rule_stats records what each cap and each filter removed — n_ranked_dropped_by_cap is every rule beyond the cap, n_dropped_by_cap_returned the runners-up kept.

Budget tiers¶

Construct through ASSOCIATION_PRESETS[...] (or the association_fast / association_balanced / association_thorough factories) and override only the task columns:

	`fast`	`balanced`	`thorough`
`target_itemset_count`	500	2 000	10 000
`max_itemset_len`	3	4	5
`max_rules`	200	500	2 000
`min_improvement`	0.02	0.01	0.005

A tier trades breadth for speed, never evidence for speed: all three keep significance on. The Fisher + FDR pass is a handful of vectorised calls that run either way, so switching it off saves no time — it only ships unvetted rules. association_balanced() is exactly AssociationConfig().

Result & artifacts¶

AssociationResult carries the full ledger, both browsable and publishable:

Attribute	Meaning
`rules`	Full rule ledger — ranked, runner-up, insignificant, and pruned rows all present, distinguishable by `rank` / `significant` / `pruned_by` (a rule id it is redundant against, `"baseline"`, or `"max_rules"` for a runner-up beyond the cap).
`itemsets`	All frequent itemsets the rules were built from.
`item_stats`	Per-item support + the exclusion ledger. Measured over all baskets, so it is paired with `n_baskets`, not `n_baskets_mined`.
`holdings`	One row per `(entity, held item)` over the mined vocabulary — the audience fact (or `None` for the basket grain).
`n_baskets`	The whole population.
`n_baskets_mined`	The denominator every rule `support` is a share of. Equal to `n_baskets` without a holdout; the train count with one. Anything turning a support back into a basket count must divide by this — using `n_baskets` overstates the evidence by `1 / (1 - holdout_fraction)`.
`holdout`	The out-of-sample evaluation (`None` when no holdout ran), including how many ranked rules held up, how many were unmeasurable, and how many the holdout refuted.
`rule_stats`	The generation funnel — what each filter and each cap removed; `n_pruned_behind_sub_rule` counts the rules the superset gate set aside.
`lift_inflation_factor`	How much retained empty baskets scale every reported lift.

Save with result.save("assoc.joblib") and reload with AssociationResult.load(...). With outputs_dir set, fit also writes the tables under deploy/association/.

Details: the artifact format and the run directory

save writes the joblib and then a JSON format stamp beside it (assoc.stamp.json), and load checks that stamp before unpickling anything. There is no compatibility layer: any field or column change bumps the major version, and an artifact whose stamp is missing or from another version is rejected with an actionable message rather than deserialised. Re-run the experiment to get a loadable artifact.

With outputs_dir set, fit emits the standard events.ndjson lifecycle (transactionize → mine_itemsets → generate_rules → persist, cancellable) and writes deploy/association/ — association_result.joblib with its association_result.stamp.json, rules.parquet, items.parquet, itemsets.parquet, and holdings.parquet (only when an entity id was resolvable).

Opportunity audiences (rule → targeting list)¶

Every rule points at an actionable audience: for {A,B} → C, the entities holding all the antecedent items but not (all of) the consequent — the opportunity the rule describes, minus the customers who already converted. This is a pure set operation over the holdings fact (a NOT EXISTS anti-join). The audience size is exact; a "yield reference" (size × the rule's confidence) is a labelled historical co-hold line, not a campaign forecast — measuring incremental effect is an uplift experiment's job.

Engine¶

Mining runs on mlxtend's H-Mine by default. engine="fpgrowth" is an explicit expert escape hatch — nothing selects it automatically — and it is 23–806× slower on the matrices this pipeline produces (measured across scripts/benchmark_association_engines.csv; the gap grows with basket count). The recorded benchmark is on Association mining engines.

Details: why fpgrowth is slower, and where the engines differ

The cause is not sparsity and not FP-growth itself: mlxtend's fpcommon.generate_itemsets, which fpgrowth calls and hmine does not, re-densifies the frame once per itemset and loops over every row in Python — null-value bookkeeping that is dead work on the null_values=False path this pipeline always takes. The cost is therefore Θ(n_baskets × n_itemsets).

The two engines produce the same itemsets except exactly at a support boundary: hmine admits on count ≥ ceil(min_support × n) while fpgrowth filters on the fraction count / n ≥ min_support, and those disagree whenever the float product rounds up across an integer. Treat them as interchangeable in practice, not bit-identical at the threshold.

Every AssociationConfig field is on the Configuration parameters page; the API is on the AssociationMiner reference, and there is a runnable walkthrough in the market-basket example.

Next: Example: market basket · Recommendation