Playbook — recommendation and learning-to-rank

For experiments of type: "recommendation" (next-best product: one calibrated propensity model per product, blended into a top-K list) and type: "ltr" (learning-to-rank: one global ranking model over the products at each decision occasion). Both rank a catalogue of products for each entity; both read the same two input shapes (grain):

• event — a purchase log, one row per acquisition (entity, product, time); every row is a positive and the engine mines the negatives;
• long — a candidate frame, one row per (entity, candidate product) with a 0/1 label_col; the negatives are written out.

The plan carries a recommendation or an ltr block (the column mapping and its config; describe_platform("plan_schema", name="recommendation") or name="ltr"). The delivery lifecycle is get_playbook("deliver"); this playbook is what each of its steps does here.

Frame

• The decision — who receives the list, how many items they are shown (config.k), and what a good list is: the bought item near the top (primary_metric: ndcg, mrr, hit), or reach across the catalogue (ranking_objective: accuracy, balanced, coverage).
• The floor the person cares about — popularity (the bestsellers), and a per-segment popularity when the person names one: the model has to beat it at the person's K to be worth serving.
• The engine — the requirement choice. recommendation when products differ a lot and each has enough buyers (min_positives per product), and per-product models are wanted (their drivers, drift and cost lens); ltr when the catalogue is large or long-tailed, or a decision occasion (bundle_col) groups several purchases. A per-product requirement (regulated, real time) is config.submodel_template_name — any supervised template for the sub-models.
• States and eligibility — establish whether held products or win-backs are allowed. holdings_col and prior_holdings_col route new / existing / returning submodels; they do not exclude held products. Both engines' evaluation and published lists rank the full fitted catalogue, including products absent from an entity's long-grain candidate rows: an ltr run is no closer to "order only the offered rows" than a recommendation run. Missing rows are not an eligibility declaration. No matched eligible-only policy is exposed by the current recommendation plan. Identify that limitation before fitting when exclusion is required. Proceed with an unfiltered offline baseline only when that scope is authorized; do not claim its metrics validate a filtered shortlist or invent an eligibility setting.
• The prediction moment — on the event grain the timestamp_col; features are the entity's state before that moment. A latest-event row is still a pre-acquisition snapshot, not automatically the customer's current holdings. Label lists from it as historical illustrations, or construct a justified post-event state that includes the observed acquisition and discloses unknown later changes.
• Why this product — when a person must explain a list (a call-centre script): a recommendation run publishes, per recommended product, its holding (held, previously held, not held) and the top drivers of that product's sub-model for the entity (reason_<j>_*) with a reason_text to say out loud; an ltr run publishes no reasons. That can decide the engine.
• Metric — describe_platform("metric_catalog", context="recommendation") or context="ranking".

Data

• Recipe — the event log or the candidate frame, bounded at the moment (author_recipe). Events dated after the moment (or after today) are dropped and reported.
• Roles — on the plan's block: entity_col, target_key_col (the product), label_col (long grain only), timestamp_col, target_group_col (a product category, never a feature), holdings_col / prior_holdings_col (recommendation), bundle_col (ltr). feature_cols lists the entity features (or is omitted for every column that is not a role and not ignored). In long-grain recommendation, features must be constant across an entity's candidate rows. Event-grain recommendation permits features that change across events; its evaluation projects those varying features from the previous event of that entity (first events have missing history), while entity-constant features keep their values. This is not a single first/last customer snapshot. Check that this history interpretation matches the supplied feature semantics; an already pre-event feature can consequently be older than intended. LTR features describe the decision occasion and must agree within its candidate group. A product-derived attribute can identify the answer and must not become a predictor. Do not also list a role column in feature_cols. Check the fitted pipeline's realized encodings: assigning a tag-list domain on the dataset does not establish that the ranker built per-tag features. Report raw categorical treatment when that is what the model used.
• Leakage — prepare_data(action="leakage"); the leak is a feature measured after the purchase (a tenure counted to today, holdings that include the bought product, a spend that includes it). Excluding a whole driver family for a suspected snapshot leak costs signal: prefer an as-of version of it (computed before each event) and say what the exclusion costs.
• Signal — no probe: say so at the data stop.

Validation

Recommendation: a holdout of entities (config.holdout_fraction), ordered by each entity's first appearance when timestamp_col is set. Whole entities stay together; their event ranges can overlap across sides. This measures unseen-entity performance, not a strict future-time backtest. Each product's sub-model also carries its own validation and test split. For long-grain recommendation, ranking grades one row per positive customer/product outcome, repeating a customer's score list for each positive. Customers with no positive outcome do not enter those ranking metrics. Report both graded rows and evaluated customers and distinguish them from the full outer holdout. This is positive-outcome-weighted ranking, not customer-averaged multi-relevance NDCG or an estimate of conversion across all customers. Entity bootstrap uncertainty preserves customer grouping but does not change that estimand. LTR: config.split_strategy — auto, temporal (fit on the past, rank the future: matches deployment), entity (cold start), random (optimistic whenever an occasion is finer than a customer). Ranking metrics at K are measured on the holdout and compared with popularity. LTR reserves that outer holdout before tuning: Optuna receives only the training side, and the catalogue and popularity reference are built from training rows. The final ranking report is therefore separate from trial selection within that fit. Repeatedly choosing new experiments from this same report still turns it into selection evidence; it is not a fresh test for each new scientific decision. The summary exposes the resolved split strategy and holdout occasions omitted because their products are outside the fitted catalogue.

The comparison must use the same K, candidate catalogue, eligibility/holdings policy and population. Changing K or filtering published lists changes the policy; the unfiltered metric does not certify the filtered list. Use the ranking report's paired model-minus-popularity interval where returned, rather than comparing marginal confidence intervals. Observed purchases reflect exposure and availability; an unpurchased or sampled negative is not necessarily disliked. LTR scores are squashed ranking scores, not calibrated purchase probabilities. Per-product propensities also require the population and prior-correction caveats of their fitted models.

Baseline

On an acquisition event log — one row per customer purchase (event grain):

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "cross-sell baseline", "type": "recommendation",
    "dataset_id": "<dataset_id>",
    "recommendation": {
      "grain": "event", "entity_col": "customer_id", "target_key_col": "product",
      "timestamp_col": "event_date", "holdings_col": "held_products",
      "prior_holdings_col": "prior_products", "target_group_col": "product_category",
      "config": {"k": 3, "primary_metric": "ndcg"}
    }
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "Household features rank the next product above popularity."}
}

On a candidate table — one row per customer and offered product, with a bought flag (long grain):

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "candidate baseline", "type": "recommendation",
    "dataset_id": "<dataset_id>",
    "recommendation": {
      "grain": "long", "entity_col": "customer_id", "target_key_col": "product",
      "label_col": "bought", "holdings_col": "held_products", "target_group_col": "product_category"
    }
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "Per-product propensities beat popularity at K=10."}
}

The same event log as a ranking problem (LTR):

On the same acquisition event log:

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "cross-sell ranker", "type": "ltr",
    "dataset_id": "<dataset_id>",
    "ltr": {
      "grain": "event", "entity_col": "customer_id", "target_key_col": "product",
      "timestamp_col": "event_date", "holdings_col": "held_products",
      "target_group_col": "product_category",
      "config": {"split_strategy": "temporal", "k": 3}
    }
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "One global ranker beats popularity on the latest purchases."}
}

The floor first: the popularity ranking, reported beside the model's metrics (get_insight(kind="ranking"): baselines.global_pop). A model below popularity at the person's K recommends worse than the bestsellers: it is not a candidate, and that is the headline — first check the evaluation, available signal and policy match. Do not assume an excluded driver is the cause or restore a prohibited one. Popularity itself may be a useful reference or deliverable. An edge needs its interval: uncertainty.baselines.global_pop.difference["hit@K"] is the model-minus-popularity gap with a paired bootstrap interval over the holdout's customers (uncertainty.n_units); verdict: "undecided" means the holdout cannot tell the model from popularity — say so, with the interval, instead of calling +0.02 a win.

Iterate

Lever (hypothesis.lever)	Change
features	feature_cols, or a new entity feature in the recipe (as-of history: prior events, recency)
recipe	the history window, the candidate set
split	holdout_fraction, split_strategy (ltr)
metric	primary_metric, ranking_objective, k — the person's
other	negative sampling (neg_ratio, negative_strategy; ltr negatives_per_positive), min_positives — with the evidence in note (a validation–test ROC gap in recommendation_submodels)
decision_layer	the popularity discount and value weights: not a refit (Decide)

• Removing entity features needs a mechanism that predicts a gain: dropping age or household size "to shrink a validation–test gap" is none.
• Sub-models that score worse than chance on test (ROC below 0.5 in recommendation_submodels) across a family of products: investigate label direction, population shift, sampling, small evaluation pools and pipeline integrity before more tuning. Report a platform defect only with supporting evidence; below-chance estimates alone do not identify the cause.

Insights

Read get_insight(kind="kinds") for each insight's supported query_params. Class-scoped interpretation reads may accept class_key; do not copy that filter to run-wide diagnosis, trust or operating-point reads that do not advertise it. Recommendation runs do not support single-row what_if: use published lists and native batch scoring for the supported questions.

Interpretation checklist: choose the relevant available reads for the question and candidate:

1. get_insight(kind="recommendation") or get_insight(kind="ltr") — the run's headline; get_insight(kind="ranking") — Hit@K, MRR, NDCG@K against popularity, with the bootstrap interval of each metric and of the model-minus-popularity gap (uncertainty).
2. get_insight(kind="recommendation_submodels") and get_insight(kind="recommendation_classes") — per product: positives, validation and test ROC, the quality gate.
3. get_insight(kind="recommendation_drift", params={target_key}) for the top three products — how their buyers differ from the rest.
4. get_insight(kind="ale"), get_insight(kind="archetype_summary"), get_insight(kind="error_tree") — the explanation profile the explain stage computed.
5. LTR: get_rules(experiment_id) — ranking rules: a condition on the entity's features with a set of products as its head (target_products, or target_products_excluded for "any product except …"); quote metrics_holdout and holds_up. A rule that holds up is a targeting segment in plain words.
6. Recommendation: the published reasons — query_sql on the *__recommendations table (holding, reason_text, reason_<j>_*) for a few entities: are the drivers plausible, and do the held products follow the approved states?

Then the Gems table (get_playbook("iterate")).

Challenge

Check	Reads required
Leakage	the features against the moment: holdings and tenure measured before the event; no per-product attribute among the features (get_run(view="settings")); a ranking rule whose condition is a holdings column measured after the event
Validation design	the split against how lists are used: temporal for a model that ranks the future
Stability	get_insight(kind="recommendation_submodels"): products that failed their quality gate (below_chance_on_test: the sub-model ranks its held-out data backwards) or have few positives
Plausible drivers	get_insight(kind="recommendation_drift"), get_insight(kind="ale"), the published reasons (recommendation), the ranking rules (ltr)
Beats popularity	the model against popularity at the person's K — the gap's bootstrap interval, not the two point values (get_insight(kind="ranking"), uncertainty) — overall and per segment the person named

Decide

• The tilt — simulate_decision(kind="ranking_at", params={alpha, value_weights, value_alpha}): ranking metrics with a popularity discount (alpha, accuracy ↔ coverage) and per-product value weights, from the holdout — no refit.
• One product's cost — simulate_decision(kind="reco_business_value", params={target_key, cost_fp, cost_fn, gain_tp}) (recommendation) or kind="ltr_business_value" (ltr).
• The committed operating point is get_insight(kind="recommendation_operating_point") or get_insight(kind="ltr_operating_point"); the person commits a new one in the experiment's UI.

Release and operate

• A recommendation or LTR run is not registered: register_model refuses it. Its release is the committed operating point and the published lists (the run_all chain trains, explains, scores and publishes them — a failed score or publish stage means there is nothing to release). Record it with register_model(action="record_release", experiment_id, …); bind a committed tilt with apply {"kind": "ranking_operating_point"}.
• New entities are scored with score_batch(action="submit") (reasons_per_item, default 3, 0–5: how many drivers each recommended product carries).
• Results: get_playbook("schema-recommendation") — the ranked lists with each item's score; a recommendation run adds holding and the reasons (reason_<j>_feature, _value, _direction, _contribution, reason_text). Quote reason_text and the drivers; never invent a reason. An ltr run has none.
• Operate: a product's buyer drift (get_insight(kind="recommendation_drift")) and compare(kind="datasets") on the entity features.