Learning-to-Rank¶

LTRRanker trains a single global ranking booster over decision-occasion groups and projects it onto a calibrated-shaped (n, K) matrix over the shared ranking surface — the same ranking.json contract as the recommendation backend, but from one model instead of many.

from parleml_ml4.ltr import LTRRanker, LTRConfig

cfg = LTRConfig(grain="event", entity_col="customer_id", target_key_col="product")
pipeline = LTRRanker(cfg).fit(df)          # returns a deployable pipeline
result = pipeline.result

In short

LTRRanker trains one global ranking booster over decision-occasion groups and emits the same ranking.json contract as the recommendation backend.
ranker_objective picks the training loss and library (lambdarank by default); ranking_objective picks how the holdout is reported. They are independent.
split_strategy="auto" grades the latest occasions when timestamp_col is set, else whole entities. Avoid random when occasions are finer than a customer.
Optuna tunes on full-catalogue NDCG@k (n_trials=20, k=10 by default).
Serve frames with the dtypes the model trained on; score_matrix refuses a mismatch.

Boosters in core; search dependency in the extra

LightGBM and CatBoost already ship in core. The lambdarank / rank_xendcg objectives use LightGBM; yetirank / yetirank_pairwise use CatBoost. The ltr extra adds Optuna for local hyperparameter search. Only the backend selected by ranker_objective is imported.

Decision-occasion groups¶

Learning-to-rank scores one ranked list per decision occasion. A "group" is one such occasion; the candidate product is a categorical feature of each row. Group resolution follows a priority order:

bundle_col set → group by (entity_col, bundle_col);
else timestamp_col set → (entity_col, floor(timestamp, bundle_time_granularity));
else → (entity_col).

In event grain every observed row is a positive and negatives are sampled from the product catalogue per occasion (negatives_per_positive, default 9, bounded by min_neg / max_neg). negative_sampling="popularity" (the default) makes popular items hard negatives; uniform draws evenly. In long grain you supply a binary label_col and pre-materialised candidates.

The backends: two libraries, four objectives¶

A single ranker_objective knob picks both the ranking loss and the gradient-boosting library that owns it:

`ranker_objective`	Library	Loss
`lambdarank` (default)	LightGBM	LambdaMART — classic pairwise NDCG optimiser (fast).
`rank_xendcg`	LightGBM	Listwise XE-NDCG cross-entropy surrogate.
`yetirank`	CatBoost	YetiRank — pair weights from document positions.
`yetirank_pairwise`	CatBoost	YetiRankPairwise — more accurate, slower.

The lambdarank-only knobs (lambdarank_truncation_level, lambdarank_norm, sigmoid) apply only to lambdarank; yetirank_mode applies only to the CatBoost objectives.

Two similarly named knobs, opposite jobs

ranker_objective is the training loss (how the booster fits). ranking_objective (default accuracy) is the evaluation metric (how the holdout is scored and ranked in the report). They are independent: one is "how to fit", the other "how to report".

How the holdout is carved¶

split_strategy (default auto) decides which decision occasions are graded, and it changes what every reported score means:

value	holdout	use when
`auto`	`temporal` if `timestamp_col` is set, else `entity`	the default; always one of the two honest options
`temporal`	the latest occasions, ordered by each occasion's last timestamp	"what will this customer take next?": fit on the past, grade the future, exactly as deployment runs
`entity`	whole entities	cold start: "can we rank for a customer we have never seen?"
`random`	a random sample of occasions	reproducing an older run only

random is optimistic whenever an occasion is finer than a customer (i.e. whenever bundle_col or timestamp_col is set): the same customer's other baskets sit in training while their held-out basket is graded, with no time ordering to justify it. Entity features are taken from the occasion's first row and so are near-constant across that customer's occasions, which lets a high-cardinality feature act as a customer fingerprint. temporal deliberately allows the same customer on both sides: in production their history is known, and what must not leak is the future.

Occasions with no usable timestamp sort oldest and stay on the train side; a null date is missing information, not evidence of recency. temporal without a timestamp_col is rejected at config construction. The resolved strategy (never auto) is recorded on LTRResult.split_strategy and in the persisted manifest.json, so an artefact always says how it was graded.

The hyperparameter search splits its inner validation set the same way, so best_valid_ndcg is graded under the same protocol as the holdout. A random inner split under split_strategy: temporal would select hyperparameters under the protocol this table calls "optimistic" and report a number that only looks temporal.

Long grain rejects per-candidate features

A feature that varies between an occasion's candidate rows (a per-product price, a per-candidate offer flag) is refused at fit. Training keeps each candidate's own row, but evaluation and serving score one representative row tiled across the whole catalogue, so such a column would be scored with another product's value for every candidate but one, and no reported metric would show it. Aggregate it to the entity/occasion level, or drop it. (Event grain does not refuse them, because its row-0 reduction is applied identically on both sides. There each row is a purchase, so such a column describes the item bought. The default feature set leaves out the columns that vary within an occasion or that the product determines (entity attributes excepted), detected on the training split and named in a warning; an explicit feature_cols overrides this.)

fit checks the training split. The ParleML launch runs the same rule earlier (key_varying_features over resolve_group_keys) on the whole dataset and the feature set the run would use, and refuses the launch with the columns named, so there it also catches a column that varies only inside held-out occasions.

NDCG@k tuning¶

Fitting runs five stages: prepare (catalogue + group split + training pairs) → optuna (group-aware NDCG@k tuning) → fit_final (refit on the full train split) → evaluate (holdout ranking report) → persist (deployable pipeline).

Optuna maximises full-catalogue NDCG@k computed by ml4's own ranking metrics over the validation lists — not LightGBM's internal sampled-group NDCG, which reports a spurious 1.0 for a degenerate constant-score booster. The search budget is n_trials (default 20) with max_depth / max_iterations bounds; the ltr_fast / ltr_balanced / ltr_thorough factories set the tier.

Details: both backends can reach their own defaults

A tuned backend that cannot sample its library's default configuration turns a leaderboard into a comparison of search spaces rather than of models. CatBoost's random_strength is therefore sampled log-uniformly. Linear sampling over [1e-9, 50] would give the default 1.0 a ~2% draw probability and put the median trial at 25× it. border_count is drawn from a ladder that includes CatBoost's own 254, and leaf_estimation_iterations reaches down to its default of 1. The ranges are shared with the supervised CatBoost space — see hyperparameter search.

The one default deliberately not reachable is min_data_in_leaf (CatBoost's is 1): a single-row leaf memorises individual training rows, so the floor is 5. That is a modelling policy rather than a handicap, because the reason is symmetric: LightGBM's own default of 20 already satisfies it.

LightGBM additionally tunes min_data_per_group and max_cat_threshold here. In the event grain the candidate-product column is the only feature that varies inside a group, and it is what the ranking problem is about. LightGBM's stock min_data_per_group=100 excludes every product with fewer than 100 training rows from being a categorical split candidate at all, while CatBoost's target statistics have no such floor. Left at stock, the comparison would partly measure categorical handling.

Each trial trains the shape that ships: its own tuned boosting-round count, start to finish. Nothing truncates a trial's fit, so the search ranks configurations by the model you would deploy rather than by a stub of it. Budget for that: on one fixture at 3,000 rounds a full trial takes 1.36 s against 0.25 s (5.4×) for an early-stopped one. max_iterations (caps the round count; up to 3000 when left None) and n_trials are the two levers that bound the search. There is no early-stopping knob, by design.

max_iterations also caps the refit that runs when there was no search to draw from: too few groups to tune, no usable inner validation split, or every trial failed. A smoke cap therefore keeps capping in exactly the situations that have already gone wrong.

from parleml_ml4.ltr.config import ltr_thorough

cfg = ltr_thorough(grain="event", entity_col="customer_id",
                   target_key_col="product", ranker_objective="lambdarank", k=10)
pipeline = LTRRanker(cfg, outputs_dir="runs/ltr").fit(df)

Metrics¶

Evaluation reports through the shared surface: Hit@1 / Hit@5 / Hit@K, MRR@K, NDCG@K, plus coverage and value metrics. primary_metric (default ndcg) with k (default 10) forms the primary score key ndcg@10. Set eval_all_objectives=True to score every objective in one report.

result = pipeline.result
print(result.ranking)                # shared ranking.json payload
print(result.best_valid_ndcg)        # best trial's held-out full-catalogue NDCG@k
                                     # (0.0 when tuning was skipped / every trial failed)
print(result.feature_importance)     # per-feature gains from the booster
print(result.backend, result.objective, result.model_label)

result.ranking["uncertainty"] resamples entities — every occasion of one customer, and every product of one basket, moves together — for an interval on each micro metric and on the gap to the popularity floor (How sure is the lift?).

LTRRanker.fit(df, feature_cols=None) derives feature columns automatically from every non-reserved column (entity / product / label / timestamp / bundle / holdings / group columns are reserved) unless you pass them explicitly.

Details: which columns the default feature set leaves out

On the event grain the default also excludes item attributes, detected on the training split: columns that vary within a decision occasion, or that the product determines without being constant per entity (an entity attribute that segments the catalogue is kept). A warning names them. An explicit feature_cols overrides the exclusion and is used as given. The run still warns when the list names a column the detector would have excluded, since a value known only once the item is chosen inflates the holdout scores.

On the long grain every feature, listed or derived, must take one value per decision occasion: the fit refuses one that differs between an occasion's candidate rows.

Serving¶

fit returns a FittedLTRPipeline directly and, when outputs_dir= is set, persists it under <outputs_dir>/deploy/ltr/ alongside events.ndjson / manifest.json.

Serve the frame with the dtypes it was trained on

The artefact records feature_dtypes, the dtype family (numeric / boolean / categorical / datetime) each feature trained in, and score_matrix refuses a frame that disagrees, naming the columns.

That guard is not pedantry. A feature trained as object strings and served as int64 does not merely misrank: every row collapses to one identical score, because the categorical is absent from the model's view. You get a well-formed matrix returning the popularity ordering for every entity, presented as a personalised ranking, with nothing in ranking.json to show it, because the holdout report is scored through the same path. It is reachable from an ordinary parquet-vs-CSV loader difference, so it is refused rather than coerced: which dtype you meant is a question only you can answer.

Families, not exact dtypes: int32 vs int64 round-trips freely. bool is kept distinct from numeric, because a booster treats it as a two-level category.

See the LTRRanker reference for the full API.

Next: Ranking · Learning-to-rank example