Stateful encoders¶

Stateful encoders learn something from the data: a per-category target mean, a set of bin edges, a group distribution, a mined interaction structure. The label-free ones fit once (GLOBAL). The target-aware ones re-fit inside each bag fold (FOLD-LOCAL), so a held-out row is never encoded by statistics that saw it. For the locus model that makes this safe, see the FE overview.

In short

This page covers the stateful categorical and numeric family: target encoders, the clustering encoder, the fold-local aggregates and tree-guided interactions. You configure them in the target_encoding and stateful_fe sections of Ml4Config.
By default every categorical gets a smoothed target mean (TE) and a frequency (CN). Training rows are encoded out-of-fold, so no row carries its own label.
On a multiclass target each target-dependent code emits one column per class. Budget for the width.
Feature selection also judges the columns the recipe builds, and records the losers as select steps in the recipe.
When a transformer cannot run as specified, the run emits a warning that names the columns and the cause.

Target encoders¶

Target encoders replace a category with a numeric summary of the target for that category. All variants share one step kind (target_encode) and run FOLD-LOCAL. Two things together make them leak-safe:

the locus: a per-fold re-fit protects each fold's validation rows;
an inner cross-validation: it protects the training rows (see below).

Where the chain runs

The fold-local chain runs on AutoGluon's per-model feature-generator channel. AutoGluon fits it on each model's training call, however the family marks that call, and replays it on every later frame before aligning features. Every family in the roster therefore runs the chain once per bag fold, and a scoring call can never refit it.

Configure the code list on target_encoding:

cfg.target_encoding.target_encoders = ["TE", "CN"]   # default

target_encoders applies to every categorical column, whatever its cardinality. A binary categorical's target mean is a 0/1 indicator. The default ["TE", "CN"] pairs a smoothed target mean with a normalised frequency. Every model family, linear included, consumes these numeric twins directly; there is no separate one-hot representation. Each extra code widens the feature space at roughly linear cost. An empty list disables classification target encoding.

Training rows are encoded out-of-fold¶

A target encoder fitted on a slice and applied to that same slice puts every row's own label inside its own value. AutoGluon's bagging protects each fold's validation rows, but the training rows, the ones the model learns from, would stay self-encoded. With a near-unique categorical (a customer id, an email, a transaction id) the encoded column is the target, and the booster trains on it.

So each fold runs a fixed four-way inner cross-validation: every training row's value comes from a fit that excluded it. This is a leak-safety invariant, not a template parameter.

On a fixture with one category per row, the self-encoded column scores AUC 1.000 against the target. Out-of-fold it scores chance, which is the truth for a column that carries no repeatable information.

Three things to know:

Serving is unaffected. transform uses the full-slice fit: at serve time there is no label to hold out, and every row is the better estimate. fit_transform and transform therefore return different values for the same rows, by design.
The inner split honours the run's isolation key. A grouped run splits inner folds by group, and a temporal run rolls forward: the same rule as the outer split. The key reaches the encoder beside the frame rather than in it, because the run drops it from the model matrix as structural rather than predictive. See The split columns stay in the frame.
It says so when it cannot. Too few rows, a class too rare to stratify, or a declared isolation column that did not reach the fold: each falls back to the plain fit and records a degradation naming the columns. An encoder that silently stopped holding out would look identical in every number ml4 reports.

CN (count encoding) is target-independent, so nothing is held out for it. The same inner split covers multi_token, and text_linear cross-fits under the same isolation rule. The fold-local aggregates (cv_cat_num, group_rank, entity_cardinality) summarise a feature rather than the target and need none of this.

Multiclass is encoded one-vs-rest¶

category_encoders has no multiclass mode. Handed a 3-class target, its TargetEncoder returns a single column holding the mean of the ordinal class index, which is meaningless for unordered classes. So on a multiclass target ml4 fits one inner encoder per class against the 0/1 indicator y == class_k and emits <col>_<CODE>_<slot>, the per-class rate:

binary      city  ->  city_TE            city_CN
3-class     city  ->  city_TE_0  city_TE_1  city_TE_2   city_CN

CN is not expanded: a category's frequency does not depend on the target, so per-class copies would be identical columns. Slot k is classes[k], pinned in the recipe at plan time. A bag fold that misses a rare class still emits its column (the indicator is all zeros and the encoder returns its prior), and every child agrees on what slot 2 means.

Details: slots are matched by rank, not by label value

The recipe pins classes from the raw target, but a fold-local fit receives AutoGluon's cleaned 0 … K-1 codes. LabelCleanerMulticlass maps the labels through enumerate over sorted categories before any model sees them. The two spaces are reconciled by rank: the planner sorts, AutoGluon sorts, so position k denotes the same class on both sides, and a fold missing a class does not shift any other slot.

The fold-local locus does not infer which space it is in; it declares it. AutoGluon always cleans a classification target before any model sees it, so the mixin states that fact to each step before handing it y. That settles the one case rank alone cannot: integer class labels combined with a fold that never saw the lowest class. There the observed codes are also members of the raw class list, so the raw reading would win and shift every slot by one. Inference remains only as a fallback for a direct call or a replay driven without AutoGluon.

This matters because comparing a raw label against a coded y does not fail loudly: numpy returns an all-False array. Without the reconciliation, a multiclass run with string labels would ship every per-class column as a constant, and one with {1, 2, 3} labels would attach each slot to the wrong class while still carrying plausible-looking signal. Labels in neither space raise an error rather than silently encoding to nothing.

Budget for the width

Each target-dependent code costs n_classes × as many columns as on a binary target. Ten categoricals, two codes and five classes is 100 engineered columns.

Code	Encoder	What it computes	Reach for it when
`TE`	Smoothed Target Encoder	Sigmoid-smoothed target mean per category (category_encoders)	The default; robust general-purpose mean encoder
`WE`	Weight of Evidence	Log-odds of the binary target per category, coarse-classed and count-shrunk (see below)	Binary only; log-odds features for linear models
`ME`	M-Estimate Encoder	Additively-smoothed target mean, pseudo-count `m = 20` (see below)	You prefer additive (m-estimate) smoothing to `TE`'s sigmoid
`JS`	James-Stein Encoder	Empirical-Bayes shrinkage toward the prior	Many low-count categories
`LO`	Leave-One-Out Encoder	Per-row leave-one-out target mean	An alternative leakage-safe mean
`CN`	Count Encoder	How often each value occurs, as a share of the fit slice (`normalize=True`) rather than a raw count	Raw popularity signal (target-agnostic)
`GM`	GLMM Encoder	Mixed-effects (GLMM) shrinkage estimate	Principled shrinkage on hierarchical categories

The estimator mathematics come directly from the category_encoders library, and so do the constructor defaults, with exactly two exceptions: ME's pseudo-count (m = 20) and CN's normalize=True (a units choice: a raw count scales with the dataset). ml4 contributes the fold-local, out-of-fold fit lifecycle and the multiclass one-vs-rest orchestration. It does not maintain a second target-encoding implementation or a parallel set of estimator knobs.

Details: why ME runs at m = 20

MEstimateEncoder estimates a category of fold-train size n as (Σy + prior·m) / (n + m). The category keeps self-weight n / (n + m), and its half-weight point is exactly n = m. category_encoders ships m = 1.0, which leaves a two-row category carrying a third of its own label: the weakest smoothing in the family, and the opposite of what "additive smoothing, stronger than TE" promises.

ml4 therefore runs ME at m = 20 (target_encoding.m_estimate_prior_count). 20 is TargetEncoder's own min_samples_leaf default, the category size at which TE's sigmoid crosses ½. The two encoders then shrink comparably, and the choice between them is about the shape of the shrinkage rather than an unstated 20x difference in its strength.

JS and GM are not tunable from here. Their shrinkage is a mode and a random-effect variance respectively, both estimated from the data, and they keep their library defaults.

Details: the shrinkage ml4 supplies for WE

WOEEncoder ships no shrinkage. Its only guards are a Laplace term inside the log and a hard woe[count == 1] = 0, so a three-row level would reach the model as a full-strength log-odds decided by three labels. Every sibling has something: TE sigmoid-blends, ME adds a pseudo-count, JS and GM estimate their own shrinkage, LO self-excludes, CN is label-free. So WE alone gets ml4's:

Rare-level pooling. Levels seen fewer than 20 times in the fit slice become one shared class, so their log-odds is estimated once over all of them.
Count shrinkage. What survives is multiplied by n / (n + 20). WOE's prior is exactly 0, so shrinking toward it is a plain multiplication: the log-odds twin of ME's m = 20, and the same "a category is worth its own estimate at ~20 rows" convention said a third way.
Monotone coarse-classing of numeric predictors, on the regulated_kind="scorecard" path only, where WE is also applied to the continuous columns. See Constraints → coarse-classing for the bands, the floors and what the monotone requirement costs.

All three are re-derived inside each inner out-of-fold fold, so the decisions are out of fold too, not only the arithmetic on top of them.

Planner guards for task type

The planner drops codes that would be meaningless for the task:

WE is skipped on regression. On multiclass it is fitted against each binary one-vs-rest indicator.
On multiclass, target-dependent codes are fitted one-vs-rest and emit one column per class. CN is target-agnostic and remains a single column.

Regression has its own switch and list: enable_regression_target_encoding (on by default) and regression_target_encoders (default ["TE", "CN"]).

Beyond the raw declared categoricals, the planner also target-encodes the domain-generated categoricals and raw domain columns (an email _DOMAIN, an exact reputation value…) into leak-free numeric twins.

Feature selection judges what the recipe builds¶

feature_selection runs twice, on two different populations:

Pass	When	Judges
Declared	plan time, before any FE	the raw `num_features` / `cat_features`
Engineered	plan time, after the recipe is built	every column the recipe produces

The second pass is what makes selection real. Without it, a 512-column embedding block, the clustering codes, the mined interactions and every target encoding would enter every model unjudged, however little they contributed.

It materialises the recipe once on the planning frame, ranks the whole pile with the same instrumental machinery, and records the losers as select steps in the recipe: one per locus, because the two loci remove columns at different times.

recipe.global_steps()[-1]              # RecipeStep(step_id='fs_engineered', kind='select', …)
recipe.fold_local_spec_for("GBM").steps[-1]
                                       # …and, when fold-local columns lost,
                                       # RecipeStep(step_id='fs_engineered_fold_local', …)
recipe.dropped_engineered_features     # the audit trail — the union of both drop lists

Details: why there are two drop steps, and the rules that keep the prune safe

A global drop list runs at the end of the global chain, strictly before any fold-local encoder exists. A fold-local output on that list would never be found, and the model would train on a column every surface reports as pruned. Fold-local losers therefore ride the second step, which every family's per-fold chain executes last, after the encoders have re-emitted their columns. One prune, recorded where it can actually happen.

Being steps is deliberate. The prune is fingerprinted with the recipe, visible when you print the plan, replayed identically on a rebuild, and the removed columns are named in one place rather than implied by their absence.

Two rules keep it safe:

Nothing a later step consumes is ever pruned. A mined <A>$<B> concat feeds a target encoder; dropping it on its own importance would break the chain rather than trim it. Columns another surviving step reads are protected whatever their rank.
A fold-local step whose outputs all lost is removed outright. A step with nothing left to emit is pure cost, so it is deleted rather than run-and-dropped. Its columns stay on the fold-local drop list all the same: the list is the authoritative record of what lost, and dropping a column that never comes to exist is a free no-op.

The acceptance guard is honest about its own rows too. The held-out slice it scores the prune on is carved before anything ranks, and the rankers see only the other side of that cut. A kept set chosen on every row and then scored on a fifth of those same rows would be tuned on the scoring labels.

The ranking is trustworthy only because the fold-local encoders hold their training rows out (above). A self-encoded target encoding would rank at its memorised value, and the prune would keep precisely the columns that deserve to lose.

Details: it is a probe, and it is bounded

The transformers fitted for the measurement are discarded; the real fit re-fits everything per fold. Anything that cannot be measured (a step that will not build, a ranking that raises) is skipped with a warning, and those columns go unjudged. A recipe that cannot be measured must still be trainable.

The probe reads at most 20,000 rows. Importance ranking converges long before a large frame is exhausted, and plan time must not grow with the dataset. That cap, not the ranker count, keeps the pass affordable: its cost is driven by the width of the pile.

Both rankers run whenever the pile holds categorical columns, and LightGBM alone runs when it is all-numeric. CatBoost is there for the categoricals. On a domain-heavy frame the engineered pile is ~30% categorical (the _DOMAIN / _TLD / _STATE / _ORG outputs), and most of those have no numeric twin. On a frame whose target is driven by email_DOMAIN, LightGBM alone prunes that column and CatBoost keeps it.

The saving comes from the boosting budget instead: the probe runs 100 rounds, not the declared pass's 600 (fs_rounds). It produces a ranking, and importance ordering stabilises long before the loss does. On the same frame, 100 rounds reaches the identical decision in a third of the time. A wrong ranking is not silent: the prune still has to pass the acceptance test, which scores kept-vs-full out-of-sample and reverts if it cost too much.

The over-correlation filter is numeric-only, by declaration¶

After the rank cut, overcorrelated_threshold drops the lower-ranked member of every |Pearson r| > 0.95 pair. Its candidate set is the columns you declared in num_features, not whatever pandas calls a number.

Details: why integer-coded categoricals are not candidates

An integer-coded categorical (store_id, age_band, an ICD chapter) is an int64 column. A dtype test would sweep it into the matrix, and Pearson would then answer the wrong question: whether the level codes move together, rather than whether the two level partitions are redundant. The two agree only by accident of how the levels were numbered. Both directions go wrong:

store_id and region_id numbered region-major correlate at |r| ≈ 1 although one is a coarsening of the other, and the coarse column (the one whose target encoder has support per level) is the one that loses the tie;
two string-coded categoricals that genuinely are the same partition are never compared at all, because strings are not numbers.

So a dtype test would make a nominal column's candidacy depend on how its levels happen to be serialised. Losing one is not cheap: it takes the column's target-encoded twin, its cv_cat_num / group_rank / entity_cardinality aggregates and its $ concat interactions with it.

Categorical redundancy is not measured

Detecting that two categoricals are duplicates needs Cramér's V (cat × cat) and η² (cat × num), which this filter does not implement. A declared categorical is therefore never dropped by the correlation filter. That under-prunes, which the rank cut and the acceptance probe both bound, rather than deleting a column on a meaningless statistic.

Clustering encoder¶

clustering (GLOBAL, unsupervised) discretises each numeric into a nominal bucket plus a distance-to-centroid feature. It is a cheap non-linearity that lets tree and linear models split on regions of a numeric range.

cfg.stateful_fe.add_clustering_encoding = True     # on by default
cfg.stateful_fe.clustering_max_k = 15
cfg.stateful_fe.clustering_binning_method = "consensus"   # or "kmeans"

consensus (default) agrees on bins across multiple methods (more stable; Sturges-style bin count).
kmeans runs 1-D k-means, sweeping k up to clustering_max_k and keeping the k with the best silhouette. Faster but more outlier-sensitive.

Fold-local aggregates¶

These twins summarise a numeric by a category, or an entity's fan-out. They are label-free (UNSUPERVISED_FITTED) but placed FOLD-LOCAL, so that a tiny group's aggregate re-fits per fold (cross-fitting) and a singleton group cannot memorise its own row. All three are opt-in (off by default).

Each carries a *_max_candidates budget, and on any real table there are more (cat, num) candidates than budget (5 categoricals × 30 numerics is 150 pairs against a default of 30). The pairs that fit are chosen by the TGI miner's own pair ranking: the same tree co-occurrence score that drives interaction mining, reused rather than recomputed. The budget goes to the pairs a tree actually split on. None of the three requires add_tree_guided_interactions; with mining off there is no ranking, and the candidates fall back to a name order that is at least invariant to how the columns were declared.

cv_cat_num — category → numeric aggregategroup_rank — within-group percentileentity_cardinality — fan-out count

<NUM>_<AGG>_PER_<CAT> with empirical-Bayes shrinkage toward the global aggregate.

cfg.stateful_fe.add_cv_cat_num = True
cfg.stateful_fe.cv_cat_num_aggregations = ["MEDIAN", "STD"]  # default
cfg.stateful_fe.cv_cat_num_alpha = 10           # EB shrinkage
cfg.stateful_fe.cv_cat_num_min_group_count = 20
cfg.stateful_fe.cv_cat_num_max_candidates = 60  # caps n_cat × n_num × n_agg

Aggregations: MEAN, MEDIAN, STD, MIN, MAX, SUM, NUNIQUE.

Pairs are a capped cartesian product over the feature-selection survivors, ordered by the mined pair ranking when it exists. The encoder mines nothing itself and works with add_tree_guided_interactions off.

<NUM>_RANK_PER_<CAT>: each row's percentile rank of the numeric within its category, in (0, 1]. It reuses the (cat, num) pairing rule, not cv_cat_num's output, so it needs neither that encoder nor tree-guided interactions switched on.

cfg.stateful_fe.add_group_rank = True
cfg.stateful_fe.group_rank_grid_size = 101      # ~1% resolution
cfg.stateful_fe.group_rank_min_group_count = 20
cfg.stateful_fe.group_rank_max_candidates = 30

<VALUE>_NUNIQUE_PER_<KEY>: distinct values of one column per entity key (distinct emails per IP, distinct cards per device). A classic fraud signal. Pairs are entity-scoped (data.key1 / key2 × their linked columns), not cartesian, so the count stays bounded.

cfg.stateful_fe.add_entity_cardinality = True
cfg.stateful_fe.entity_cardinality_max_candidates = 30
cfg.stateful_fe.entity_cardinality_min_group_count = 1

Tree-guided interactions¶

Rather than materialise every pairwise interaction, ml4 mines promising pairs with a small plan-time instrumental LightGBM, then materialises the frozen structure as GLOBAL, row-local steps. On by default:

cfg.stateful_fe.add_tree_guided_interactions = True

The operand pool is the raw declared columns plus the label-free stateless outputs: a domain encoder's _DOMAIN / _STATE, a clustering code, a time part. Those are exactly the categoricals a joint interaction is worth mining on. The mining frame is materialised without the target, so nothing the miner sees was itself built from y.

Target encodings are deliberately not operands. TE(city) is already a target summary, and crossing it with another categorical comes close to re-deriving TE(city$state). The mining → concat → target-encode handoff below already produces that, without stacking two label-derived transforms.

Three families come out of mining:

num×num arithmetic (interactions): operators PL (add), MI (subtract), DI (divide), MU (multiply). Divide is guarded against extreme scale ratios (tgi_scale_ratio_mi_threshold) and near-zero denominators (tgi_di_zero_fraction_threshold).
cat×cat / num-cat concat (interactions.concat): string concatenation of categorical levels, and quantile-binned numeric × categorical combinations. The minted <A>$<B> joint categories are target-encoded like any other categorical. They are the highest-cardinality ones in the plan, so a CV target statistic is exactly what they need. With several configured target_encoders the raw joint column is kept alongside its twins; with one, the twin takes its place.
standalone quantile bin (interactions.bin): a <NUM>_BIN bucket.

Mining is tuned by tgi_* knobs (tgi_mining_depth, tgi_top_k_pairs, tgi_min_pair_score, tgi_max_total_candidates…). An optional importance prune (tgi_importance_prune, on by default) materialises the mined columns, fits one cheap balanced LightGBM over [base + interactions], and thins by the booster's actual per-column gain. It measures the real column, not the mining pair proxy. For num×num it keeps the top tgi_max_variants_per_pair operator variants per pair (default 1, which removes collinear copies) and caps the total at tgi_max_final_survivors.

Details: the num_cat_cl route is measured too

Its left operand is the <NUM>_CL cluster twin, which the clustering step only materialises at fit time, so the column is absent from the plan frame. With clustering on (the default) that route covers every mined num×cat pair. It is also the route most likely to duplicate a split the booster already gets from the raw numeric, because its left operand is a coarse binning. So the prune builds the twin for the candidate numerics and leaves it on the frame: the concat has to earn gain over and above <NUM>_CL and <CAT> standing alone.

Cost: clustering_binning_method="consensus" (the default) is fitted exactly on the whole prune frame and adds ~0.3 s at 5,000 rows and ~3 s at 50,000 rows for six candidate numerics. The opt-in "kmeans" method runs a kmeans1d + silhouette sweep whose cost is superlinear in rows (30.5 s for one column at 50,000 rows). On that path the centroids are fitted on a deterministic 5,000-row subsample and then applied to every row. That is a scoring proxy inside an instrumental pass, never the shipped column: the GLOBAL clustering step still fits exactly on the full training frame.

Concat cardinality guard¶

A joint category is only useful while it groups rows. Two 200-level columns crossed over 100,000 rows produce ~37,000 combinations. Nothing downstream removes them (AutoGluon's CategoryFeatureGenerator keeps every level seen twice), so the column arrives at the model as an identifier the booster memorises. The guard runs at mining time, so a pair over budget is never minted:

cfg.stateful_fe.tgi_concat_max_levels = 500          # absolute ceiling
cfg.stateful_fe.tgi_concat_min_rows_per_level = 20   # ≥ 20 rows behind each level

The effective budget is the tighter of the two, min(tgi_concat_max_levels, n_rows // tgi_concat_min_rows_per_level). The ratio shrinks the cap on small data, and the ceiling stops a very large table authorising a 500,000-level column.

Details: how levels are counted

Levels are observed, not assumed, where both operands are real columns (city$state is nowhere near |city| × |state|). For the <NUM>_BIN and <NUM>_CL routes, whose left operand only exists at fit time, the bound is the product against tgi_bin_fallback_q / clustering_max_k. Rejections are logged with the pair and its level count.

The feedback round (round 2)¶

Round 1 ranks operands by their raw importance, because that is all it has. It cannot see a pair whose value only appears after feature engineering: a joint category that becomes predictive once target-encoded, or a column that only matters crossed with a cluster code.

The feedback round closes the loop. After feature selection has ranked the engineered columns, each column's standing is attributed back to the raw operands that produced it, and the miner runs again over the operands that earned it:

cfg.stateful_fe.add_tgi_feedback_round = True     # off by default; on in `high_accuracy`
cfg.stateful_fe.tgi_feedback_top_operand_fraction = 0.5
cfg.stateful_fe.tgi_round2_max_candidates = 40

Details: attribution and bounds

Attribution follows the recipe's provenance map transitively to the roots: city$state_TE credits city and state, not the intermediate city$state. Mining an intermediate would build a join of a join: a three-way joint category whose cardinality is the product of three columns, a different and much riskier feature than the pair asked for.

Two things bound it:

Round 2 is not judged by feature selection. It exists to use that ranking, so it cannot also precede it. tgi_round2_max_candidates caps what it may mint.
Its candidates face round 1's importance prune, the same materialised marginal-contribution check. On a 16-numeric frame that cut 40 mined candidates to 6: the columns the booster actually splits on.

It is often a no-op, and that is correct

The round only pays when round 1's tgi_top_k_pairs cap was binding: many operands, so good pairs were crowded out. When round 1 already found everything, round 2 logs that it found nothing new and adds no columns. It also cannot rescue a pair the cardinality gate rejected; that gate applies to both rounds.

When a transformer degrades¶

Several transformers are deliberately resilient: a missing GeoIP database, an unreachable embedding backend, an absent usaddress, a degenerate Gamma-Poisson vocabulary. All of them emit their columns anyway, as inert ones, rather than failing the run. A missing optional dependency should not cost you a model. But a run fitted without its embeddings emits a block of zero columns, which AutoGluon's pipeline drops as constant, so the model is fitted without them. It scores worse than it should, and the blame would land on the model rather than on the environment.

So ml4 collects these admissions off the fitted chains and emits them once, after the fit, as a run warning naming the affected columns and each cause:

14 engineered column(s) shipped degraded — the pipeline produced them, but not as
specified. Read the cause: a step that could not run at all leaves an inert column, which
AutoGluon drops as constant, so the model was fitted WITHOUT it; a step that ran in a
reduced mode leaves the column in place but worth less than its name implies. Either way,
a disappointing score here may be the run — the environment, or a step that could not work
as specified — rather than the data.

Causes:
  - EmbeddingEncode: embedding backend unavailable (connection refused)
  - IPEncode: IP geo/ASN unavailable (MaxMind GeoLite2 databases not found at /data/maxmind …)
  - TargetEncode: target encoding is not out-of-fold (no 4-fold inner split exists
    for this slice (18 rows))

Columns: c_TE, ip_CITY, ip_COUNTRY_ISO, …, notes_EMB_0, …

Both loci are covered. GLOBAL steps are fitted once, in-process, and reachable from the persisted predictor. Fold-local steps are re-fitted per bag fold inside AutoGluon's model fits, possibly in worker processes. The fitted chain is state that inference replays, so it is persisted with each leaf model and read back off the strongest model's leaves. One entry per bag fold is expected and de-duplicated.

That second locus matters: the out-of-fold target encoders live there. "This encoding is not out-of-fold" is the only signal that an encoder fell back to self-encoding, a downgrade no metric ml4 reports can distinguish from an encoder that is still holding out.

A fold-local text column that lost its text¶

multi_token and text_linear are the two fold-local encoders that need the raw strings, and they run inside the model fit, after AutoGluon's GLOBAL pipeline. Stage 2 of that pipeline is AutoGluon's stock CategoryFeatureGenerator. Its defaults null every cell value seen once and recode the surviving categories as integers. A mt_domain / text_domain column that reaches it arrives at the encoder as level codes, and the original strings are gone from the frame.

Details: what happens to the column, by level count

The outcome depends on how many distinct values the column has:

Levels	Dtype at the encoder	Effect without the ledger entry
many	`category` of ints	`fillna("")` raises inside pandas; AutoGluon swallows it per model, so the run says only "No models were trained successfully during fit()"
two	plain `int8`	nothing visible: `"0"` / `"1"` tokenise as single tokens, and the encoder silently becomes a per-cell target encoder with `mean` == `max`

Both cases are recorded on the degradation ledger above, so the run names the column and the cause. The fit is not failed: a per-cell target encoding is redundant rather than wrong, the inner cross-fit still applies, and no reported number is inflated. Refusing would cost a model over a feature-quality problem whose real fix is upstream. What the ledger prevents is the column quietly meaning something other than its name.

A category column whose levels are still strings is fine, tokenises normally, and records nothing. An all-null column likewise: no values means no strings were destroyed.

Text columns can still be encoded per cell

ml4 does not yet keep the fold-local text and multi-token step inputs away from that stage-2 generator (the fix belongs in parleml_ml4.engine.feature_generator). A declared mt_domain / text_domain column that reaches the encoder as level codes is encoded per cell rather than per token.