Skip to content

Stateless domain encoders

Real-world tables are full of columns that are strings on the surface but structured underneath: an email address, a URL, an IP, a User-Agent, a phone number, a postal address. The stateless domain encoders parse each of these into typed features — categories, flags, counts, distances — while keeping the raw column (AutoGluon prunes it if it turns out near-unique). Card, IBAN, phone, email (with its local part), name, IP and device-info (with the device's IP) columns are kept as a keyed pseudonym instead of the raw value, so their count / target encodings still work while the model holds pseudonyms rather than the identifiers — under a secret key you supply at run time; see Identifier pseudonyms and their key.

They are all GLOBAL, ROW_LOCAL steps (see the FE overview): purely row-wise, no label, no fitted state — so they fit once and leak nothing. The categorical outputs they emit are additionally fold-local target-encoded into leak-free reputation twins.

In short

  • You switch each domain on by listing its columns in the matching role list on cfg.data (for example email_domain, ip_domain). Nothing is auto-detected.
  • Each encoder keeps the raw column and adds typed features: categories, flags, counts, distances. describe_domains() lists what each domain emits and what it needs.
  • Identifier columns (card, IBAN, phone, email, name, IP, device info) are replaced by a keyed pseudonym. Set ML4_PSEUDONYMISATION_KEY (32+ characters) before you plan, and keep the same key for every later fit and prediction.
  • Missing reference data (MaxMind, the names corpus, usaddress) degrades the affected features and the run says so; it never fails the run.
  • The match step turns the same field from different sources into _MATCH consistency flags, a strong fraud signal.

Wiring columns via DataConfig

Each domain is switched on by listing its columns in the matching role list on cfg.data. Nothing is auto-detected — you assign the domain.

from parleml_ml4 import Ml4Config

cfg = Ml4Config()
cfg.data.target_column   = "is_fraud"
cfg.data.email_domain    = ["contact_email"]
cfg.data.ip_domain       = ["signup_ip"]
cfg.data.useragent_domain = ["ua_string"]
cfg.data.phone_domain    = ["phone"]
cfg.data.address_domain  = ["billing_address"]
cfg.data.names_domains   = [["first_name", "last_name"]]   # a [first, last] pair
Role list Domain Example input Encoder class
email_domain Email jane.doe@gmail.com EmailEncode
url_domain URL / link https://shop.example.com/cart?id=7 URLEncode
ip_domain IP address 203.0.113.42 IPEncode
useragent_domain Raw User-Agent string Mozilla/5.0 (iPhone…) Safari/604.1 UserAgentEncode
device_info_domain Device JSON blob {"os":"iOS","model":"iPhone14"} DeviceInfoEncode
names_domains Person / entity names Jane Doe NamesEncode
address_domain Full postal address 513 Lynn St, Hurt, VA, 24563 AddressEncode
zip_domain ZIP / postal code 94103 ZipEncode
phone_domain Phone number +1 415-555-0132 PhoneEncode
creditcard_domain Credit card (PAN) 4111 1111 1111 1111 CreditCardEncode
iban_domain IBAN DE89 3704 0044 0532 0130 00 IBANEncode
vat_domain VAT number DE123456789 VATEncode
time_domain / base_time_columns Datetime 2026-03-22 TimeFeatures
mt_domain Multi-token tags action, comedy, drama (see Text)
wt_domain Weighted multi-token Netflix(2), McAfee(1) (NMF — see Text)
dirty_domain Dirty categorical Sr. Manager, Sr Manager (Gamma-Poisson — see Text)
embedding_domain Text embedding The product arrived late… (see Text)

Same domains, mapped in the app

These are the identical column domains surfaced in ParleML Studio — on a dataset's Columns tab (the default) and per-experiment under Column roles — and read by agents. All of them read one catalogue, served at GET /api/describe/domains: the label, a one-line description and an example, plus the engine facts from describe_domains(). The mapping you make in the UI and the DataConfig role lists here are one and the same. time and text are derived from the column type (datetime / plain text) rather than assigned.

Assign original dataset attributes to domains. The corresponding transformer then emits its complete supported feature set; unavailable enrichments degrade or are rejected naturally during feature engineering. The suffixes below describe generated outputs for inspection and explainability, not user-facing selection values.

What each domain makes — describe_domains()

parleml_ml4.config.describe.describe_domains() answers, per domain, the questions that decide which columns to declare and which to keep:

from parleml_ml4.config.describe import describe_domains

zip_row = next(r for r in describe_domains() if r["domain"] == "zip")
[f["name"] for f in zip_row["features"]]
# ['<col>_PREF', '<col>_PLACE', '<col>_STATE']
[(c["domain"], [f["name"] for f in c["features"]]) for c in zip_row["companions"]]
# [('zip', ['<zip_a>_<zip_b>_DIST', '<zip_a>_<zip_b>_MATCH', '<zip_a>_<zip_b>_PREF_MATCH']),
#  ('ip',  ['<zip>_<ip>_ZIP_DIST', '<zip>_<ip>_ZIP_MATCH', '<zip>_<ip>_ZIP_PREF_MATCH'])]
Key What it says
features What one column of the domain emits on its own, each with a description, the needs it depends on, the switch that turns it off or on (on_by_default), and on_regulated_runs (false: a regulated run never receives it).
companions Cross-features that exist only with one more declared column: a second column of the same domain (two ZIPs → their distance and match flags), or another domain (an e-mail with a names column → the local-part ↔ name match _FN_k / _LN_k / _N_k; a ZIP with an IP → ZIP-vs-IP distance; a city, state or street with an IP or address → consistency _MATCH flags).
settings The config keys that change what the domain emits (stateless.zip_region_features, stateless.names_demographic_proxies, stateless.country, …).
requirements The reference data and parsers its features need, as this installation has them now: maxmind (GeoLite2 City + ASN for IP geo), names_corpus (names-dataset, the stateless extra), us_address_parser (usaddress, US only), postal_data (the GeoNames countries present) and phone_metadata (phonenumbers). Each has available and a detail saying where it was found or why not.

<col> stands for the declared column, <a> / <b> for two columns of one domain; a pair feature joins its two names in alphabetical order. The catalogue is read off the encoders themselves (their output names, suffix constants and the planner's match buckets), so it cannot list a feature the engine does not make; availability is read from disk at each call.

Identifier pseudonyms and their key

For the card, IBAN, phone, email, name, IP and device-info domains the encoder first parses the raw value — every feature in the tables below is computed from it — and then replaces the raw column with its keyed pseudonym, HMAC-SHA256(key, value) truncated, as it does the email local part (_PREF) and the device blob's IP (_IPADDRESS). Equal values keep equal pseudonyms, so count and target encodings, entity cardinality and the cross-source _MATCH flags work unchanged.

What this protects, precisely. Everything fitted on these columns holds pseudonyms, never the values: the target encoders' maps, AutoGluon's category levels and the training data AutoGluon caches with the predictor are keyed by pseudonym. A pseudonym is private only to someone who does not hold the key — with it, anyone can hash every candidate value (all 2³² IPv4 addresses, a phone-number range, the ~10⁹ card numbers behind a known BIN) and read the pseudonyms back as identifiers. So the key is a runtime secret and never part of the model: the recipe (recipe.json in a bundle) records only its fingerprint, pseudonymisation_key_fingerprint — 16 hex characters that identify the key without revealing it — and no pickle, MLflow parameter, log line or error message carries the key. Guard it like any credential: whoever holds both the key and a model can recover the identifiers the model was trained on.

What stays readable: the features derived from the raw value (the card's scheme and length, the email domain, the IP's /24 prefix and geo fields, the phone's region and carrier), the 2–4-character name prefixes the email name-match learns from the training names, and the domains that are not identifiers (zip, address, URL, VAT, user-agent), which are not pseudonymised.

Supplying the key. Set ML4_PSEUDONYMISATION_KEY to a secret of at least 32 characters before you plan a recipe that declares any of these columns:

export ML4_PSEUDONYMISATION_KEY="$(python -c 'import secrets; print(secrets.token_hex(32))')"

Every later fit and prediction of that model needs the same key, so store it with your other secrets. ParleML Studio generates one per installation and sets it for its own process and workers. Code that builds a transformer itself can pass the key explicitly as FERuntime(pseudonymisation_key=...); the variable is read when none is passed. A regulated run drops the raw name columns rather than pseudonymising them, and needs no key for them.

When it is missing or wrong, ml4 raises parleml_ml4.contracts.pseudonymisation.PseudonymisationKeyError rather than predict on pseudonyms that match nothing the model learned:

  • planning without a key names the identifier columns and the variable to set;
  • fitting or predicting without it names the fingerprint of the key the model was fitted with — … set ML4_PSEUDONYMISATION_KEY to the key this model was fitted with (fingerprint 1da9df3cd757c63f);
  • a key with another fingerprint names both fingerprints;
  • a key shorter than 32 characters is refused, because the fingerprint is public and a short key could be guessed from it.

Serving such a model — a bundle, an MLflow pyfunc, the airgapped export — needs the same variable on the serving host; see Deployment.

The parsers

Each table shows the raw input and the feature suffixes appended to the column name (e.g. contact_email_DOMAIN). Category-typed outputs are the ones that get target-encoded downstream.

Email — EmailEncode

Group Suffixes
Categorical _DOMAIN, _SDOMAIN, _PREF
Flags _DOMAIN_DISPOSABLE, _DOMAIN_FREE, _DOMAIN_ROLE, _DOMAIN_TYPO, _DOMAIN_TLD_SUSPICIOUS, _HAS_PLUS, _HAS_DIG
Local-part stats _LOCAL_LEN, _LOCAL_DIGIT_RATIO, _LOCAL_ENTROPY
Name match (if name cols configured) _FN_k, _LN_k, _N_k for k ∈

Disposable/free/role lists come from disposable_email_domains; typo detection uses rapidfuzz (Levenshtein) against common providers. The name-match features compare the local part to configured first/last-name columns.

URL — URLEncode

Group Suffixes
Categorical _SUBDOMAIN, _DOMAIN, _TLD
Flags & counts _HAS_PORT, _HAS_QUERY, _IS_SHORTENER, _IS_IP_LITERAL, _QUERY_PARAM_COUNT, _URL_LEN, _URL_DIGIT_RATIO, _PATH_DEPTH

Domain/TLD splitting uses tldextract with an offline bundled Public Suffix List snapshot — no network access at fit or serve time.

IP — IPEncode

Group Suffixes
Geo / ASN (needs MaxMind) _COUNTRY_ISO, _STATE_NAME, _STATE_ABBREV, _CITY, _ZIP, _ZIP_PREF, _ORG, _NET, _ORG_TYPE, _NET_PLEN
Always-on (stdlib) _PREF, _IS_IPV6, _IS_PRIVATE, _IS_BOGON

The geo/ASN block reads MaxMind GeoLite2 .mmdb databases via geoip2 (a core dependency; the databases are bring-your-own). If the databases are absent, those ten features degrade — _ORG_TYPE to the constant "UNKNOWN", the rest to null — and the run emits a warning naming the affected columns; describe_domains() reports maxmind as unavailable and says which file is missing where. The four stdlib features (_PREF, _IS_IPV6, _IS_PRIVATE, _IS_BOGON — including CGNAT) always carry real signal. See Geography & reference data for the MAXMIND_DB_DIR setup. The raw address is pseudonymised on the spelling the cross-source ip match compares, so that flag is unaffected; the derived _PREF (the /24 prefix) and geo / ASN fields stay readable.

User-Agent — UserAgentEncode

Group Suffixes
Family / version (categorical) _BROWSER_FAMILY, _BROWSER_VERSION, _OS_FAMILY, _OS_VERSION, _DEVICE_FAMILY, _DEVICE_BRAND, _DEVICE_MODEL
Flags _IS_MOBILE, _IS_TABLET, _IS_TOUCH_CAPABLE, _IS_PC, _IS_BOT

Parsing uses the user_agents (ua-parser) library.

Device info — DeviceInfoEncode

Parses a structured device JSON blob. Direct fields carry suffixes _BROWSER, _MODEL, _OS, _MANUFACTURER, _OSVERSION, _NAME, _RESOLUTION, _IPADDRESS (the IP's keyed pseudonym, comparable with an ip_domain column's). Nested User-Agent and IP fields are re-parsed with the UA and IP parsers under _UA_… and _IPADDRESS_… infixes (so you get e.g. <col>_IPADDRESS_COUNTRY_ISO). Distinct from useragent_domain, which parses a raw UA string rather than a JSON object.

Names — NamesEncode

Group Suffixes
Corpus-backed _F_KNOWN, _L_KNOWN, _L_TOP_COUNTRY, _F_P_FEMALE, _F_RANK, _L_RANK
Dependency-free _LEN, _REPEATED_CHARS, _MIXED_SCRIPT

Corpus features use the names-dataset corpus (the stateless extra); when it is absent those six degrade to null with a warning while the three dependency-free features still emit. names_domains entries are [first, last] pairs (a single [col] entry is analysed as both). The stateless.country setting selects which country's rank _F_RANK / _L_RANK report.

_F_P_FEMALE and _L_TOP_COUNTRY proxy sex and ethnic origin, so they are planned only when stateless.names_demographic_proxies is on. That setting covers those two columns only. Name identity — the name columns themselves and their _F_RANK / _L_RANK popularity ranks — still reaches a stock-roster model with it off, and a model can learn a per-name rate from it, which proxies sex, ethnic origin and age much as the raw name does. Where that is unlawful, declare no name columns, or train a regulated run (roster.regulated): it refuses the setting and also keeps name identity out of the model — it skips the ranks and drops the raw name columns once the features are derived, so they are neither model inputs nor target-encoded.

Address — AddressEncode

Group Suffixes
CRF fields (categorical) _NUM, _STREET, _STTYPE, _OCCUP, _OCCUP_NUM, _CITY, _STATE
Flags _IS_POBOX, _IS_COMMERCIAL_HEURISTIC

US CRF parsing uses usaddress. When it is absent or country != "us", the CRF fields degrade to null — including _IS_COMMERCIAL_HEURISTIC, which is derived from the parse — and the run emits a warning naming them. The regex _IS_POBOX flag reads the raw string and still computes.

ZIP — ZipEncode

Group Suffixes
Per-column _PREF, _PLACE, _STATE
Per-pair <a>_<b>_DIST (haversine km), <a>_<b>_MATCH, <a>_<b>_PREF_MATCH

Geo lookups use pgeocode with bundled GeoNames data (PGEOCODE_DATA_DIR, no network). The planner also feeds the IP encoder's <ip>_ZIP column in, so raw-zip ↔ IP-zip distance and match features fall out here.

The per-column features say where one address is, and geography can stand in for race and ethnicity (redlining). stateless.zip_region_features (on by default) controls them: off, the planner withholds _PREF / _PLACE / _STATE and keeps the per-pair features, which say how two addresses relate — billing far from shipping is a fraud signal, not a statement about where someone lives. A regulated run (roster.regulated) never receives the per-column features, whatever the setting says.

Phone — PhoneEncode

_PHONE_VALID (flag) + _PHONE_REGION, _PHONE_LINE_TYPE, _PHONE_CARRIER (categorical), via the phonenumbers library. Line types include MOBILE, FIXED_LINE, FIXED_OR_MOBILE, TOLL_FREE, PREMIUM_RATE, VOIP, and more. The stateless.country value supplies the default calling region.

Calendar — TimeFeatures

time_domain columns emit calendar parts _YEAR, _MONTH, _DAY, _WDAY, _HOUR plus flags _WKEND, _HOLIDAY, _BUSINESS_HOUR, all toggleable on cfg.stateless. When you also declare base_time_columns (recency anchors), each other time column emits a <col>_DS_<anchor> whole-days-since feature — usually the strongest temporal signal. Holiday/business-hour flags use the optional holidays package and the stateless.country setting. For per-entity lags and rolling windows, see Temporal encoders.

When either endpoint of a days-since feature is unusable, the cell holds stateless.days_since_null_replacement (default 999), a real in-range number of days. measures.json lists every column that carries such a filler. If you need "unknown" separable, set a value outside your own range.

Details: the days-since \"unknown\" filler is an in-range value, not an out-of-range marker

When either endpoint of a <col>_DS_<anchor> feature is unusable — null, or a non-null cell that does not parse as a date — the cell is filled with stateless.days_since_null_replacement, default 999. That is a real number of days: on any history longer than about 2.7 years it collides exactly with a genuine "999 days before the anchor", and the other obvious choice, -1, collides with an event one day after it. Both sit inside the column's own range, so no model can separate "unknown" from that one real recency.

What it costs depends on the model. A booster can still isolate the bucket with two splits; an EBM shape function or a linear term reads "unknown" as a moderate recency; and a partial-dependence plot renders the value as an ordinary point on the axis.

The fact is published, so a consumer can annotate it. Anything downstream (MLI, AutoDoc, a reviewer reading the sidecar) needs it to tell "999 days ago" from "never". measures.json carries the map under feature_engineering.sentinel_encoded_features:

"sentinel_encoded_features": { "last_login_DS_asof": 999.0 }

It lists the value, not merely a flag, and only the columns that actually carry one (a time column with no anchor emits no _DS_ feature and appears nowhere). It is read off the recipe's frozen params, so a replayed recipe reports the sentinel that is really in the served column rather than whatever the current config says.

The filler is a knob rather than NaN, because the field is typed int end to end (the emitted column is int64) and because AutoGluon's LinearModel median-imputes numerics, which would move "unknown" to the centre of the distribution instead of the edge. If you need the two separable, set a value provably outside both signs of your own range — cfg.stateless.days_since_null_replacement = -10**6.

An integer date column is decoded with an epoch unit that ml4 guesses from the values and logs. The guess is frozen at fit. Convert integer dates upstream if you can.

Details: integer date columns — the epoch unit is a guess, and it is frozen at fit

Marking an integer column datetime on the Columns tab hands the encoders raw int64. pd.to_datetime would read those as nanoseconds, so an epoch-seconds column — the most common integer date encoding there is — would decode to 1970-01-01 for every row, and the whole calendar block would be dropped as constant. ml4 instead reads the unit from the median magnitude of the column's non-zero values (~1.7e9 → s, ~1.7e12 → ms, and so on), which one dirty sentinel cannot re-base, and logs the unit it chose — the alternative reading (YYYYMMDD integers, a bare year) is one the data cannot distinguish and you can.

That reading is a property of the column, so it is taken once, at fit. The always-on clean step keeps it on epoch_units_ and replays it on every later batch. Without the freeze the same value would decode differently depending on which rows were batched beside it (50_000_000_000 → 1971-08-02 next to a 1.7e12 row, NaT on its own), which matters because scoring runs in chunks. Freezing is what makes clean a fitted step whenever date_columns is declared: transform before fit_transform raises, and a column fitted as text that arrives numeric later is refused rather than guessed at.

TimeFeatures holds no fitted state, so it re-reads the unit per call — sound when it is handed a whole column (which is how the recipe runs it), but the reason to convert an integer date column upstream if you can.

Credit card / IBAN / VAT

Pure-Python validators — no third-party dependency.

Domain Suffixes
CreditCardEncode _CC_VALID (Luhn), _CC_SCHEME (VISA/MASTERCARD/AMEX…), _CC_LEN
IBANEncode _IBAN_VALID (mod-97), _IBAN_COUNTRY, _IBAN_LEN
VATEncode _VAT_VALID (structural), _VAT_COUNTRY, _VAT_LEN

Cross-domain match flags

The match step (MatchEncode, GLOBAL, row-local) is where the domain features become a fraud consistency check. It collects the same field from different sources into buckets and emits a <A>_<B>_MATCH 0/1 flag for every within- bucket pair:

  • city — city_domain + IP _CITY + address _CITY
  • state — state_domain + IP _STATE_ABBREV + address _STATE
  • street — street_domain + address _STREET
  • ip — ip_domain + device _IPADDRESS
  • address — two address_domain columns

The buckets are one table, parleml_ml4.fe.transformers.match.MATCH_BUCKETS, which both the planner and describe_domains() read. A pair's flag is 1 when both columns are present, non-null, and their normalised (trimmed, upper-cased) strings are equal; a missing column degrades to an all-zero flag rather than an error. So "the billing city disagrees with the IP-geolocated city" becomes a first-class model feature. (ZIP matching is produced inside ZipEncode instead, to avoid double-emitting the name.)

See also

Next: Geography & reference data · Temporal encoders