Geography & reference data¶

Several stateless domain encoders enrich raw columns with country, network, postal, and name intelligence. Most of the underlying reference data ships inside the package; one dataset — MaxMind GeoLite2 — is bring-your-own for licensing reasons. This page explains what each feature needs, how to provide it, and how the stateless.country setting ties it all together.

In short

IP geo and ASN features need the MaxMind GeoLite2 City and ASN databases, which you download yourself and point MAXMIND_DB_DIR at.
GeoNames postal data ships with ml4 and runs offline; the name corpus comes with the stateless extra.
Missing reference data never fails a run: the affected features degrade and the run warns. describe_domains() reports what this installation has.
stateless.country (default "us") sets the default locale for names, postal codes, addresses, phones and holidays.

At a glance¶

Data	Powers	Bundled?	You must provide it?
MaxMind GeoLite2 City + ASN `.mmdb`	IP features (`ip_domain`)	No — EULA forbids redistribution	Yes, to enable IP geo-enrichment
GeoNames postal extract (via `pgeocode`)	Postal features (`zip_domain`)	Yes (bundled, offline)	No — attribution only
Personal-name corpus (`names-dataset`)	Name features (`names_domains`)	with the `stateless` extra	No
Holiday calendars (`holidays`)	`<date>_HOLIDAY` flag	Optional: ships with the `forecasting` extra; without it the flag is all-False, with a warning	No

MaxMind GeoLite2 — bring your own (IP features)¶

IP enrichment (ip_domain) turns a raw address like 137.118.191.41 into country, region, city, postal, ASN / organisation, network type, and hosting-vs-ISP classification. Those lookups use the GeoLite2 City and ASN databases via the geoip2 reader.

Not bundled — and that's deliberate

MaxMind's GeoLite2 EULA does not permit redistributing their databases, so ml4 ships no .mmdb files (only an empty placeholder directory). Each deployment obtains its own free copy, which keeps you — not us — as the party who accepts MaxMind's terms.

1. Download the two databases¶

Create a free MaxMind account and download both GeoLite2-City.mmdb and GeoLite2-ASN.mmdb from https://www.maxmind.com/en/geolite2/signup (or automate refreshes with MaxMind's geoipupdate tool).

2. Point ml4 at them¶

Set MAXMIND_DB_DIR to a folder holding both files:

# Windows
setx MAXMIND_DB_DIR "C:\data\maxmind"

# Linux / macOS
export MAXMIND_DB_DIR=/data/maxmind

# Docker — mount a volume and set the variable
docker run -v /host/maxmind:/data/maxmind -e MAXMIND_DB_DIR=/data/maxmind ...

Alternatively, drop the two files into the bundled parleml_ml4/fe/transformers/data/maxmind/ directory.

The geoip2 reader is a core ml4 dependency, so the files are all it takes: the next process that parses an IP opens them (a running worker keeps the answer it read first; a new run starts a new one).

Is it on? Ask the catalogue¶

parleml_ml4.config.describe.describe_domains() (served by ParleML as GET /api/describe/domains) reports, per column domain, a requirements block read from disk at call time. The maxmind entry says available and where the files were found, or which file is missing from which folder. The same block covers the other reference data: the name corpus, the GeoNames postal countries present, the US address parser and the phone metadata. See Stateless encoders → what each domain makes.

If the databases are absent¶

Declaring ip_domain=[...] without the databases is non-fatal: the geo / ASN features degrade (_ORG_TYPE to the constant "UNKNOWN", the rest to null) and the run emits a warning naming the affected columns, while the four stdlib classification features still carry real signal — _PREF, _IS_IPV6, _IS_PRIVATE, _IS_BOGON (including CGNAT). Everything compared with the IP's geo goes empty too: the ZIP-vs-IP distance and match features and the city / state consistency flags against the IP. To skip IP features entirely, leave ip_domain=[].

Attribution & freshness

Using GeoLite2 binds you to MaxMind's EULA — including the display attribution ("includes GeoLite2 data created by MaxMind") and keeping the databases reasonably fresh.

GeoNames postal data — bundled (postal features)¶

Postal features (zip_domain) resolve place / state and compute the geographic distance between two postal columns, powered by pgeocode over the GeoNames postal corpus. The redistributable extract is bundled and used entirely offline — ml4 pins PGEOCODE_DATA_DIR at the bundled directory so pgeocode never downloads at runtime. Override the location only if you need to, with PGEOCODE_DATA_DIR.

Required attribution (CC BY 4.0)

GeoNames data is licensed under CC BY 4.0, which requires attribution even in a hosted product:

This product includes data from GeoNames (https://www.geonames.org/), licensed under CC BY 4.0.

Postal lookups for a country not in the bundled set fall back to a generic distribution (place / state come back empty; distances still work where coordinates exist).

Personal-name corpus — the `stateless` extra¶

Name features (names_domains) read known-name flags and popularity ranks — plus, only with stateless.names_demographic_proxies, likely country-of-origin and gender probability — from the openly-licensed names-dataset corpus (part of the stateless extra). It runs entirely on-device. When the corpus is absent, the six corpus-backed features degrade to null with a warning while the dependency-free features (_LEN, _REPEATED_CHARS, _MIXED_SCRIPT) still emit.

The `country` setting ties it together¶

Most of this reference data is country-relative. One knob — config.stateless.country (default "us") — selects the default locale and drives:

Name statistics — which country's popularity ranks / gender priors apply (_F_RANK / _L_RANK).
Postal lookups — the default country index for zip_domain.
Address parsing — the US CRF parser (usaddress) only activates for "us"; other countries degrade gracefully to a no-op (with the regex _IS_POBOX flag still computed).
Phone parsing — the default calling region for phone_domain.
Holiday calendar — which nation's public holidays drive <date>_HOLIDAY (requires the holidays package).

from parleml_ml4 import Ml4Config

cfg = Ml4Config()
cfg.stateless.country = "gb"        # switch the default locale to the UK

The field is open-ended: set any ISO country code. For a country without bundled locale data the encoders degrade gracefully rather than fail (generic name / postal distributions, address parser off).

IP-derived country is per-row, not global

stateless.country picks the default locale for name / postal / holiday / phone features. IP enrichment resolves the country per address from MaxMind, so a global dataset still gets correct per-row country / region / city from ip_domain regardless of this setting.