Playbook — supervised: binary, multiclass, regression¶

For experiments of type: "supervised": one row per entity at the prediction moment, a target column, and a model that predicts it. The task is declared, never inferred: config.ml_task is binary_classification (the default), multiclass_classification or regression, and a plan that declares the wrong one is refused. The delivery lifecycle is get_playbook("deliver"); this playbook is what each of its steps does for a supervised problem. Free text that carries the signal adds get_playbook("type-text").

ParleML is more than a model search: it parses business domains into features, encodes text several ways, engineers interactions, and explains every run with rules that hold up on Test, archetypes, error trees, ALE, surrogates and per-row cases. Each step below names useful evidence; choose the available reads that answer the question, with their split, class and model provenance.

Frame¶

Target — the column or SQL expression of the outcome, its classes or range, and:
binary: the positive class (the outcome the decision acts on, config.positive_class) and its base rate;
multiclass: every class with its share, the rarest one named;
regression: the range, the skew and the unit (a heavy right tail makes the error metric a real choice).
Population and prediction moment — get_playbook("deliver"), Point-in-time.
Metric — config.eval_metric, from describe_platform("metrics", task=…):
binary: average_precision for a rare positive, auc when the ranking is what is used, logloss when probabilities themselves are used;
multiclass: logloss, or f1_macro when every class matters equally however rare;
regression: rmse, mae when large errors should not dominate, mape for relative error on a target never near zero.
Costs — binary: what a false positive and a false negative cost, and how many cases can be acted on (a review capacity); multiclass: the cost of each confusion that matters, written by class label ("At Risk predicted as Loyal costs 5"); regression: the cost of an error by size and direction, and the interval width a decision can live with.
Template — the requirement choice (ml-frame step 6), with its modifiers: config.roster.include_foundation_challenger: true (the foundation model as one more member, on any non-regulated template — a run that asks for it fails when it cannot train, so read the plan's foundation-memory warning before launch); column_roles.monotone_constraints (a pin per driver whose business direction the person states, regulated_monotonic); config.roster.regulated_kind (glm, ebm or rulefit under regulated_glassbox).
Identity and location columns — every column whose values fit a domain of describe_platform("domains") (an e-mail, a ZIP, an IP, an address, a name …), and every pair of sources for one entity (billing and shipping ZIP, an IP and a billing address, a customer state and an incident state): ParleML derives distance and match features from such pairs (Data, Business domains).
Events of an entity — when rows are events and the history matters, propose the per-entity lags: column_roles.temporal_event_time_column, temporal_entity_columns, temporal_lag_features.

Data¶

Recipe — one row per entity at the prediction moment, the target as target_column (author_recipe); a per-row moment is the as_of_column.
Roles — prepare_data(action="columns", params={target_column, id_column, columns}): the row key, ignored columns (identifiers, fields written after the outcome), a base-time column (the prediction moment itself, never a feature), each string column's domain (below) and a one-line description for every column you understood.
Fairness — mark is_fairness: true on every protected attribute or proxy: gender, age, race or ethnicity, religion, marital status, nationality, disability, a ZIP or region standing for a neighbourhood. is_fairness enables an audit; it does not itself require exclusion. Set is_ignored: true only for an actual use restriction or a stated scientific reason, such as leakage or unavailable production inputs. Resolve genuinely unknown use restrictions with the person; do not invent one from a column's name. With no fairness column, get_insight(kind="fairness") has nothing to measure. A fairness column groups people: a name, an e-mail or a street address is one group per person — never mark it; the attribute it may stand for is the fairness question (ask whether the data has it). A ZIP with thousands of values is measured on its groups of 30 rows or more; prefer a coarser region column when the data has one.
Leakage — prepare_data(action="leakage", params={target}), then triage every finding.
Signal — probe_dataset(kind="signal", params={target, task}) and probe_dataset(kind="baseline_floor", params={target, task, metric, positive_class}). The signal probe runs no domain or text encoder and ranks drivers from one shuffle of a few hundred rows: its caveats name the domain columns it read as plain values — their signal shows only in a trained run.

Business domains¶

Map every string column, not only text: get_eda(section="columns") value checks, the name and the samples (get_playbook("deliver"), Column domains). Then read describe_platform("domains") and, for every column you map, quote what it makes, what its companion adds and whether this server has the data:

Declare the companions. A cross-feature needs both columns declared — a name column beside an e-mail, both ZIPs of a billing/shipping pair, a ZIP and an IP: keep useful, permitted inputs declared and avoid hand-building what the pair derives. Prediction-time availability and privacy constraints take priority over extra features.
Say what is unavailable. A feature whose reference data this server lacks (IP geo without the GeoLite2 files) ships empty: say so at the data gate, and expect no such driver.
Separate proxies from relations. Region and name features can stand in for protected attributes; the pair features (distance, match flags) say how two values relate. The domain entry names the switch for each (a ZIP's region features, the name-derived guesses) and what a regulated template withholds by itself: apply the actual constraint to both source and derived features. Relation features can also carry sensitive information; do not presume they are automatically permissible.

A web-order table with customer contact, address and device columns maps as:

{"columns": {
  "order_id": {"is_key": true}, "order_ts": {"is_base_time": true},
  "customer_name": {"domain": "names"}, "email": {"domain": "email"}, "phone": {"domain": "phone"},
  "billing_address": {"domain": "address"}, "billing_zip": {"domain": "zip"},
  "shipping_zip": {"domain": "zip"}, "ip_address": {"domain": "ip"},
  "user_agent": {"domain": "useragent"}, "delivery_note": {"type": "text"}
}}

Validation¶

The experiment's validation block decides how rows are split:

`strategy`	When	Set
`iid_stratified` (default)	rows are independent	`holdout_frac`
`grouped`	an entity has several rows (a customer's claims)	`group_column`: each entity stays on one side
`temporal_holdout`	the model scores the future	`time_column`, `temporal_boundary_mode` (`fraction` or `date` with `outer_test_cutoff`), `embargo` for a gap

probe_dataset(kind="temporal_split") previews a time split in the plan's own field names: paste its launch.validation verbatim into the plan (describe_platform("plan_schema", name="validation") lists every field). Test is the outer holdout no fit touched: read it once, at the challenge. Validation is what the run selected on (reserved rows, or out-of-fold scores): get_insight(kind="scores") gives both with their row counts, and iterations are compared on validation. Leakage specifics: a field filled after the outcome, an identifier that encodes time or the label, one entity on both sides of a non-grouped split.

Baseline¶

The approved template, the task and the metric; nothing tuned. On a subscriber table (binary):

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "churn baseline", "type": "supervised",
    "dataset_id": "<dataset_id>", "target": "churn", "template_name": "default",
    "config": {"ml_task": "binary_classification", "eval_metric": "average_precision", "positive_class": "1"},
    "validation": {"strategy": "iid_stratified"}
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "Recent logins separate churners from the rest."},
  "note": "Approved framing: churn within the next quarter, template default."
}

On a customer table with a segment label (multiclass):

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "segment baseline", "type": "supervised",
    "dataset_id": "<dataset_id>", "target": "segment", "template_name": "default",
    "config": {"ml_task": "multiclass_classification", "eval_metric": "f1_macro"}
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "Order recency and spend separate the four segments."}
}

On a property-listing table (regression):

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "price baseline", "type": "supervised",
    "dataset_id": "<dataset_id>", "target": "sale_price", "template_name": "default",
    "config": {"ml_task": "regression", "eval_metric": "mae", "intervals": {"levels": [80, 95]}}
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "Living area and location explain most of the price."}
}

On a web-order table (binary, business domains mapped as in Data, a time split):

{
  "experiment": {
    "workspace_id": "<workspace_id>", "name": "order fraud baseline", "type": "supervised",
    "dataset_id": "<dataset_id>", "target": "is_fraud", "template_name": "default",
    "config": {"ml_task": "binary_classification", "eval_metric": "average_precision", "positive_class": "1"},
    "validation": {"strategy": "temporal_holdout", "time_column": "order_ts",
                   "temporal_boundary_mode": "fraction", "holdout_frac": 0.2}
  },
  "hypothesis": {"lever": "baseline", "expected_effect": "Disposable e-mails, far shipping ZIPs and scripted browsers lift AP well above the 0.04 base rate."}
}

The floor first. The baseline_floor probe, and the run's own constant-baseline gate (get_insight(kind="scores"), baseline: the margin over a constant predictor and whether it passed). A run below the floor has learned nothing usable. A run warning about a perfect search score means a rule-like label or a leak. Read get_model_evaluation(experiment_id) (its tiles carry ci_low / ci_high) and compare(kind="experiments", ids=[experiment_id], include_production_model=True).

Iterate¶

Use the levers and pre-run checklist in get_playbook("iterate"), preserving the objective and explicit user constraints. Supervised specifics:

Before a feature idea, read get_insight(kind="engineered_features"): ParleML already built target encodings, arithmetic and group crosses, date parts, domain and text features. Propose only what is not there.
A string column read as one opaque category (its engineered_features row is a single encoded column) whose values are an e-mail, IP, address … is a role lever: the plan's column_roles.overrides gives it the domain for that run (one column per experiment).
Text settings (lever other, one per experiment, citing the text_insight read behind it): config.stateful_fe.text_analyzer: "char_wb" when top terms are fragments, codes or typos; text_ngram_range: [1, 3] when phrases carry the meaning; text_stop_words: "english" when stop words crowd the top terms; add_text_embeddings: true (text_embedding_mode augment or replace). Embeddings run on this server by default; text_embedding: {source: "provider", connection} sends the text to that provider — a paid service — at training and at every scoring call, so the user approves it before it is used (overview.md, Paid external services): a gate's question_id in a session, paid_services_approved: true outside one.
A transformation needs a model-specific hypothesis. Read the actual fitted members and existing engineered features first. A strictly monotone transform preserves feature ordering, so it generally adds no split information to an exact decision tree; it can change linear or distance-based models and numerical behavior. A feature divided by itself is constant where defined, not a useful ratio. Do not launch a fit for a redundant transformation.
An imbalanced target is not a lever to resample; a heavy-tailed regression target is handled by the run's target-transform tuning; a feature that is the target in other units is a leak.

Insights¶

Interpretation checklist. These complementary reads help explain an unfamiliar baseline or review a candidate. Select relevant available reads for the question; record material findings and missing evidence. get_capabilities(experiment_id) lists readiness and insight_encoders. Test-derived findings used to guide changes are development evidence thereafter.

get_rules(experiment_id, kind="business") (its evaluation is the rule list's own Test performance), then get_rules(experiment_id): keep holds_up: "holds_up", rank by metrics_holdout.lift; get_rules(experiment_id, kind="qa") when there are more than 20.
get_insight(kind="archetype_summary"), then get_insight(kind="archetype", params={cluster_id}) for the highest-target-rate and the highest-error archetype.
get_insight(kind="error_tree") and get_insight(kind="slices"): where the errors concentrate.
get_insight(kind="engineered_features"): which raw columns and steps built the top features — domain and text features among them?
get_insight(kind="ale"): direction and shape of the top drivers; does each make business sense?
get_insight(kind="surrogate"): how well the surrogate reproduces the model, using the returned fidelity metric and evaluation population. Interpret agreement in light of class balance and any available class-specific or reference evidence; high overall agreement can conceal poor minority-case agreement. Fidelity is not predictive accuracy against truth, and no universal cutoff establishes that an explanation is adequate for the intended use. Disclose missing fidelity evidence.
get_insight(kind="drift"): how the target cohort differs from the rest (not time drift).
get_insight(kind="label_noise") (classification) and get_insight(kind="fairness") (with a fairness column): every feature with its breaches and worst group; then params={"feature": "<name>"} for each feature with a breach — its groups, worst first. A selection-rate breach (adverse impact ratio) also fires when the groups' true rates differ: an accurate model flags more where there is more of the outcome. Before dropping a column or hunting its proxies, compare each group's flag rate with its actual outcome rate and its false-positive rate on the test rows. Breaches that track the outcome rates with close error rates are a base-rate fact for the person to sign off; a group whose false-positive rate stands out is the unfairness to fix. Dropping the column and seeing the breaches unchanged says the same: the disparity is in the outcome, or in another path to it.
get_insight(kind="text_insight", params={encoder, feature}) for every pair in insight_encoders.
get_insight(kind="glassbox") under a regulated_* template: coefficients, shapes or points.
After a deep explain only: get_insight(kind="shap_interactions"), get_insight(kind="archetype_diagnostics"), get_chart_data("mli", "pdp", experiment_id) (per_cluster_average: where an archetype reacts differently) and get_insight(kind="surrogate", params={"target": "residual"}).

Summarize important findings with their split, sample support and proposed action (get_playbook("iterate"), Learn and stop). An estimated noisy label needs review; it is not a proven error or a hard performance ceiling. A low-fidelity surrogate failing to explain residuals does not prove that errors are random or irreducible.

Challenge¶

Check	Reads required
Leakage	the top drivers (`get_model_evaluation`), `get_insight(kind="engineered_features")`, `get_insight(kind="text_insight")` on text drivers, the top business rules, `get_eda(section="leakage")` (who confirmed each finding)
Validation design	the resolved `validation` in `get_run(view="settings")`: `grouped` when entities repeat, `temporal_holdout` when the model scores the future
Stability	`get_model_evaluation` per split, `get_insight(kind="diagnose")`, `get_insight(kind="label_noise")`, `get_insight(kind="fairness")`, `get_insight(kind="archetype_diagnostics")`
Plausible drivers	`get_insight(kind="ale")` (or the PDP) of the top drivers, `get_rules(kind="business")` Test numbers, `get_insight(kind="surrogate")` fidelity, `get_insight(kind="glassbox")` under a regulated template

A _MATCH, _DIST, IP or e-mail feature among the top drivers: when is the value captured (an IP at signup or at the order; an address revised after the claim)?
Investigate fairness flags: support, uncertainty, metric, operating point and intended use. Escalate material unresolved harms; dropping a grouping column does not necessarily remove disparities or proxies. Reassess the selected policy, not only the original model threshold.
Under a regulated template, a coefficient or shape against the business direction, or against a monotone pin, is a condition.

Per task: binary — is a probability what it says (get_chart_data("mli", "calibration", experiment_id), get_insight(kind="decision_layer")); multiclass — the rarest class on its own (get_model_evaluation(experiment_id, class_key=…)); regression — the residual bias, the largest errors (get_insight(kind="error_tree")) and the interval coverage on Test (interval_coverage_<level>, interval_width_<level> in get_insight(kind="scores")): an 80 % interval that covers 70 % of test rows is a finding.

The scores payload's intervals block records the fitted method, nominal levels and calibration sample count. Its coverage is the default nominal level, not observed coverage; use the Test metrics above for that. A marginal coverage statement does not establish coverage for expensive homes, a geography or another subgroup. The construction summary does not identify individual calibration rows; do not infer their provenance from classification calibration metadata.

Decide¶

split picks the split a simulation reads (its result lists available_splits; the native business-value and multiclass-cost reads prefer validation when available). Pass validation explicitly for policy selection and inspect the returned split: the fallback may be Test when validation is absent, which does not provide independent selection evidence. Quote the chosen point's test yield once. The options are what the simulations return — never a point worked out by hand from a curve.

Binary — simulate_decision(kind="business_value", params={split, cost_fp, cost_fn, gain_tp, gain_tn}), with capacity_top_k or capacity_fraction when only so many cases can be acted on; simulate_decision(kind="decision_layer") compares calibration and threshold candidates. The operating point is a sampled threshold, not an exact top-K quota: ties stay together and grid spacing can underfill the cap. Report the returned selected count/fraction, and whether counts estimate a population from weighted samples. An infeasible sampled cap is refused rather than exceeded. Arbitrary threshold is not a supported simulation parameter; do not substitute a nearby curve point for missing exact-cutoff evidence.
Multiclass — simulate_decision(kind="multiclass_cost", params={split, cost_matrix}), the costs keyed by class label, {"<true class>": {"<predicted class>": cost, …}, …} — every class a row, every row naming every class. The per-class operating points are tuned on validation. When the decision acts on the top few classes, get_insight(kind="ranking") and simulate_decision(kind="ranking_at").
Regression — no simulation kind: the operating point is the interval level the decision acts on, with its Test coverage and width (get_insight(kind="scores")) and where the errors concentrate (get_insight(kind="error_tree")). A user-specified business rule can still be inspected with query_sql over published held-out rows: for example, count listings for which the specified offer of 92% of prediction exceeds the recorded sale price. Use the schema's actual prediction/target fields, verify keyed joins and row_role, and report the numerator, denominator and evaluation population. Such descriptive counts do not reimplement ML metrics or apply a serving policy. They do not establish future loss, acceptance, causal value or an optimized discount. Do not invent costs, reconstruct native metrics or tune the rule on Test; disclose unavailable policy evaluation separately instead of refusing ordinary evidence queries.
The rule list as a policy (classification, when decisions must be explained): get_rules(experiment_id, kind="business") → evaluation, the rules' own Test precision and recall. Set it beside the model's threshold at the same review capacity: a rule list a person can apply without the model is a real option.

Release and operate¶

Register with register_model(action="register") and its release_record after the release approval (ml-release).
Binary and multiclass: verify the fitted decision layer and its revision before describing serving behavior. A capacity simulation evaluates a proposed cutoff; it does not commit it. The current Calibration & threshold control rebuilds calibration and fits its classifier threshold; it does not expose an arbitrary numeric cutoff setter. If a recommended capacity cutoff cannot be committed through a supported control, record it as documented only and preserve the actual served threshold in the release record. Binding a release record is not evidence that the proposed cutoff was applied. Regression serves the configured interval levels.
Score new rows with score_batch(action="submit"); action="evaluation" when they are labelled.
Operate: time drift is compare(kind="datasets") against the training dataset, read against the drivers in get_model_evaluation. get_insight(kind="drift") is the target cohort against the rest, not drift over time.
Reading published results: get_playbook("schema-supervised") (per-row tables carry the key columns and row_role: join on them).