Iterate: learn from each useful experiment

Start from the question, the existing runs and ParleML's capabilities. Another experiment is useful only when it can resolve an uncertainty or meet a requirement that existing evidence cannot. Zero additional runs is a valid outcome. Read the type's Iterate and Insights sections for supported settings and evidence; they are a menu, not a requirement to exhaust every read.

Requirements and technical choices

Preserve the user's objective, budget, prediction-time constraints and explicit instructions. Templates, encoders, detectors and backends are ordinarily technical choices: choose or revise them with evidence within that scope. Ask when changing the business question, an explicitly fixed choice, allowed data use, paid external services or an operational approval. Do not ask the person to perform routine model selection for you. A template name alone does not prove latency, interpretability, fairness or suitability; check its resolved settings and measured behavior.

A useful pre-run checklist

Keep the reasoning short and proportional to the decision:

1. Question and mechanism: what uncertainty will this run resolve, and why might the change help this model family or satisfy a constraint?
2. Evidence: cite the run, insight, data-quality finding or domain fact motivating it. Inspect get_capabilities and relevant engineered_features, text, error, rule or other insights before duplicating work the platform already performs. SQL aggregates can establish data facts; they do not alone establish predictive value or causal effects.
3. Comparison: identify a valid reference, the evaluation population/split and what outcome would change the decision. Give an expected direction; quantify a minimum useful gain or acceptable loss when justified, without inventing a precise expected effect.
4. Existing capability: prefer a supported encoder, domain, transformation, simulation, explanation refresh or existing artifact over a new training run or custom ML code.
5. Value and cost: is the information worth the compute and time, within the agreed budget?

Prefer controlled contrasts. Several changes can belong to one coherent correction (such as removing leakage and repairing its split); record them together and do not attribute the gain to one component. A split repair establishes a new reference, not an improvement over an invalid old score. Check the task schema before naming settings.

Features and ablations

Low importance alone does not justify removal or a retrain. ParleML already engineers and selects features; inspect what the actual run did. Trees can still suffer from noisy or redundant features, and implementation details can break ideal monotone-transform invariance. Require a plausible reason rather than declaring feature removal always useless or always beneficial.

Use an ablation when the question is whether retraining without an input changes performance, cost or compliance. Importance, SHAP and surrogate rules explain the fitted model; they are not substitutes for that intervention. Reuse a comparable existing full model; run only needed arms. Remove unavailable, leaking or prohibited inputs before the baseline. Domain companion and relation features are useful only when their inputs and derived information are permitted and available at prediction time; distances and match flags can also carry sensitive proxies.

Threshold, capacity, cost-matrix and ranking-tilt questions use simulate_decision where supported. Deeper explanations use the existing explain operation. Neither inherently needs another fit.

The levers

What each lever changes for a type — which block, which field — is that type's playbook, section Iterate. Every setting you plan is one describe_platform("plan_schema", name=<type>) lists. hypothesis.lever takes only these names — baseline for the first run, then the ones below — and plan validation refuses any other ("ablate the notes" is column_roles, with the change in note).

Lever (hypothesis.lever)	How you change it
recipe / features	A new recipe for the iteration (author_recipe(action="create"), named after the approved one and the iteration): the approved SQL with the new columns, then materialise with roles_from_dataset_id = the approved dataset (it keeps the approved roles for every column still there; set roles only on the columns you added); or the feature columns a type's launch block lists. Then scan the new dataset for leakage (confirmations carry over; new findings go to the person). The approved recipe is never edited. Feature ideas become SQL columns, never Python.
column_roles	The plan's own roles, never the approved dataset: a supervised plan's experiment.column_roles (overrides per column — is_ignored, domain, type; monotone_constraints), which win over the dataset's for this run only; the other types' launch-block columns (feature_cols, passive_columns). prepare_data(action="columns") changes the approved dataset for every run — a data-step act that needs the data gate.
split	How the run validates: a supervised run's validation block, a forecast's CV windows, a holdout fraction.
decision_layer	Not a training change: simulate_decision, and a person applies the operating point in the ParleML UI.
template / preset / metric	Technical choices may change within the agreed goal and budget. A metric change needs a scientific reason and re-evaluation of comparable candidates; never switch metrics to manufacture a win. Respect explicitly fixed choices.
constraint	A requirement a person set after the baseline (a column ruled out for fairness, law or timing): every such requirement in one run, judged with a tolerance (negative margin). It re-baselines — the run it supersedes is no longer the best so far — and later levers start from it. Requirements known at the framing or data gate are never a run: they are in the baseline's plan.
other	A setting no lever above names — a text-encoder setting, an algorithm, a seed for a stability check: say which in note. Never removing inputs: the plan warns that such a run tests nothing.

Before launching

Name the parent with run_experiment(action="plan", parent_experiment_id=<parent>, params=…). Keep existing recipes and approved dataset roles intact: new data transformations get a new recipe; per-experiment role changes use plan overrides. Read the resolved config, estimate and warnings. Resolve invalid splits, leakage and missing requirements. Imbalance by itself is not a reason to abort: assess minority counts, split feasibility and the chosen metric.

The plan's stop_rule is a practical margin check, not a significance test. fold_noise is a fold standard deviation, not uncertainty in the paired difference. Respect the agreed stopping budget and report stop_rule.verdict without recomputing it; an undecidable result needs its reason. A different, justified scientific question may remain after one line stops.

Evaluate without fooling yourself

• Read get_run: distinguish training success from missing explanations, scores or publication. A resource failure is not evidence against the hypothesis; check capacity before any retry.
• Compare with a relevant simple baseline and the business requirement. A weak score identifies a limitation; no universal numerical floor proves that a model is useful.
• Use compare(kind="experiments", ids=[parent, candidate], include_production_model=True) and available evaluation evidence. Check matching outcome, population, split, horizon, metric, positive class and operating point. A different task or holdout is not a controlled comparison.
• A candidate inside or outside the parent's confidence interval does not test their difference. Separate model intervals and fold spreads do not give a paired comparison. Use a returned paired difference interval when provided: ranking reports already provide entity-cluster bootstrap differences against popularity baselines. That interval supports that specific comparison, not an arbitrary pair of experiments. Quote observed changes and returned uncertainty; when comparative uncertainty is unavailable, call it an observed validation gain with uncertain reliability, not statistically confirmed.
• Use development evidence for model and operating-point selection. If Test-derived rules, error trees, archetypes or scores guide changes, that Test set has become development evidence. Record that exposure. Re-reading it once in a later challenge does not restore independence. Use a genuinely untouched evaluation set when available; otherwise disclose the limitation.
• Multiple comparisons, tiny slices and repeated tuning weaken conclusions. A missing metric is unavailable evidence; failure to detect an effect does not establish equivalence.

Learn and stop

Choose interpretation reads that resolve the current question. A broad review is valuable for an unfamiliar baseline or consequential release; repeated full sweeps of unchanged evidence are not automatically useful. Record important findings, their split and support, uncertainty, and what action they warrant. Different explainers agreeing on the same model/data are corroborating views, not independent replications. Model explanations and what-if responses are not causal claims about interventions in the world.

Stop when the question is answered, the candidate meets the goal, useful ideas run out, or the budget is reached. A simpler, cheaper or compliant candidate can be preferred without a score gain. Report unresolved limitations and the reason for the choice. Never invent runs to complete a lifecycle stage.

A compact Gems table can help hand off several findings: finding, evidence and object id, evaluation split and support, uncertainty, and next action. Exploratory leads are useful too; label them as such rather than forcing every finding to claim independent validation.