---
name: train-evaluate-ml-model
description: Trains and evaluates a classic (non-LLM) ML model — business-aligned metric selection, leakage-safe train/validation/test splits, Pipeline-scoped feature engineering, baseline-first model selection, cross-validated hyperparameter tuning, bias/variance diagnosis, and experiment tracking — guarding against data leakage and overfitting.
when_to_use: Fitting and validating a classification, regression, ranking, forecasting, or clustering model on tabular/feature data. Distinct from profile-dataset (EDA only), wrangle-tabular-data (cleaning/shaping the feature table), serve-deploy-ml-model (deployment), monitor-ml-drift (post-deploy), and rag-pipeline/prompt-engineering (LLM work).
---

## When to Use

Reach for this skill when the request is to **fit and validate a model that predicts**, not to explore or clean data:

- "Train a classifier to predict churn / fraud / default and tell me if it's good"
- "Build a regression/forecasting model for demand/price and pick the best one"
- "My model gets 99% accuracy — is that real or leakage?"
- "Tune hyperparameters / cross-validate / compare XGBoost vs logistic regression"
- "Cluster these customers into segments"

NOT this skill:
- Summary stats, distributions, correlations, missingness *before* modeling → profile-dataset
- Cleaning, type coercion, joins, dedup, resampling to build the feature table → wrangle-tabular-data
- Asserting schema/range/null contracts on the data before training → validate-data-quality
- Packaging the trained model behind an API / batch job → serve-deploy-ml-model
- Watching a live model's inputs/outputs degrade over time → monitor-ml-drift
- Scoring or grading an LLM's outputs against a rubric/golden set → llm-eval-harness
- Getting an LLM to answer over a corpus, or designing prompts → rag-pipeline / prompt-engineering

## Steps

1. **Choose the metric from the business cost FIRST — never optimize bare accuracy on imbalanced data.** A 1%-positive fraud set scores 99% accuracy by predicting all-negative and catches zero fraud. Pick before you train:

   | Task / situation | Metric | Why |
   |---|---|---|
   | Imbalanced classification, cost of missing a positive high | Recall @ fixed precision, or **PR-AUC** | accuracy & ROC-AUC look great while recall is ~0 |
   | Imbalanced, ranking/threshold-free comparison | **PR-AUC** (not ROC-AUC) | ROC-AUC is optimistic under heavy imbalance |
   | Balanced classification | F1 or ROC-AUC | symmetric cost |
   | Asymmetric FP vs FN cost | Expected cost = `cFP·FP + cFN·FN`, tune threshold | maps directly to money |
   | Regression, outliers matter | RMSE | penalizes large errors |
   | Regression, robust to outliers | MAE / MAPE | business reads "off by X" |
   | Ranking / recsys | NDCG@k, MAP@k | position-aware |
   | Forecasting | MASE / sMAPE vs naive | scale-free, must beat seasonal-naive |
   | Clustering (no labels) | silhouette + downstream business check | inertia alone is meaningless |

   Fix the **decision threshold** from the cost matrix later — don't ship the default 0.5.

2. **Split BEFORE any feature engineering or fitting — this is the #1 leakage source.** Three sets: train / validation (or CV folds) / **held-out test touched once at the very end**. The split strategy is not optional — pick by data structure:
   - **Random stratified** for i.i.d. rows: `train_test_split(X, y, stratify=y, test_size=0.2, random_state=42)`.
   - **Time-based** for any temporal data — train on past, test on future, never shuffle. Use `TimeSeriesSplit` for CV. A random split on time-series leaks the future and inflates every metric.
   - **Group split** when rows share an entity (same user/patient/device across rows): `GroupKFold` / `StratifiedGroupKFold` so the same group never appears in both train and test.

3. **Engineer features inside a `Pipeline` fit only on train, then run a leakage audit.** Every transform that *learns* (imputation means, scaler stats, target/one-hot encoders, feature selection) must `.fit` on train and only `.transform` val/test — otherwise test statistics leak in. Wrap it:

   ```python
   from sklearn.pipeline import Pipeline
   from sklearn.compose import ColumnTransformer
   from sklearn.impute import SimpleImputer
   from sklearn.preprocessing import StandardScaler, OneHotEncoder
   from sklearn.model_selection import cross_val_score

   pre = ColumnTransformer([
       ("num", Pipeline([("imp", SimpleImputer(strategy="median")),
                         ("sc", StandardScaler())]), num_cols),
       ("cat", OneHotEncoder(handle_unknown="ignore"), cat_cols),
   ])
   pipe = Pipeline([("pre", pre), ("model", model)])
   # CV refits `pre` per fold → no leakage across folds
   cross_val_score(pipe, X_train, y_train, cv=cv, scoring="average_precision")
   ```

   **Leakage audit checklist** — a feature is leaky if it: (a) is derived from the target or post-outcome (e.g. `payment_received` predicting `will_pay`); (b) encodes future information unavailable at prediction time; (c) is an ID/timestamp that proxies the label; (d) was computed using full-dataset statistics before the split. If any single feature gives a near-perfect score, it's leakage, not skill.

4. **Establish a dumb baseline before any real model.** `DummyClassifier(strategy="most_frequent")` / `DummyRegressor(strategy="mean")`, then a linear baseline (`LogisticRegression` / `Ridge`). This is the bar every model must clear; a fancy model that barely beats majority-class isn't worth the complexity.

5. **For tabular data, reach for gradient boosting before deep nets.** Default order: linear baseline → **gradient boosting** (`XGBoost` / `LightGBM` / `HistGradientBoostingClassifier`) → deep net only if GBM plateaus and you have ample data. GBMs win on heterogeneous tabular data, need little preprocessing, and train in minutes. Handle imbalance with `class_weight` / `scale_pos_weight` or threshold tuning — not blind SMOTE (and if you oversample, do it inside the CV fold only, never before the split).

6. **Tune hyperparameters with CV, search smart not exhaustive.** `RandomizedSearchCV` or Optuna over a sensible space beats `GridSearchCV` on a huge grid. Always pass `scoring=` your step-1 metric (not accuracy) and use the same CV object as step 2. Key GBM knobs: `n_estimators` + `learning_rate` (trade off), `max_depth` / `num_leaves`, `min_child_samples`, `subsample`, `colsample_bytree`, plus `early_stopping_rounds` on a validation set.

7. **Diagnose bias vs variance, then act.** Compare train vs validation score:
   - Train high, val low (large gap) = **overfit/variance** → regularize, reduce depth/leaves, add data, drop features, stronger early stopping.
   - Train and val both low = **underfit/bias** → richer model, better features, less regularization.
   Plot a learning curve to decide whether more data would even help before collecting it.

8. **Track every experiment — params, metric, data version, code version.** Log to MLflow / Weights & Biases (or a CSV at minimum): hyperparameters, all CV metrics with std, the data snapshot hash, git commit, and the random seed. An untracked best-run is unreproducible. Pin seeds (`random_state`) everywhere.

9. **Evaluate the held-out test set EXACTLY ONCE, at the end.** Report the step-1 metric with a confidence interval (bootstrap), the confusion matrix at your chosen threshold, and the gap vs baseline. Repeatedly peeking at test = overfitting to test by hand.

## Common Errors

- **Splitting after feature engineering / scaling on the full dataset.** Test statistics bleed into train; metrics inflate, prod collapses. Split first, fit transforms on train only (use a `Pipeline`).
- **Random split on temporal data.** The model trains on future rows and "predicts" the past. Use a time-based split / `TimeSeriesSplit`.
- **Reporting accuracy on an imbalanced problem.** 99% accuracy with 0% recall is useless. Pick PR-AUC / recall-at-precision from the cost (step 1).
- **A feature that's too good.** One column driving a near-perfect score is almost always leakage (post-outcome field, ID proxy, target-derived). Audit and drop it.
- **Target encoding / imputation / feature selection computed before the split or outside the CV fold.** Subtle leakage that survives CV but not production. Fit them inside the pipeline, per fold.
- **SMOTE/oversampling applied before splitting.** Synthetic copies of test rows land in train. Resample inside the training fold only.
- **Tuning against the test set / peeking repeatedly.** You overfit to it manually. Tune on CV/validation; touch test once.
- **No baseline.** Without DummyClassifier/linear you can't tell if your model learned anything. Always beat the dumb baseline first.
- **`scoring=` left at default (accuracy/R²) during search.** The search optimizes the wrong thing. Pass your business metric to `cross_val_score` / `*SearchCV`.
- **Train/serve feature skew.** Features computed differently (or with different code/library versions) at training vs inference. Reuse the exact fitted pipeline artifact for serving.
- **Unpinned seeds / untracked runs.** Results aren't reproducible and "best model" can't be recovered. Pin `random_state`, log params+metrics+data+code version.

## Verify

1. **Beats baseline:** held-out test metric (step-1 metric) > the DummyClassifier/DummyRegressor and the linear baseline by a margin larger than the bootstrap CI. If it doesn't, there's no model.
2. **Leakage check passes:** drop each top-importance feature individually — no single feature should collapse the score to chance-level; remove any post-outcome/target-derived/ID-proxy column; confirm all learned transforms were fit on train only.
3. **Split integrity:** for temporal data, every test timestamp is strictly after every train timestamp; for grouped data, no group ID appears in two sets (assert programmatically).
4. **Generalization gap sane:** train metric − validation metric is small and explained by your bias/variance call; test ≈ validation (a big test drop means tuning overfit to validation).
5. **Metric matches business cost:** the reported metric and the chosen decision threshold come from step 1, not the library default.
6. **No train/serve skew:** running the saved pipeline on a held-out row reproduces the exact training-time prediction (same features, same library versions).
7. **Reproducible:** re-running with the logged seed + data snapshot + code commit yields the same metric (within float tolerance); the experiment tracker has params, CV metrics with std, data hash, and git commit.

Done = the held-out test metric (chosen for business cost, evaluated once) beats both baselines beyond its CI, the leakage and split-integrity checks pass, the generalization gap is explained, and the saved pipeline reproduces predictions with no train/serve skew under a pinned, tracked run.