---
name: predictive-modeling-best-practices
description: "Guides predictor selection, collinearity checks, cross-validation strategy, and hyperparameter tuning for ecological predictive models. Use this skill when the user mentions VIF, collinearity, feature selection, spatial cross-validation, block CV, hyperparameter tuning, overfitting prevention, data leakage auditing, background point selection, pseudo-absence generation, ENMeval tuning, regularization, or spatial autocorrelation correction."
skill_version: 1.0.0
---

# Skill: predictive-modeling-best-practices

**Domain:** CV · Tuning · Leakage · Collinearity · Overfitting  
**Phase:** 1 — Foundation  
**Used by:** run-sdm-study, build-fire-risk-map

---

## Purpose

Ensures that any predictive model in the project is built with sound ML practices: proper data splitting, cross-validation strategy, hyperparameter tuning, collinearity reduction, leakage prevention, and overfitting diagnosis.

---

## When to Invoke

- Before fitting any algorithmic model (MaxEnt, BRT, Random Forest, ANN, GLM for prediction)
- When designing the validation strategy for a modeling study
- When the user asks about feature selection, predictor reduction, or model tuning

---

## Inputs

| Input | Format | Required |
|-------|--------|----------|
| Feature matrix (predictors) | CSV, data frame | Yes |
| Target variable | Vector (binary, continuous, multiclass) | Yes |
| Spatial coordinates (if applicable) | lat/lon columns | Recommended |
| Candidate model list | Text description | Optional |

---

## Outputs

| Output | Description |
|--------|-------------|
| `cv_strategy.md` | Chosen CV method with rationale |
| `collinearity_report.csv` | VIF and pairwise correlation for all predictors |
| `selected_predictors.txt` | Final predictor set after reduction |
| `tuning_results.csv` | Hyperparameter grid search results |
| `leakage_audit.md` | Confirmation of no data leakage |
| `modeling_plan.md` | Complete modeling plan document |

---

## Steps

### 1. Define the Modeling Objective
- Regression, binary classification, or multiclass?
- Interpolation within the study area or extrapolation to new areas/times?
- Primary metric: AUC, TSS, RMSE, R², F1?

### 2. Data Splitting Strategy

**For non-spatial data:**
- Random split: 70% train / 15% validation / 15% test (or use k-fold CV)

**For spatial data (required for SDMs and most ecological models):**
- Spatial block cross-validation (checkerboard or custom blocks)
- Block size should exceed the spatial autocorrelation range
- Never use random splits for spatially autocorrelated data

**For temporal data:**
- Forward-chaining (walk-forward) CV; never shuffle temporal order

### 3. Collinearity Assessment
- Compute Pearson/Spearman correlation matrix for all predictors
- Flag pairs with |r| > 0.7
- Compute VIF; flag predictors with VIF > 5
- Reduce collinear predictors using:
  - Ecological/domain knowledge priority
  - PCA (when interpretability is not critical)
  - VIF-stepwise removal

### 4. Leakage Audit
- Confirm that target variable information does not appear in any predictor
- Confirm that future information is not used for past predictions
- Confirm that validation/test data were not used during feature engineering or scaling

### 5. Hyperparameter Tuning
- Define tuning grid for each candidate algorithm
- Use the training set + CV folds only; never touch the test set
- Report best hyperparameters and CV performance curve
- Flag overfitting: large gap between train and CV performance

### 6. Feature Importance Pre-selection (optional)
- Run a preliminary model to rank feature importance
- Remove predictors with near-zero importance AND high collinearity burden
- Re-run CV with reduced predictor set; confirm no performance loss

### 7. Finalize and Document Modeling Plan
- Chosen algorithm(s)
- CV strategy
- Final predictor set
- Tuned hyperparameters
- Primary evaluation metric

---

## Decision Points

| Condition | Diagnosis | Recommended Action |
|-----------|-----------|-------------------|
| VIF > 10 for any predictor | Severe collinearity — model coefficients are unstable | Remove predictor with highest VIF iteratively until all VIF < 10 |
| Pearson \|r\| > 0.7 between predictors | High correlation — retaining both inflates false precision | Retain only one from each correlated pair based on ecological relevance |
| CV AUC substantially > independent test AUC (> 0.1 gap) | Spatial autocorrelation causing overly optimistic CV | Switch to spatial block CV; document block size in `decision_log.md` |
| n_folds > n_observations / 10 | Too many folds — test sets are too small for reliable metrics | Reduce to max(5, floor(n/10)) folds; use repeated CV if n < 50 |

---

## Key Decisions to Document

- CV strategy and block size (for spatial CV)
- Collinearity threshold used
- Predictor selection method
- Tuning method (grid search, random search, Bayesian)
- Train/validation/test split sizes

---

## Tools and Libraries

**R:** `caret`, `tidymodels`, `blockCV`, `ENMeval`, `corrplot`, `usdm`  
**Python:** `scikit-learn`, `optuna`, `shap`, `scipy.spatial`

---

## Resources

- `resources/spatial-cv-guide.md` — spatial block CV configuration guide
- `resources/collinearity-decision-tree.md` — when and how to remove predictors
- `examples/` — worked tuning examples for BRT and Random Forest

---

## Notes

- Spatial CV is mandatory for SDMs and any model with spatially autocorrelated responses
- Report both training and CV/test performance; never report training performance alone
- Regularisation (LASSO, ridge) is preferred over manual stepwise selection