---
name: calibrate_probabilities
description: How to evaluate and fix probability calibration — holdout discipline, Brier/log-loss reading, Platt-style rescaling.
triggers: [calibrate, calibration, brier, log loss, log-loss, overconfident]
---
# Calibrating probabilities

A model that says 70% should be right ~70% of the time. Calibration is measured,
never assumed.

## Measure (always on holdout)

- `calibration_metrics(pairs)` where pairs = holdout `{prob, outcome}` rows.
- **Brier**: mean squared error. 0 = oracle; 0.25 = coin flip on a balanced set;
  beating the market baseline matters more than the absolute number.
- **Log-loss**: punishes confident wrongness. If log-loss looks much worse than
  Brier, the model is overconfident in its tails.

## Fix overconfidence (in run_python)

- Shrink toward the base rate: `p' = w * p + (1 - w) * base_rate`, fit w on a
  validation slice (grid over w ∈ [0.5..1.0] minimising log-loss is fine).
- Platt-style: fit logistic regression of outcome on logit(p) — two parameters,
  hard to overfit; refuse fancier recalibration without more than ~200 samples.
- Re-run `calibration_metrics` AFTER rescaling and report both before/after.

## Persist

`save_model` with the post-calibration metrics and the rescaling parameters in
`params` — the next session must be able to reproduce the pipeline from the row.