---
name: python-pandas
description: Write Pandas code to this project's standards — vectorised operations, Pandas-native types, and Pandera schemas. Use when importing or using pandas — transforming DataFrames or Series, handling missing values, or type-hinting tabular data.
---

# Python Pandas

Standards for working with Pandas DataFrames and Series. Extends the "prefer library
idioms" rule in `python-code-style` with Pandas specifics.

## Vectorise

- Operate on whole Series and DataFrames; don't loop with `.apply`, `.iterrows`, or a
  Python `for` over rows. Reach for Series methods — `.where`/`.mask`/`.map`/`.clip`/
  `.str.*`, `.between`, `.isin` — and `df.eval`/`df.query` for arithmetic and filtering.
- Prefer `Series.map(mapping)` to `.apply(lambda x: mapping[x])`, and use `&`/`|` only for
  row-wise boolean masks — keep `and`/`or` for scalar conditionals.
- When you genuinely must iterate, use `.itertuples()` (named, typed, fast), never
  `.iterrows()`.
- Stay in Pandas types all the way to the function boundary — don't drop to Python lists
  or NumPy arrays mid-pipeline and convert back.
- Suffix DataFrame variables `_df` so the type is visible at the call site; name Series
  for their contents (e.g. `unit_price`, not `s` or `data`).

## Don't mix NumPy into Pandas

- Use `pd.NA`/`pd.NaT` for missing values, not `np.nan`, and prefer the nullable
  extension dtypes (`Int64`, `boolean`, `string`) so missingness is first-class — NumPy
  float columns silently coerce `pd.NA` to `NaN`.
- Prefer Series methods over `np.*` functions on a Series (including via `.apply`), and
  never use `.values` (the `PD011` lint flags it) — reach for `.to_numpy()` only at a
  boundary needing a raw array.
- Don't store NaN as a sentinel; model "missing" explicitly with a nullable dtype.
- Keeping everything Pandas-native also keeps a future move to Polars tractable.

## Pandera schemas

- Lean on Pandera: type-hint every DataFrame parameter and return with `DataFrame[Model]`.
  The payoff is readability — the schema becomes explicit at every reference — so use it
  ubiquitously, not sparingly.
- Give each schema model its own module, and keep the raw (as-ingested) schema in a
  separate module from the processed (validated or derived) one.
- Back categorical columns with a `Category` dtype built from an `Enum`'s values (iterate
  the Enum to build the categories), and annotate timestamp columns as `pd.Timestamp`.