---
name: project-organization
description: Use when finishing a piece of analysis and about to commit or push, when a project directory has accreted intermediate or diagnostic files and it's unclear what is a deliverable versus scratch, when setting up or reorganizing a research repository, or when asked to clean up the directory, organize the repo, standardize naming, make a project reproducible for collaborators, or decide what folder structure to use. Also fires throughout the work — when creating a new script, dataset, table, or figure, place and name it per the project's structure instead of dumping it in the root or a flat folder to sort out later, and at phase boundaries (plan agreed, a clean dataset built, a result validated) to commit a local checkpoint so the work is durable. For empirical and structural economics research projects across R, Julia, Python, and Stata, organized around the paper they produce. Triggers on "clean up the repo", "organize the project", "what should the folder structure be", "before I push this", "standardize the naming", "make this reproducible for a collaborator", "commit a checkpoint", or "I haven't committed in a while".
---

# Project Organization

## Overview

A research repository has one destination — the **paper** — and one test: six months in, a collaborator (or future you) opens it and must answer fast — *what is a finished result, what is scratch, and how was each number produced?* Organization is what makes the work reproducible and legible to someone who wasn't in the conversation: `analysis-craft`'s clean-diff discipline, scaled to the directory.

**Core principle:** organize around the paper and its pipeline; place and name every artifact so its role (deliverable / intermediate / raw) and its producer are obvious; and at the end of a workflow, before you commit, tidy so the repo shows deliverables and code, not scratch.

## The structure

```
project-root/
├── README.md             # the map: what this is, data provenance, one-command reproduction
├── paper/                # the destination — manuscript + the FINAL tables/figures it uses
├── data/
│   ├── raw/              # original, immutable — code reads it, never writes it
│   ├── intermediate/     # cleaned / merged working data (reproducible)
│   └── output/           # analysis-ready datasets the analysis reads
│       └── mortality/ crime/ …     # subject subfolders where the project spans several
├── code/                 # pipeline ordered by paper stage (R / Julia / Python / Stata)
│   ├── 00_data/          # raw → intermediate → output    ── subjects: mortality/ crime/ …
│   ├── 01_stylized_facts/
│   ├── 02_main/          #                                 ── subjects: did_mortality/ did_crime/ …
│   │   └── archive/      # superseded runs for this category — kept, not deleted, out of the active path
│   ├── 03_estimation/ 04_montecarlo/ 05_robustness/ 06_counterfactual/   # (structural)
│   └── lib/              # shared functions
├── results/              # generated artifacts — tables/ figures/ (canonical) · diagnostics/ (scratch) · archive/ (old runs, kept)
├── docs/                 # MODEL_CARD.md · pre-analysis-plan.md · LESSONS.md
├── sandbox/              # exploratory throwaway, clearly not a deliverable
└── .gitignore
```

Adapt it: reduced-form projects drop the structural stages; single-subject projects drop the subject subfolders; the pipeline folder may be `code/`, `scripts/`, or `analysis/`. **Read the repo's `README.md` first and follow the conventions it already has** — impose this only where there are none. The point is the separation (**code ≠ data ≠ outputs ≠ scratch ≠ archive**) and the paper-centric staging, organized **stage-first with subject subfolders throughout — the data stage included.**

**Each category keeps an `archive/`.** Old runs that are no longer used but worth keeping — a superseded specification, last quarter's results, a model version you moved off of — go into an `archive/` inside their category (`code/02_main/archive/`, `results/archive/`, …), not into the active tree and **not into the trash**. Archive is distinct from `sandbox/` (exploratory throwaway, disposable) and from `results/diagnostics/` (regenerable scratch): archive is **deliberately retained, inactive provenance.** It's the default destination when you retire a run — moving to `archive/` beats deleting, and keeps the active folders showing only what's current.

## Naming — standardize it

A filename announces its order, role, and what it produces. Lowercase, `snake_case`, no spaces.

- **Scripts:** `NN_verb_subject` — `01_clean_mortality.R`, `02_build_county_panel.py` (numbers = run order).
- **Subject folders:** `method_subject` — `did_mortality/`, `blp_demand/`.
- **Outputs:** named by what they show, traceable to their producer — `tab_main_did_mortality.tex`, `fig_eventstudy_mortality.pdf`.
- **Data:** content + grain — `county_year_panel.parquet`.
- **Banned:** dates, `final`/`v2`, spaces, capitals, and two names that differ only by an undecodable suffix. Git versions; the filename doesn't.

## Data — raw → intermediate → output

One direction. `raw/` is **immutable**: code reads it and never writes it; a fix happens in a cleaning script that writes `intermediate/`. The data folders carry subject subfolders when the project spans several.

**Track the data by default** — a research repo should ship what a replicator needs. Gitignore only two things: data that is **sensitive and can't be shared** (PII, restricted, licensed), and files **past GitHub's ~100 MB limit**. For a file that is merely too big (not sensitive), don't drop it — **convert it to a compact format (parquet, or compressed tsv) first**; that usually clears the limit and keeps it tracked and replication-ready. Whatever you genuinely can't ship, document its provenance in the README so a replicator can obtain it.

## Git — tracked vs. ignored

The repo a collaborator sees should be code + manuscript + results + the shareable data.

- **Track:** `code/`, `paper/` (incl. final tables/figures), `docs/`, `README`, the `archive/` folders (old runs are kept, so they're tracked — subject to the same shareable/size rule), and the `data/` and `results/` that are shareable and under the size limit.
- **Gitignore:** secrets (`.env`, keys) **always**; sensitive data; oversized files (after trying to shrink them); `results/diagnostics/`, `sandbox/`, caches, logs.
- **One-command reproduction:** a master script / `Makefile` that runs the pipeline in order (data → … → paper). It's the replication standard and the proof the structure is real.

## Checkpoint as you go — commit locally in phases

A multi-hour analysis sitting at **zero commits** is a failure mode, not caution. `analysis-plan.md` and your validated intermediates are only durable as *disk-as-RAM* if they're checkpointed — otherwise one stray `git switch`, `reset`, or editor mishap loses the whole uncommitted tree, and a fresh session can't resume from work that was never saved. So **commit locally at phase boundaries, proactively, without being asked** — "only commit when the user asks" is wrong for a research repo.

- **When to checkpoint:** the plan is written and agreed; a `data-preparation` phase produces a clean dataset; a result is validated and frozen (`data-contracts` golden output); a debugging session lands a fix. Each is a natural restore point — close it with a commit.
- **Commit ≠ push.** A **local commit** on your working branch is cheap, private, and reversible — that's the checkpoint, and you do it on your own. **Pushing** (or opening a PR) is outward and shared — *that* stays the user's explicit call (`analysis-checkpoints`). The conservative "only when asked" default governs **push**, never the local checkpoint.
- **Name the milestone:** say what phase closed — `data: clean panel assembled, 1:1 join asserted`, `did: primary spec estimated + verified`, `plan: PAP signed off` — so the history reads as the pipeline's progress and is recoverable at any step. Label work-in-progress `wip:` so a half-done state isn't mistaken for a validated one.
- **Don't checkpoint junk:** the Track/Gitignore rule above still holds — never commit secrets, `raw/` data, or oversized files, even at a checkpoint.

## Enforce throughout, tidy before git

- **Throughout:** when you create a script, dataset, table, or figure, put it in its folder and name it by convention *then* — not "in the root for now."
- **Before you commit (the trigger):** read the README's conventions; inventory what the workflow produced; classify each file — deliverable → its folder; **superseded-but-keep (an old run no longer used) → the category's `archive/`** (move, never delete); intermediate/reproducible → keep or gitignore; scratch/diagnostic → `sandbox/` or `results/diagnostics/`; sensitive/oversized → shrink-or-gitignore; secret → gitignore now. Propose the moves/renames/gitignore/deletes; update `.gitignore`.
- **Safety:** offer, don't delete on your own — deleting files the user may want is a checkpoint (`analysis-checkpoints`). **Never touch `raw/`.** The default for a retired run is **move to `archive/`, not delete**; prefer gitignore/move over delete throughout, and move a doubtful file to `sandbox/` rather than removing it.

## Red flags — STOP

- Code, data, and generated outputs in one flat folder.
- Diagnostic CSVs / check-plots committed beside the canonical figures, with nothing marking which is which.
- `raw/` edited in place, or written to by analysis code.
- Secrets tracked in git; a large file gitignored without first trying to shrink it to parquet/tsv.
- Filenames with dates, `final`/`v2`, spaces, or capitals.
- No master script — "run them in the right order, they're in here somewhere" is not reproduction.
- About to delete files in a cleanup without surfacing the plan, or about to touch `raw/`.

## Common rationalizations

| Excuse | Reality |
|---|---|
| "I'll dump it in the root and organize later." | Later never comes; the flat dump is the mess this cleans up. Place it now. |
| "Gitignore all the data to be safe." | A research repo should ship the data a replicator needs. Ignore only the sensitive or the oversized — and shrink the oversized to parquet/tsv first. |
| "Dates and `v2` track versions." | Git does. `results_final_v3_REALLY.csv` is how you lose the canonical one. One name per artifact. |
| "Just delete the scratch to tidy up." | Deleting files the user might want is their call. Offer; prefer gitignore/move; never touch `raw/`. |

## When to Use → where this hands off

Organization is the **terminal** step — the chain *routes into* it and *ends* here. Don't propel to a successor; land the work as shippable:

```dot
digraph project_organization_next {
    "Result verified? (reconciled / reproduced clean)" [shape=diamond];
    "invoke result-verification — freeze before you organize" [shape=box style=filled fillcolor=lightyellow];
    "About to delete/move files the user may want?" [shape=diamond];
    "invoke analysis-checkpoints — moves/deletes are a gate" [shape=box style=filled fillcolor=lightgreen];
    "TERMINAL: deliverables placed, scratch archived, names standardized → shippable" [shape=box style=filled fillcolor=lightgrey];
    "Result verified? (reconciled / reproduced clean)" -> "invoke result-verification — freeze before you organize" [label="no — go back"];
    "Result verified? (reconciled / reproduced clean)" -> "About to delete/move files the user may want?" [label="yes"];
    "About to delete/move files the user may want?" -> "invoke analysis-checkpoints — moves/deletes are a gate" [label="yes"];
    "About to delete/move files the user may want?" -> "TERMINAL: deliverables placed, scratch archived, names standardized → shippable" [label="no"];
}
```

## The Process

1. **Read the README's conventions first**, then inventory what the workflow produced.
2. **Classify each artifact** — deliverable → its folder; superseded-but-keep → the category's `archive/` (move, never delete); intermediate → keep or gitignore; scratch/diagnostic → `sandbox/` or `results/diagnostics/`; sensitive/oversized → shrink-or-gitignore; secret → gitignore now.
3. **Before any move or delete the user might want kept → STOP and invoke `analysis-checkpoints`.** Never touch `raw/`; prefer move/gitignore over delete.
4. **Standardize names, update `.gitignore`, confirm one-command reproduction.**
5. **This is the end of the chain — do not invoke a successor.** When deliverables are placed, scratch is archived, and names are standardized, the work is **shippable**: ready to commit and push.

## The bottom line

```
Organized  →  paper-centric stages × subject subfolders, raw→intermediate→output, shareable data
              tracked (only sensitive/oversized excluded, big files shrunk to parquet/tsv),
              standardized names, one-command reproduction, scratch tidied before commit
Otherwise  →  a folder only the author can navigate, and only this week
```