---
name: jop-data-analysis
description: Use for analysis-stage decisions on a The Journal of Politics (JOP) manuscript — uncertainty, robustness, and reporting norms — written so the work is reproducible from line one. JOP makes acceptance contingent on replicability and a JOP replication analyst re-runs the code, so every reported number must come from a script. Guides analysis; it does not fabricate results.
---

# Data Analysis (jop-data-analysis)

At JOP, analysis and **reproducibility are the same task**: acceptance is **contingent on
replicability**, and a **JOP replication analyst** re-runs your code at conditional acceptance. Write the
analysis so that every number in the paper is **regenerated by a script** — and reported with honest
uncertainty within the **page budget**.

## When to trigger

- Setting up the estimation/analysis pipeline
- Deciding which robustness checks belong in the main text vs the Online Appendix
- A reviewer asked for additional specifications, uncertainty, or sensitivity
- Preparing numbers that must match the deposited replication package exactly

## Analysis norms

- **Report uncertainty**, not just point estimates: CIs, SEs (clustered appropriately), and
  substantive effect sizes a general reader can interpret.
- **Specification transparency**: show the primary specification clearly; relegate the grid of
  alternatives to the Online Appendix, but reference it.
- **Robustness that targets the threat**: each check should answer a specific objection (confounding,
  functional form, sample, measurement), not pad the count.
- **Multiple comparisons**: adjust or pre-specify when testing many implications.
- **Substantive interpretation**: translate coefficients into quantities of interest (predicted
  probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.

## Reproducible-from-line-one (the JOP analyst will re-run this)

- One **master script** runs everything in order and sets the working directory once.
- **Set a seed** for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
- **Record software and package versions** for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
- Build a **codebook** naming and defining every variable used in the analysis.
- Tables and figures are **generated by code**, never hand-edited — numbers in the text must match.

## Fit the analysis to the page budget

- Lead with the result that carries the argument; do not narrate every regression.
- Move the robustness grid, balance tables, and diagnostics to the **Online Appendix (≤ 25 pp)**.
- A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.

## Anti-patterns

- Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
- Unseeded randomness or unpinned versions ("works on my machine")
- Star-gazing with no effect sizes or uncertainty a general reader can use
- Robustness checks chosen to inflate the count rather than rebut a threat
- Cramming every specification into the main text and blowing the page budget

## What a JOP analysis referee is looking for

The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each
demand to the move that satisfies it before the page count forces an ugly cut.

| Referee demand | Pass move | Fail signal |
|----------------|-----------|-------------|
| Usable magnitude | Marginal effect or predicted probability with CI | Coefficient stars, no magnitude in prose |
| Correct uncertainty | Cluster at assignment level; randomization inference | Default SEs on clustered or experimental data |
| Targeted robustness | Each check named to the threat it rebuts | A grid with no mapping to objections |
| Multiplicity honesty | Pre-specified families; adjusted p-values | One mined "significant" interaction |
| Reproducibility | Master script regenerates every number | "Available on request"; drifting numbers |

## Worked micro-example (illustrative figures)

A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised
turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way
fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated
states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible
re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the
group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters
by state. The robustness grid goes to the Online Appendix, cited in one line of main text.

## Referee pushback patterns and the JOP fix

- *"Your DID uses naive TWFE on staggered adoption."* Re-estimate with a heterogeneity-robust estimator,
  show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.
- *"Standard errors do not reflect the design."* Cluster at the assignment level — the state in the AVR
  example — with wild-cluster bootstrap when states are few.
- *"This interaction looks fished."* Show the pre-registered family and the adjusted p-value; concede a
  null openly.

## Output format

```
【Primary result】estimand + magnitude + uncertainty
【Robustness】each check ↔ the threat it answers (main vs appendix)
【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
【Numbers match】text == deposited output? [Y/N]
【Page discipline】main text lean, overflow in appendix? [Y/N]
【Next】jop-tables-figures
```

## Supplementary resources

- [`../../resources/external_tools.md`](../../resources/external_tools.md) — estimation packages and reproducibility tooling (renv, seeds, version pinning)
- [`../../resources/official-source-map.md`](../../resources/official-source-map.md) — JOP replicability-contingent acceptance and replication-analyst check