---
name: research-methodology
description: Ethan Perez's tips for empirical alignment research - velocity, experimentation, collaboration
---

# Research Methodology: Tips for Empirical Alignment Research

*Based on [Ethan Perez's post](https://www.alignmentforum.org/posts/dZFpEdKyb9Bf4xYn7/tips-for-empirical-alignment-research). Applies to highly experimental LLM alignment research (scalable oversight, adversarial robustness, chain-of-thought faithfulness, process-based oversight, model organisms of misalignment).*

## Core Principles

### 1. Hard Work Pays Off
- Run as many experiments as you can, tinker a lot, try lots of stuff
- Often many reasonable-sounding ideas need testing—sometimes the 5th or 20th thing is what works
- **"Rich get richer" effect**: the more you experiment, the better you get at picking the right experiments
- More things work when you try more stuff → easier to stay motivated

### 2. Rapid Iteration (Velocity)
- **Reduce uncertainty at the fastest possible rate** (Jacob Steinhardt)
- Get quick feedback and iterate on ideas rapidly
- With LLMs, you can reduce uncertainty really quickly—even with a single message to GPT-4/Claude
- Can gain 1+ OOMs more information per unit time by derisking ideas in the quickest way possible

### 3. Avoid the "Swamp"
- The **swamp** = getting stuck when you really want technique X to work but nothing succeeds
- Solution: **high velocity**—test as many ideas as possible per unit time until you escape
- For LLM research: you should rarely be stuck in a swamp
  - If stuck → you've likely exhausted low-hanging fruit on that problem/approach
  - There's other low-hanging fruit elsewhere—go pick that instead

### 4. Low-Hanging Fruit Abundance
- The field is moving so quickly that there's **much more low-hanging fruit** than almost any other field
- Each new model capability = time to be opportunistic, explore what's now possible
- High velocity is useful for both:
  - Picking low-hanging fruit quickly
  - Getting through swamps when you must solve a particular problem

## Collaboration & Communication

### Strong Collaborator Qualities
Ethan puts **70% weight on "getting ideas to work quickly"** as criteria. Other qualities:
- Receptive to feedback
- Adds emotional energy rather than draining it
- Transparent/communicative about issues faced
- High-trust relationship where various topics can be discussed easily
- Notices and calls out room for improvement in collaboration

### Communication Best Practices
- **Overcommunicate**: bring up issues during meetings or privately—nip problems in the bud
- **Close mentorship** is maybe the fastest path to become an expert in a domain
- Take agency: organize coworking, discussion groups, standups—whatever helps
- Get feedback from peers, iterate on project plans

## Research Workflow

- Have a clear project plan with motivation and research goals
- List all experiments you can think of running
- Think about milestones and deliverables to stay accountable
- Know what tools are available—sharing tooling increases experimental velocity

## Paper Writing

- See [ethanperez.net/easy-paper-writing-tips/](https://ethanperez.net/easy-paper-writing-tips/) for ML paper writing tips
- Write short paragraphs composed of short sentences
- Write comprehensive abstracts
- Seek feedback from a naive audience
- Distinguish confirmation from exploration in your writeup

## Related Resources
- [Tips and Code for Empirical Research Workflows](https://www.alignmentforum.org/posts/6P8GYb4AjtPXx6LLB/tips-and-code-for-empirical-research-workflows)
- Jacob Steinhardt's blog on reducing uncertainty
- Michael Bernstein's slides on velocity