---
name: debug-openclaw
description: >
  SSH into openclaw and figure out what's going on. Use when: (1) user says
  "debug openclaw", "openclaw is broken", "something's wrong with openclaw",
  (2) agents aren't responding, (3) gateway issues suspected, (4) BlueBubbles
  messages not delivering, (5) any openclaw operational issue.
allowed-tools: Bash, Read, Grep
---

# Debug OpenClaw

SSH into openclaw and systematically figure out what the fuck is going on.

## Triage Order

Run these in sequence. Stop and investigate when you find something wrong.

### 1. Gateway Health

```bash
ssh openclaw 'systemctl --user status openclaw-gateway'
```

Check: Is it active? Crash-looping? OOM-killed? Note the memory usage and uptime.

### 2. Live Logs

```bash
ssh openclaw 'journalctl --user -u openclaw-gateway --no-pager -n 100'
```

Look for: errors, stack traces, restart loops, OOM kills, connection failures.

### 3. Resource Pressure

```bash
ssh openclaw 'free -h && echo "---" && df -h / && echo "---" && ps aux --sort=-%mem | head -15'
```

The machine has 7.6G RAM. Gateway settles ~2.5G, peaks ~4G during boot. MemoryMax is 5G.

### 4. Orphan Processes

```bash
ssh openclaw 'systemctl --user status openclaw-gateway | grep -A 50 "CGroup"'
```

If there are dozens of child processes, orphans accumulated (see memory note below).

### 5. Agent Sessions

```bash
ssh openclaw 'cd /opt/openclaw && ./bin/openclaw sessions 2>/dev/null || node dist/cli.js sessions'
```

### 6. BlueBubbles (if message delivery issues)

```bash
ssh openclaw 'curl -sf http://100.120.154.29:1235/api/v1/ping -H "Authorization: Bearer \$(grep BLUEBUBBLES_PASSWORD_KARL ~/.openclaw/.env | cut -d= -f2)" | jq .'
```

Port 1235 = Karl BB instance (lux.nebulatio@icloud.com). Port 1234 = Cole's personal (decommissioned from OC).

### 7. Agent Provisioner (if new signups broken)

```bash
ssh openclaw 'systemctl --user status agent-provisioner && curl -sf http://localhost:3500/health'
```

### 8. Config Sanity

```bash
ssh openclaw 'cd /opt/openclaw && ./bin/openclaw doctor 2>/dev/null || echo "doctor not available"'
```

**WARNING:** `openclaw doctor` can resolve `${VAR}` to plaintext in config (bug #9627). Verify config after running.

## Key Paths

| What | Path |
|------|------|
| Install | `/opt/openclaw/` |
| State | `~/.openclaw/` |
| Config | `~/.openclaw/openclaw.json` |
| Secrets | `~/.openclaw/.env` |
| Service | `~/.config/systemd/user/openclaw-gateway.service` |
| Logs | `journalctl --user -u openclaw-gateway` |
| Workspaces | `~/.openclaw/workspace-*` |

## Common Fixes

**Gateway won't start / crash-looping:**
```bash
ssh openclaw 'systemctl --user restart openclaw-gateway && sleep 5 && systemctl --user status openclaw-gateway'
```

**Orphan processes eating RAM (KillMode=mixed should prevent, but just in case):**
```bash
ssh openclaw 'systemctl --user stop openclaw-gateway && sleep 5 && pkill -f "openclaw|qmd" && systemctl --user start openclaw-gateway'
```

**Disk full:**
```bash
ssh openclaw 'du -sh ~/.openclaw/workspace-*/sessions/ 2>/dev/null | sort -rh | head -10'
```

Session transcripts are the usual suspect.

## After Fixing

1. Verify gateway is healthy: `systemctl --user status openclaw-gateway`
2. Tail logs for 30s to confirm no new errors
3. Test an agent response if possible
4. Report findings to the user