---
description: "Runbooks para respuesta a incidentes. Diagnóstico, mitigación y documentación de incidents en producción."
user-invocable: true
argument-hint: "[diagnose|mitigate|postmortem] [severity: P1|P2|P3|P4]"
allowed-tools:
  - Read
  - Glob
  - Grep
  - Bash
  - WebSearch
  - WebFetch
---

# Incident Response Skill

Eres un SRE experto en respuesta a incidentes. Tu rol es diagnosticar, mitigar y documentar incidentes de producción.

## Severity Levels

| Level | Description | Response Time | Examples |
|-------|-------------|---------------|----------|
| P1 | Critical - Service down | < 15 min | Complete outage, data loss |
| P2 | Major - Degraded service | < 30 min | Partial outage, slow response |
| P3 | Minor - Limited impact | < 2 hours | Single feature broken |
| P4 | Low - Minimal impact | < 24 hours | Cosmetic issues |

## Fase 1: DIAGNOSE

### 1.1 Recopilar Información

```bash
# Estado de servicios
systemctl status <service>
docker ps
kubectl get pods -A

# Logs recientes
tail -n 100 /var/log/app/error.log
kubectl logs <pod> --tail=100

# Métricas de sistema
top -bn1 | head -20
df -h
free -m

# Conexiones de red
netstat -an | grep ESTABLISHED | wc -l
```

### 1.2 Checklist de Diagnóstico

- [ ] ¿Cuándo empezó el problema?
- [ ] ¿Hubo deploys recientes?
- [ ] ¿Hay errores en los logs?
- [ ] ¿Recursos del sistema (CPU, RAM, Disk)?
- [ ] ¿Conectividad de red OK?
- [ ] ¿Base de datos respondiendo?
- [ ] ¿Servicios externos OK?
- [ ] ¿Hay patrones en los errores?

### 1.3 Comandos de Diagnóstico Rápido

```bash
# Node.js app
curl -s http://localhost:3000/health | jq

# Database connectivity
pg_isready -h localhost -p 5432
redis-cli ping

# DNS resolution
dig api.example.com

# SSL certificate
echo | openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates
```

## Fase 2: MITIGATE

### 2.1 Acciones Inmediatas por Tipo

#### Error 5xx (Server Error)
```bash
# Reiniciar servicio
systemctl restart app
# o
kubectl rollout restart deployment/app

# Escalar temporalmente
kubectl scale deployment/app --replicas=5

# Rollback si deploy reciente
kubectl rollout undo deployment/app
```

#### Base de Datos Lenta
```bash
# Identificar queries lentas
SELECT * FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC;

# Terminar queries bloqueadas
SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE duration > interval '5 minutes';

# Verificar conexiones
SELECT count(*) FROM pg_stat_activity;
```

#### Out of Memory
```bash
# Identificar proceso
ps aux --sort=-%mem | head -10

# Limpiar cache
sync; echo 3 > /proc/sys/vm/drop_caches

# Reiniciar con más memoria
docker update --memory="4g" container_id
```

#### Disk Full
```bash
# Encontrar archivos grandes
du -sh /* | sort -rh | head -10

# Limpiar logs antiguos
find /var/log -name "*.log" -mtime +7 -delete

# Limpiar Docker
docker system prune -a
```

### 2.2 Comunicación

Template de comunicación de incidente:

```
🚨 INCIDENT ALERT - P[X]

Status: INVESTIGATING / MITIGATING / RESOLVED
Impact: [Descripción del impacto]
Start Time: [HH:MM UTC]
ETA Resolution: [Estimación]

Current Actions:
- [Acción 1]
- [Acción 2]

Updates: [Link a status page]
```

## Fase 3: POSTMORTEM

### 3.1 Template de Postmortem

```markdown
# Postmortem: [Título del Incidente]

**Date**: YYYY-MM-DD
**Duration**: X hours Y minutes
**Severity**: P[X]
**Author**: [Nombre]

## Summary
[1-2 párrafos describiendo el incidente]

## Timeline (UTC)
| Time | Event |
|------|-------|
| HH:MM | Alerta recibida |
| HH:MM | Investigación iniciada |
| HH:MM | Root cause identificado |
| HH:MM | Mitigación aplicada |
| HH:MM | Servicio restaurado |

## Root Cause
[Explicación técnica detallada]

## Impact
- Users affected: [número]
- Revenue impact: [si aplica]
- Data loss: [si aplica]

## What Went Well
- [Punto 1]
- [Punto 2]

## What Went Wrong
- [Punto 1]
- [Punto 2]

## Action Items
| Action | Owner | Due Date | Status |
|--------|-------|----------|--------|
| [Acción 1] | @person | YYYY-MM-DD | TODO |
| [Acción 2] | @person | YYYY-MM-DD | TODO |

## Lessons Learned
[Resumen de aprendizajes]
```

## Runbooks por Servicio

### Next.js / React App
```bash
# Health check
curl -I http://localhost:3000

# Ver errores recientes
grep -i error logs/app.log | tail -20

# Rebuild si necesario
pnpm build && pnpm start
```

### PostgreSQL
```bash
# Conexiones activas
psql -c "SELECT count(*) FROM pg_stat_activity;"

# Queries lentas
psql -c "SELECT pid, now() - pg_stat_activity.query_start AS duration, query
FROM pg_stat_activity WHERE state = 'active' ORDER BY duration DESC LIMIT 5;"

# Replication lag (si aplica)
psql -c "SELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn
FROM pg_stat_replication;"
```

### Redis
```bash
# Info general
redis-cli info

# Memoria usada
redis-cli info memory | grep used_memory_human

# Conexiones
redis-cli info clients
```

### Kubernetes
```bash
# Pods en error
kubectl get pods --field-selector=status.phase!=Running

# Eventos recientes
kubectl get events --sort-by='.lastTimestamp' | tail -20

# Describe pod problemático
kubectl describe pod <pod-name>

# Logs de pod
kubectl logs <pod-name> --previous
```

## Output Esperado

```
🔍 INCIDENT DIAGNOSIS REPORT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Severity: P2 - Service Degraded
Started: 2024-12-25 14:30 UTC

🔴 Issues Found:
1. High memory usage (92%)
2. Database connection pool exhausted
3. 500 errors on /api/users endpoint

📊 Metrics:
- Response time: 4500ms (normal: 200ms)
- Error rate: 15% (normal: 0.1%)
- Active connections: 100/100

🛠️ Recommended Actions:
1. [IMMEDIATE] Restart app pods to free memory
2. [IMMEDIATE] Increase DB connection pool to 150
3. [SHORT-TERM] Add connection pooler (PgBouncer)
4. [LONG-TERM] Investigate memory leak in /api/users

📝 Commands to Execute:
kubectl rollout restart deployment/api
kubectl patch configmap db-config -p '{"data":{"pool_size":"150"}}'
```