---
name: sre-practices
description: 'SLIs/SLOs/SLAs, error budgets, incident response, postmortems, and reliability patterns'
metadata:
  author: cosmicstack-labs
  version: 1.0.0
  category: devops
  tags: [sre, reliability, slo, incident-response, postmortem]
---

# SRE Practices

Apply Site Reliability Engineering principles.

## Service Level Concepts

| Term | Definition | Example |
|------|------------|---------|
| SLI | Measured metric | Request latency p95 < 500ms |
| SLO | Target threshold for SLI | 99.9% of requests < 500ms |
| SLA | Contractual commitment (usually looser than SLO) | 99.5% uptime |

### Choosing SLOs
- Pick metrics users actually care about
- Start with availability + latency + durability
- Tighten SLOs over time as reliability improves
- Don't over-constrain (cost/effort vs. benefit)

## Error Budgets

```
Error Budget = 100% - SLO
Example: 99.9% SLO → 0.1% error budget = ~8.7 hours/month
```

### How Error Budgets Work
- If error budget remaining → can deploy new features
- If error budget exhausted → freeze deployments, focus on reliability
- Error budget burn rate alerts trigger incident response
- Balance innovation velocity with system stability

## Incident Response

### Severity Levels
| Level | Definition | Response |
|-------|-----------|----------|
| SEV1 | System down, affecting many users | Immediate, all hands |
| SEV2 | Degraded but operational | 30min response |
| SEV3 | Minor issue, workaround exists | Next business day |
| SEV4 | Cosmetic, non-critical | Next sprint |

### Incident Command System
- **Incident Commander**: Coordinates response
- **Communications Lead**: Status updates, stakeholder comms
- **Operations Lead**: Technical investigation
- **Scribe**: Timeline and action log

## Postmortem Culture
- Blameless: systems failed, not people
- Focus on: detection, response, prevention
- Action items with owners and due dates
- Share postmortems org-wide
- Track action items to completion

## Reliability Patterns
- Circuit breaker: stop cascading failures
- Bulkhead: isolate failure domains
- Retry with exponential backoff + jitter
- Rate limiting: protect against traffic spikes
- Graceful degradation: degrade features, not the whole system