Site Reliability Engineering

1. SRE là gì?

2003, Ben Treynor Sloss được Google thuê để chạy production team. Câu hỏi: "làm sao reliable hệ thống khổng lồ với engineering rigor"? Câu trả lời: thuê software engineer làm operations.

Treynor's definition:

"SRE is what happens when you ask a software engineer to design an operations team."

1.1. Khác biệt với "sysadmin truyền thống"

• Sysadmin: manual fix, ticket-driven, kỹ năng OS/network.
• SRE: tự động hóa, software-driven, kỹ năng coding + system design.
• SRE viết code: monitoring tool, automation script, control plane.

1.2. SRE principles (từ Google SRE book)

1. Embrace risk — 100% reliability vô lý + đắt; chấp nhận failure budget.
2. Service Level Objectives — define rõ "đủ reliable".
3. Eliminate toil — manual work là enemy.
4. Monitor everything — observability first.
5. Automation — code thay vì repeat manual.
6. Release engineering — deploy là engineering discipline.
7. Simplicity — complex system = unreliable; reduce complexity.

1.3. Books tham khảo

3 cuốn miễn phí online (sre.google/books):

• Site Reliability Engineering (2016) — foundation.
• The Site Reliability Workbook (2018) — practical.
• Building Secure & Reliable Systems (2020) — security focus.

2. SRE vs DevOps — Phân biệt

Liz Fong-Jones: "Class SRE implements interface DevOps".

DevOps = philosophy/culture; SRE = specific implementation (Google's recipe).

2.1. Org models

• Embedded SRE — 1 SRE trong mỗi product team.
• Centralized SRE — team riêng, partner với product team theo SLA.
• Tools team — SRE build platform (xem chương 14).
• Hybrid — combine.

2.2. Khi nào cần SRE?

• Engineering org > 50 engineers.
• Production critical (downtime = lost revenue).
• Có 24/7 on-call.
• SLO formally defined.

Startup nhỏ: SRE-style practices (error budget, postmortem) hữu ích nhưng không cần team riêng.

3. SLO Deep Dive

3.1. Choose right SLI

SLI tốt:

• Đo user experience, không proxy metric (CPU usage không direct).
• Có thể tin cậy được — không phụ thuộc service đang đo.
• Aggregate-able — tính được %.

Common SLI patterns:

3.2. Set SLO target

Quy tắc: target dựa trên user expectation, không capability.

• Quá cao (99.999%) → đắt vận hành, dev cant ship feature.
• Quá thấp (99%) → user frustrated.
• Most B2B SaaS: 99.9% (43 phút downtime/tháng).
• Banking/healthcare: 99.99%.
• Ad serving (lossy): 99% có thể đủ.

3.3. Multi-window SLO

SLO không phải instant — tính rolling window (vd 28 ngày):

# Availability (28 day rolling)
sum(rate(http_requests_total{status=~"2..|3.."}[28d]))
  /
sum(rate(http_requests_total[28d]))

Window 28 ngày phổ biến vì tránh weekend skew (4 weekend = 28 ngày).

3.4. Composite SLO

Service nhiều endpoint → SLO chia theo importance:

• "Critical" endpoints (login, checkout): 99.99%.
• "Important" (browse): 99.9%.
• "Best effort" (recommendations): 99%.

3.5. SLO không phải SLA

SLA: cam kết external có legal consequence. SLO: nội bộ, target hơn cam kết.

SLO < SLA — buffer. Nếu SLA = 99.9%, SLO = 99.95%. Khi vi phạm SLO → engineering response. Khi vi phạm SLA → financial response (refund, credit).

4. Error Budget — Kỷ luật của SRE

4.1. Concept

SLO 99.9% = 0.1% downtime cho phép = "error budget".

4.2. Error Budget Policy

Document chính thức về cách team handle budget:

# Error Budget Policy — Service X

## SLO
99.9% availability over 28-day rolling window.

## Budget
0.1% × 28 days = 40.3 minutes downtime allowed.

## When budget > 50% remaining
- Normal velocity: deploy daily, experiment freely.
- Chaos engineering allowed.

## When budget 10-50% remaining
- Slow down deploys (max 1/day).
- Prioritize reliability work over features.
- Required postmortem for any prod issue.

## When budget < 10% remaining
- HARD FREEZE on feature deploy.
- Only fixes that improve reliability.
- All hands-on-deck for stabilization.

## When SLO breached
- Public incident review.
- Roadmap commitment for reliability fixes.
- Optional: SLO re-evaluation (was target right?).

4.3. Burn rate alert

# Multi-window multi-burn rate (Google SRE book)
groups:
  - name: slo-burn-rate
    rules:
      # Fast burn: 2% budget in 1h = 14.4× rate → page on-call
      - alert: ErrorBudgetFastBurn
        expr: |
          (
            (1 - sum(rate(http_requests_total{status=~"2..|3.."}[1h])) / sum(rate(http_requests_total[1h])))
            > (14.4 * 0.001)
          )
          and
          (
            (1 - sum(rate(http_requests_total{status=~"2..|3.."}[5m])) / sum(rate(http_requests_total[5m])))
            > (14.4 * 0.001)
          )
        labels:
          severity: critical

      # Slow burn: 5% budget in 6h = 6× rate → ticket
      - alert: ErrorBudgetSlowBurn
        expr: |
          ((1 - ...rate[6h]) > (6 * 0.001))
          and
          ((1 - ...rate[30m]) > (6 * 0.001))
        labels:
          severity: warning

Multi-burn rate: alert chỉ fire khi sustained over 2 windows (long + short) → reduce false positive.

5. Toil Reduction

5.1. Định nghĩa toil

Toil = việc manual, repetitive, tactical (no enduring value), scale linearly with service growth.

Examples:

• Manually restart Pod khi crash.
• Click console add user mới.
• SSH vào server clean log.
• Edit nhiều ticket request "deploy giùm".

Non-toil examples:

• Code review (engineering work).
• Postmortem analysis (creates value).
• Architecture design.

5.2. Tại sao reduce toil?

• Toil tăng tuyến tính với service growth → SRE team không scale.
• Toil = career stagnation cho SRE (no learning).
• Toil khiến SRE không có thời gian cho engineering work.
• Burnout → attrition.

Google target: SRE work < 50% toil. Thực tế nhiều team 70-80% toil → unsustainable.

5.3. Eliminating toil

1. Identify — track toil hours/sprint. Survey team monthly.
2. Categorize — what task? frequency?
3. Automate or eliminate:
   • Script repetitive task.
   • Self-service portal cho user request (ticket → button).
   • Auto-remediation (auto-restart, auto-scale).
   • Remove root cause (improve service quality).
4. Track reduction — % toil over time.

5.4. Auto-remediation patterns

# Pod auto-restart (K8s built-in)
restartPolicy: Always

# Liveness probe → restart unhealthy
livenessProbe:
  httpGet: { path: /alive, port: 3000 }

# HPA → scale on traffic
HorizontalPodAutoscaler ...

# Karpenter → spawn nodes when pod pending
# CronJob → daily backup

# Self-healing alerts (Alertmanager + webhook)
- If "DiskFull" alert → trigger Lambda to clean log
- If "Pod OOMKilled" → notify owner team (no on-call wake)

6. On-call Best Practices

6.1. Schedule

• Primary + Secondary rotation (Secondary backup nếu Primary miss).
• Weekly rotation phổ biến (không quá ngắn để kéo context, không quá dài để burnout).
• Follow-the-sun nếu có team multi-region (no 3 AM page).
• Mỗi shift ≤ 25% thời gian SRE (fairness).

6.2. Pager hygiene

• SLO-based alert — chỉ alert khi user impact, không alert preventive.
• Page budget — ≤ 2 page/shift (Google guideline). Nhiều hơn = alert spam → fix root cause.
• Postmortem mọi page (review weekly, eliminate noise).
• Compensation — pay/PTO cho on-call (tránh free labor).

6.3. On-call playbook

Khi nhận page:

1. Acknowledge trong 5 phút.
2. Open runbook — alert có link runbook? Follow.
3. Assess severity — user impact? scope?
4. Open incident channel (Slack #inc-2024-01-15-payment-down).
5. Mitigate first, fix later — rollback / scale up / drain traffic, không debug 30 phút khi user đang impact.
6. Communicate — status page, customer support team.
7. Resolve — verify metric trở lại bình thường.
8. Postmortem trong 48h.

6.4. On-call burnout — phòng ngừa

• Compensation rõ (extra pay, comp time).
• Page budget: nếu 2+ page/shift trong 4 tuần → reliability work priority.
• Rotate roles (không cùng 1 người mãi).
• Backup plan khi sick.
• "Operational excellence reviews" — quarterly review on-call burden.

7. Incident Management

7.1. Severity levels

7.2. ICS — Incident Command System

Mượn từ FEMA wildfire response. Roles:

• Incident Commander (IC) — quyết định toàn cục, không touch code; coordinator.
• Operations Lead (OL) — actual hands-on technical work.
• Communications Lead (CL) — update status page, customer comm, exec.
• Subject Matter Expert (SME) — domain expert pulled in.
• Scribe — log timeline cho postmortem.

1 person có thể đảm nhiều roles ở incident nhỏ. SEV-1: separate roles.

7.3. War room

Communication channel:

• Dedicated Slack channel #inc-YYYY-MM-DD-summary.
• Voice: Zoom / Google Meet open suốt incident.
• Status page: external customer comm.
• Incident dashboard: Grafana focused on key SLI.

7.4. Mitigate first, fix later

Trong incident, ưu tiên: stop bleeding, không understand why.

Mitigations:

• Rollback last deploy.
• Disable feature flag.
• Scale up replicas.
• Drain bad node.
• Failover to DR region.
• Throttle/rate limit traffic source gây overload.

Sau khi mitigate, có thời gian debug root cause → fix proper.

8. Postmortem — Blameless Learning

8.1. Mục đích

Postmortem KHÔNG để punish. Để learn:

• Điều gì xảy ra?
• Vì sao? (root cause + contributing factors).
• Làm sao prevent next time?
• Action items với owner + deadline.

8.2. Blameless culture

Giả định: mọi người đã làm tốt nhất với thông tin/tool có lúc đó. Câu hỏi đúng:

• ✓ "Hệ thống/quy trình thiếu gì khiến bug này có thể xảy ra?"
• ❌ "Ai đã làm sai?"

Blame culture → người che giấu bug → repeat.

8.3. Template

# Postmortem: 2024-01-15 Payment Service Outage

**Severity**: SEV-2
**Duration**: 14:23 - 15:47 UTC (84 minutes)
**Author**: Alice Lead
**Status**: Final

## Summary
Payment processing failed for 100% of users due to expired TLS certificate
on payment-gateway service. ~$120k revenue lost.

## Timeline (UTC)
- **14:23** — Alert "PaymentErrorRate > 50%" fires.
- **14:25** — On-call (Bob) acknowledges.
- **14:30** — Bob identifies certificate expiry in logs.
- **14:35** — IC declared (Carol).
- **14:42** — Cert renewal initiated. Failed: ACME challenge issue.
- **14:55** — Manual cert deploy attempted. Failed: ConfigMap permissions.
- **15:30** — Cert deployed via Vault.
- **15:35** — Service restored.
- **15:47** — Verified all metrics normal. Incident closed.

## Impact
- 100% payment requests failed for 84 minutes.
- ~$120k transaction revenue lost.
- 25k customer support tickets opened.
- Brand impact: trending #PaymentFail on Twitter.

## Root Cause
Cert auto-renewal cron failed silently 14 days ago because:
1. cert-manager's challenge solver lost permission to write
   to namespace "kube-system" after RBAC tighten 2024-01-01.
2. cert-manager log → ERROR but no alert configured for cert-manager.
3. Cert expired 14:23.

## What Went Well
- Alert fired in < 1 min of impact.
- IC role assigned quickly.
- Customer support communicated proactively.

## What Went Poorly
- No alert for cert expiry approaching (could've prevented).
- cert-manager errors not in main observability.
- Manual cert deploy procedure undocumented.

## Action Items
| # | Action                                              | Owner   | Due       | Status |
|---|-----------------------------------------------------|---------|-----------|--------|
| 1 | Add alert "CertExpiryWithin30Days"                  | Bob     | 2024-01-22| Open   |
| 2 | Include cert-manager logs in main alert pipeline    | Carol   | 2024-01-29| Open   |
| 3 | Document manual cert renewal runbook                | Alice   | 2024-01-22| Open   |
| 4 | Audit RBAC changes process — require cert-manager test | Dave | 2024-02-15| Open   |
| 5 | Review all cron jobs for silent failures            | SRE team| 2024-02-29| Open   |

## Lessons Learned
- Auto-renewal != monitored renewal. Must alert on expiry approaching, not on expiry itself.
- RBAC changes need integration test for downstream services.
- cert-manager errors went unnoticed for 14 days — observability gap.

8.4. Action items

Critical: action items không phải wishlist:

• Specific owner.
• Deadline (vài tuần, không "sometime").
• Tracked → completed.
• Public review (quarterly: action items resolved %?).

Anti-pattern: postmortem viết hay, action items nằm rotting trong backlog. Engineering manager track completion.

8.5. Postmortem culture

• Mọi SEV-1, SEV-2 phải có postmortem (default).
• SEV-3, SEV-4 optional (nhưng good practice).
• Public — share toàn engineering org để learn.
• Read past postmortem hằng tuần (10 phút standup).
• Tooling: Jeli (acquired by Atlassian), Incident.io, BlamelessOps.

9. Runbook

9.1. Định nghĩa

Runbook = step-by-step guide để xử lý 1 alert/scenario. Goal: bất kỳ on-call nào (kể cả mới onboarding) handle được.

9.2. Format

# Runbook: HighErrorRate

## Alert
Error rate > 5% for 5 minutes on service `payment`.

## Severity
SEV-2 if duration < 15min; SEV-1 if longer.

## Impact
Users see "Payment failed" in checkout flow.

## Diagnosis Steps

### 1. Check Grafana
- Open: https://grafana.example.com/d/payment-overview
- Look at: Error breakdown panel (5xx by route, 4xx by route)
- Note: Are errors concentrated to 1 route, 1 instance, or distributed?

### 2. Check recent changes
- Open: https://github.com/myorg/payment/commits/main
- Any deploy in last 30 min? → likely cause, jump to "Mitigation: Rollback"

### 3. Check downstream
- DB latency: https://grafana.example.com/d/postgres-overview
- Stripe API status: https://status.stripe.com
- Internal auth service: https://grafana.example.com/d/auth-overview

### 4. Check pod status
```bash
kubectl get pods -n payment
kubectl logs -n payment -l app=payment --tail=200 | grep ERROR
```

## Mitigation Options

### A. Rollback (recent deploy = root cause)
```bash
kubectl rollout undo deployment/payment -n payment
kubectl rollout status deployment/payment -n payment
```
Verify: error rate drop within 2 min.

### B. Scale up (CPU/memory bound)
```bash
kubectl scale deployment/payment -n payment --replicas=20
```

### C. Disable feature flag (specific feature broken)
```bash
launchdarkly set-flag new-checkout-flow off
```

### D. Failover to DR region
See: [Multi-region Failover Runbook](./failover.md)

## After Mitigation
1. Open incident channel `#inc-YYYY-MM-DD-payment-error-rate`
2. Notify customer support team (#cs-leads)
3. Update status page: status.example.com
4. Schedule postmortem within 48h.

## Common Causes (Past)
- Stripe API rate limit: 2024-01-15 (see PM-2024-001)
- DB connection pool exhausted: 2023-11-22 (see PM-2023-019)
- Bad deploy with timeout regression: 2023-09-04 (see PM-2023-014)

## Escalation
- IC: page Carol (Engineering Manager)
- DBA: page Dave (DB on-call)
- Stripe: support@stripe.com, account #abc-123

9.3. Best practices

• Linked from alert — alert annotation có URL runbook.
• Versioned trong Git — không Wiki page rot.
• Test runbook — game day exercise (xem chaos).
• Update sau postmortem — incident mới → enrich runbook.
• Less is more — runbook 100 trang không ai đọc; focus action items.

10. Chaos Engineering

10.1. Concept

Chaos Engineering (Netflix 2010+): chủ động inject failure vào production để verify hệ thống resilient. Better hơn đợi failure thật.

Principles (principlesofchaos.org):

1. Define "steady state" (normal metric).
2. Hypothesize: steady state sẽ persist khi inject failure.
3. Run experiment với real-world failures (terminate instance, network latency).
4. Run in production (with safeguards).
5. Minimize blast radius.

10.2. Chaos Monkey (Netflix)

Tool đầu tiên (2011): random terminate EC2 instance trong production. Forced developers viết app handle failure gracefully.

Suite "Simian Army":

• Chaos Monkey — kill instance.
• Chaos Kong — kill entire region.
• Latency Monkey — inject latency.
• Conformity Monkey — enforce best practice.

10.3. Modern tools

• Chaos Mesh (CNCF) — K8s-native chaos.
• LitmusChaos (CNCF) — K8s chaos workflows.
• Gremlin — SaaS, GUI.
• AWS Fault Injection Simulator (FIS).
• Pumba — Docker chaos.

10.4. Chaos Mesh example

apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-failure-test
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: payment
  scheduler:
    cron: "@every 1h"

---
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: network-delay
spec:
  action: delay
  mode: all
  selector:
    namespaces: [production]
    labelSelectors:
      app: payment
  delay:
    latency: "100ms"
    correlation: "100"
    jitter: "10ms"
  duration: "5m"

10.5. Game days

Practice exercise: team simulate incident, follow runbook. Ví dụ:

• Nick (random IC) "It's 3am Saturday. PagerDuty wakes you up — payment is down."
• On-call follow runbook real-time, team observe.
• After: review what worked, what didn't.

Phương pháp Google "DiRT" (Disaster Recovery Testing) — yearly large-scale game day.

10.6. Practice safely

• Start in staging, not prod.
• Have abort button.
• Schedule (not random) for major experiment.
• Monitor blast radius.
• Communicate to teams (no surprise outage).
• Steady state = clear metric (revenue/min, RPS).

11. Capacity Planning

11.1. Why?

Cloud scale tự động OK, nhưng:

• Reserved instance discount cần cam kết → cần dự đoán.
• Database không scale instant.
• Cost — overprovisioning = waste, underprovisioning = outage.
• Region/AZ capacity (cloud có rate limit launch instance).

11.2. Process

1. Forecast demand — historical growth, business plan, marketing campaign.
2. Identify resource needs — CPU/RAM/disk/network per unit demand.
3. Plan headroom — không vận hành > 70% (buffer cho burst, GC).
4. Reserved/Spot mix — baseline reserved, peak spot/on-demand.
5. Quarterly review — actual vs forecast, adjust.

11.3. Tools

• k6 / Locust — load test.
• AWS Compute Optimizer — recommend instance size.
• GCP Recommender.
• Vertical Pod Autoscaler recommend mode — K8s recommend resource.
• Spreadsheet — surprisingly, simple growth model often enough.

11.4. Example forecast

Service: payment-api
Current: 5,000 RPS peak, 20 pods @ 0.5 CPU avg
Growth: 30%/year (historical)

Projected (12 months):
  RPS = 5,000 × 1.30 = 6,500 peak
  Pods = 6,500/5,000 × 20 × 1.4 (headroom) = 36 pods
  CPU = 36 × 0.5 = 18 cores
  Memory @ 256MB/pod = 9 GB

Sale event spike: 3× normal = ~108 pods burst.
  → Need cluster capacity 150 pods (with safety).
  → Karpenter spot for burst, reserved for baseline 36 pods.

DB:
  Currently: db.r5.xlarge (16 GB RAM, 4 vCPU, 60% utilization).
  Projected: 78% → upgrade to db.r5.2xlarge by Q3.

12. Bài tập

1. Define SLO: cho 1 service bạn maintain. Define SLI (availability + latency). Set SLO target. Tính error budget 28 ngày.
2. Burn rate alert: implement multi-window multi-burn rate alert. Test bằng chaos — kill pods, watch fast burn fire.
3. Toil audit: trong 1 tuần, log mọi task đã làm. Phân loại engineering vs toil. Tính %. Đề xuất 3 automation cao priority.
4. Auto-remediation: implement 1 auto-remediation: alert "DiskFullOnNode" → trigger Lambda clean log /tmp + /var/log/old.
5. Runbook: viết runbook cho alert "DatabaseConnectionPoolExhausted". Include diagnosis + 3 mitigation options.
6. Mock incident: với 1 đồng đội, simulate incident. Bạn IC; họ inject failure (kill pod, fill disk). Theo runbook handle. Time it.
7. Postmortem: viết postmortem cho 1 incident gần đây (nếu chưa có). Use template above. Action items có owner + deadline.
8. Chaos Mesh: cài Chaos Mesh trên cluster. Schedule pod-kill mỗi giờ. Monitor service vẫn meet SLO.
9. Game day: tổ chức 1h game day với team. Scenario: "DB primary down". On-call follow runbook. Document gaps.
10. Capacity planning: cho service hiện tại, dự báo capacity 12 tháng. Reserved vs Spot ratio. Submit cho manager.
11. On-call experience: nếu chưa, volunteer 1 week on-call. Record số page, time-to-mitigate. Identify alert noise.
12. SLO review: tính SLO compliance 28 ngày qua. Vi phạm? Action items?

13. Quiz