Production Mastery

1. High Availability — HA Patterns

1.1. Định nghĩa HA

HA = system tiếp tục operate khi component fail. Đo bằng availability percentage:

1.2. Availability math

Sequential components (A → B → C): tích availability từng cái.

• Mỗi cái 99.9% × 3 = 99.7% — giảm.

Parallel (redundant): 1 - (1 - a)^n.

• 2 instance 99.9% song song = 99.9999% — 6 nines!

→ HA = redundancy + remove SPOF.

1.3. SPOF — Single Points of Failure

Audit checklist:

• Web tier: ≥ 2 instance behind LB?
• App tier: stateless + multiple replica?
• DB: primary + replica? Multi-AZ?
• Cache: Redis cluster + Sentinel?
• Queue: Kafka multi-broker?
• Load balancer: cloud-managed (multi-AZ inherent)?
• DNS: multiple authoritative nameservers?
• External dependency: timeout + circuit breaker?
• Single AZ deployment? → multi-AZ.
• Single region? → multi-region cho mission critical.

1.4. Failure modes

2. Multi-region Architecture

2.1. Active-Passive (Cold/Warm DR)

• Cost: low (B mostly idle).
• RTO: minutes (manual or automated).
• RPO: seconds (replication lag).

2.2. Active-Active (Hot)

• Cost: high (full capacity 2 region).
• RTO: seconds (DNS/LB drain bad region).
• RPO: ~0 (sync replication possible).
• Complexity: cao (data consistency cross-region).

2.3. Pilot Light

Region B có core infrastructure (DB) chạy nhưng app tier idle. Failover spin up apps.

• RTO: 10s of minutes.
• RPO: seconds.
• Cost: medium.

2.4. Backup & Restore

Region B chỉ có backup. Failover restore from snapshot.

• RTO: hours.
• RPO: depend on backup frequency.
• Cost: lowest.

2.5. Routing

• Active-Active: Route53 latency-based / geo routing → user closest region.
• Active-Passive: Route53 health check → primary fail → DNS switch B (TTL low cho fast cutover).
• Anycast IP (Cloudflare, AWS Global Accelerator) — single IP route closest data center.

2.6. Data consistency challenges

• Replication lag — async replica behind primary by ms-s.
• Split-brain — network partition, both region accept write → conflict.
• Distributed transactions — cross-region transaction expensive.
• Geographic regulation — GDPR data sovereignty (EU data stay EU).

Tools handle: CockroachDB, Spanner (GCP) — distributed SQL global. DynamoDB Global Table — multi-region NoSQL với last-write-wins.

Pattern: most company chọn multi-AZ within single region + multi-region active-passive với cold DB replica. True active-active ít — complexity quá đắt.

3. Disaster Recovery

3.1. RTO vs RPO

Examples:

• Banking: RTO 15 min, RPO 0 (no data loss).
• SaaS B2B: RTO 1h, RPO 5 min.
• Marketing site: RTO 4h, RPO 1 day (daily backup).

3.2. RTO/RPO ↔ DR strategy

3.3. Recovery testing — DiRT

Disaster Recovery Testing (Google name): periodically simulate disaster để verify procedure works.

• Yearly: full region failover.
• Quarterly: critical service failover.
• Monthly: backup restore test.

Quy tắc: untested backup = no backup. Khôi phục lần đầu thường fail (cert expired, dependency missing, runbook outdated).

3.4. Runbook examples

# DR Runbook: Region us-east-1 outage

## Detection
- AWS Health Dashboard reports outage
- Internal monitoring shows 100% requests failing in us-east-1
- Cross-region monitoring (Route53 health check) confirms

## Decision
- IC declares DR mode
- Communicate to: exec, customer support, status page

## Failover Steps (RTO target: 15 min)

1. Enable us-west-2 traffic via Route53:
   ```
   aws route53 change-resource-record-sets --change-batch file://failover.json
   ```

2. Promote us-west-2 RDS replica to primary:
   ```
   aws rds promote-read-replica --db-instance-identifier prod-db-uswest2
   ```

3. Scale up us-west-2 app deployments:
   ```
   kubectl --context=uswest2 scale deployment/web --replicas=20
   ```

4. Verify health:
   - Check status page metric
   - Smoke test critical endpoints
   - Customer support sample tickets

5. Update status page to "Degraded — failover complete"

## Failback (when us-east-1 returns)
[Detailed steps... ensure data sync direction reversed]

4. Backup Strategy

4.1. 3-2-1 Rule

• 3 copies of data.
• 2 different storage media.
• 1 off-site (cross-region or different cloud).

4.2. Backup types

• Full — toàn bộ data. Slow + expensive.
• Incremental — chỉ delta từ backup trước. Fast nhưng restore phải apply tất cả.
• Differential — delta từ full backup gần nhất. Compromise.
• Snapshot — point-in-time copy (cloud disk). Cheap, fast.
• Continuous (CDC) — every change stream to backup. RPO ~0.

4.3. PITR — Point-In-Time Recovery

Modern managed DB (RDS, Cloud SQL) support PITR — restore tới timestamp cụ thể trong retention window. Combine: daily snapshot + WAL/binlog continuous archive.

Example: prod DB corrupt at 14:30 (bad migration). Restore tới 14:25 = lose 5 minute data, không 24h.

4.4. Backup automation

# K8s CronJob backup
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: backup
              image: postgres:15
              command:
                - sh
                - -c
                - |
                  pg_dump -h $DB_HOST -U $DB_USER $DB_NAME | \
                  gzip | \
                  aws s3 cp - s3://backups/$(date +%Y%m%d).sql.gz \
                    --storage-class STANDARD_IA \
                    --sse aws:kms
              env:
                - name: PGPASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: db-secret
                      key: password
          restartPolicy: OnFailure

4.5. Backup security

• Encrypt backup (KMS).
• Cross-region replication.
• Immutable storage (S3 Object Lock) — ransomware can't delete.
• Separate AWS account cho backup (compromised primary account không destroy backup).
• Audit access to backup (CloudTrail).

4.6. Restore drill

Quy tắc số 1: untested backup = no backup. Quarterly: pick random backup, restore to staging, verify data integrity.

Common restore failures:

• Backup file corrupt (silent for years).
• Schema mismatch (migration after backup).
• Missing dependency (extension, function).
• KMS key rotated, can't decrypt.
• Permission missing.

5. Deployment Strategies — Deep

5.1. Recap (chương 4)

• Recreate — downtime, simple.
• Rolling update — no downtime, default K8s.
• Blue-Green — instant switch, 2× resource.
• Canary — gradual rollout, low risk.

5.2. Canary với Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web
spec:
  replicas: 10
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: web
          image: ghcr.io/me/web:1.5.0
  strategy:
    canary:
      canaryService: web-canary       # Service trỏ canary pods
      stableService: web-stable        # Service trỏ stable pods
      trafficRouting:
        istio:
          virtualService:
            name: web-vs
            routes: [primary]

      steps:
        - setWeight: 5                 # 5% traffic to canary
        - pause: { duration: 10m }
        - analysis:
            templates:
              - templateName: success-rate
            args:
              - name: service-name
                value: web-canary
        - setWeight: 25
        - pause: { duration: 10m }
        - analysis:
            templates: [{ templateName: success-rate }]
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100               # full rollout

---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 30s
      successCondition: result[0] >= 0.99
      failureLimit: 3
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}",status!~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[5m]))

Argo Rollouts auto-promote nếu metric tốt; auto-rollback nếu fail.

5.3. Flagger (Flux)

Alternative cho Argo Rollouts. Tích hợp Flux + Prometheus + Istio/Linkerd.

5.4. Database migration during deploy

Schema thay đổi cùng deploy = nguy hiểm. Pattern expand-contract:

1. Expand: thêm column mới (nullable). Code v1 vẫn chạy (ignore column mới).
2. Migrate: backfill data cho column mới.
3. Deploy code v2: write cả old + new column (dual write).
4. Backfill complete.
5. Deploy code v3: read new column only.
6. Contract: drop old column.

Mỗi step backward-compatible. Có thể rollback giữa step. Avoid "schema thay đổi cùng app deploy" antipattern.

5.5. Zero-downtime deploy checklist

• App stateless (state ở DB/cache).
• Health check + readiness probe đúng.
• Graceful shutdown (handle SIGTERM, drain connection).
• preStop hook trong K8s — drain trước shutdown.
• Database migration backward-compat.
• API backward-compat (versioning, hoặc additive only).
• Connection draining trên LB.
• Resource budget cho 2× pod tạm thời (rolling).

6. Feature Flags — Decouple Deploy & Release

6.1. Concept

Deploy = code lên prod. Release = code accessible cho users. Feature flag tách 2:

if (await flags.isEnabled('new-checkout', { userId, country })) {
  return renderNewCheckout();
} else {
  return renderOldCheckout();
}

Flag controls runtime — không cần redeploy để bật/tắt.

6.2. Flag types

6.3. Targeting rules

{
  "name": "new-checkout",
  "rules": [
    {
      "if": { "userId": { "in": ["alice", "bob"] } },
      "value": true
    },
    {
      "if": { "country": "US", "userType": "beta_tester" },
      "value": true
    },
    {
      "rollout": {
        "percentage": 10,
        "byAttribute": "userId"
      },
      "value": true
    }
  ],
  "default": false
}

6.4. Tools

• LaunchDarkly — popular SaaS.
• Statsig — A/B + flags + analytics.
• Split.io — feature flag + experiment.
• Unleash — open source.
• Flagsmith — open source / SaaS.
• OpenFeature — vendor-neutral standard (CNCF).

6.5. Flag lifecycle hygiene

Flag debt: flag không bao giờ được clean up → code branches forever.

Best practice:

• Tag flag với owner + planned removal date.
• Quarterly review: flag > 6 tháng = cleanup.
• "Stale flag" alert.
• Tools support flag lifecycle (LaunchDarkly Code References).

7. Progressive Delivery

Progressive Delivery (James Governor 2018): combine canary + feature flag + observability + auto rollback.

7.1. Pattern

1. Deploy v2 với flag new-feature tắt 100% — code chạy nhưng không user nào thấy.
2. Bật flag cho 1% internal users (employee).
3. Bật cho 1% beta tester (opt-in).
4. Bật cho 5% general users (canary). Monitor SLI.
5. Bật 25%, 50%, 100% gradually.
6. Remove flag + old code.

7.2. Auto-promote / Auto-rollback

Define metric SLO cho rollout. Tools (Flagger, Argo Rollouts) auto:

• Tăng % nếu metric OK.
• Pause hoặc rollback nếu metric vi phạm.

7.3. Beyond canary: Dark launches

Dark launch: deploy code, gọi API mới, nhưng không return result cho user. Mục đích: load test với real traffic shape.

async function search(query: string) {
  const oldResult = await searchV1(query);

  // Dark launch new search engine
  if (await flags.isEnabled('search-shadow')) {
    searchV2(query).catch(err => logger.warn('shadow err', err));
    // Don't await, don't use result
  }

  return oldResult;
}

8. A/B Testing

8.1. Vì sao A/B test?

Believe vs measure. Designer "tin" button đỏ tốt hơn xanh? Test thực tế.

Common metrics: click-through rate, conversion, revenue, retention.

8.2. Statistical foundation

Cần đủ sample size cho statistical significance:

• Effect size — minimum detectable difference (vd 1% conversion lift).
• Power — 80% standard.
• Significance level (α) — 5% standard (p < 0.05).

Sample calculator: evanmiller.org.

Quy tắc: 1% lift detect cần ~10k user/variant.

8.3. Common pitfalls

• Peeking — kiểm tra kết quả mid-test → false positive.
• Sample ratio mismatch (SRM) — nếu A/B không 50/50 thực tế → bias.
• Novelty effect — feature mới có boost ngắn hạn.
• Multiple testing — test nhiều variant đồng thời → tăng false positive (Bonferroni correction).
• Network effects — variant ảnh hưởng nhau (social network).

8.4. Tools

• Statsig, Eppo (modern stats).
• Optimizely (enterprise).
• LaunchDarkly Experiments.
• Google Optimize (deprecated 2023, GA4 has).
• VWO.

8.5. Beyond A/B: Bandit algorithms

Multi-armed bandit dynamically allocate traffic tới winning variant trong test → tốt cho cost-sensitive (minimize regret).

Use case: recommendation, ranking, ad. Pure A/B vẫn standard cho strategic decision.

9. Platform Engineering

9.1. Vì sao Platform Engineering?

Modern app stack quá phức tạp: K8s + Terraform + 20 cloud services + 10 SaaS tools. Mỗi dev không thể master tất cả → friction.

Solution: Platform team build internal abstraction layer cho dev tự service:

• Self-service deploy.
• Self-service infra provision.
• Standard observability built-in.
• Compliance built-in.
• Golden paths.

9.2. Team Topologies (Skelton 2019)

4 fundamental team types:

Platform team treats stream-aligned teams như customers — collect feedback, ship features, measure satisfaction (Developer Experience).

9.3. "Platform as a Product"

• PM cho platform team.
• Roadmap based on dev pain points.
• Adoption metrics (% teams onboarded).
• SLO cho platform itself (uptime, support response time).
• Documentation as priority.
• Office hours / Slack support.

9.4. Golden Path

"Cách đúng" được khuyến nghị mặc định. Vd:

• Tạo new service: template Backstage → CI/CD + monitoring + alerting + logging tự setup.
• Add new microservice: 1 PR change vs 10 PRs setup.

Dev có thể đi off-path nếu cần, nhưng default = paved road, fast.

9.5. Platform Engineering vs DevOps vs SRE

Sam Newman: "Platform Engineering is what happens when you industrialize DevOps." Platform team build tool để stream-aligned teams tự deploy / monitor / scale.

10. Internal Developer Platform (IDP)

10.1. Components

• Developer Portal — UI for self-service (Backstage).
• Service Catalog — metadata mọi service (owner, tier, dependencies).
• Software Templates — scaffold new service.
• Tech Docs — markdown auto-render.
• Pipelines — CI/CD integrated.
• Observability portal — link to dashboards/logs/traces.
• Cost / FinOps portal.
• Compliance portal.

10.2. Backstage

Backstage (Spotify 2020, CNCF incubating) là IDP framework phổ biến nhất.

# catalog-info.yaml — file mỗi service
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  description: Handles all payment processing
  annotations:
    github.com/project-slug: myorg/payment-service
    grafana/dashboard-selector: "service=payment"
    pagerduty.com/integration-key: ABC123
    sentry.io/project-slug: payment
  tags:
    - critical
    - tier-1
spec:
  type: service
  lifecycle: production
  owner: payments-team
  system: checkout
  dependsOn:
    - resource:payment-database
    - component:auth-service
  providesApis:
    - payment-api

10.3. Software Templates

# template-go-service.yaml
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: go-microservice
  title: Go microservice (REST + gRPC)
spec:
  parameters:
    - title: Service info
      properties:
        name:
          type: string
          pattern: ^[a-z][a-z0-9-]*$
        owner:
          type: string
          ui:field: OwnerPicker
  steps:
    - id: fetch-template
      action: fetch:template
      input:
        url: ./skeleton
        values:
          name: ${{ parameters.name }}

    - id: create-repo
      action: github:repo:create
      input:
        repoUrl: github.com?owner=myorg&repo=${{ parameters.name }}

    - id: register
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.create-repo.output.repoContentsUrl }}

Dev "Create new service" → form fill → Backstage creates: GitHub repo + skeleton code + CI/CD + register catalog. ~30s thay 1 ngày setup thủ công.

10.4. Other IDP tools

• Port — modern alternative.
• Cortex — focus on service catalog + scorecards.
• OpsLevel — service maturity scoring.
• Crossplane — K8s-native infrastructure abstraction.
• Humanitec — declarative IDP.

10.5. Score — Workload spec

# score.yaml — declarative workload, platform-agnostic
apiVersion: score.dev/v1b1
metadata:
  name: my-app
containers:
  app:
    image: myapp:1.0
    variables:
      DB_URL: ${resources.db.url}
resources:
  db:
    type: postgres
  cache:
    type: redis
service:
  ports:
    web:
      port: 80
      targetPort: 8080

Platform infer infrastructure (K8s + Terraform) từ score.yaml. Dev không touch K8s YAML.

11. DevOps Maturity Roadmap

11.1. Self-assessment 5 levels

11.2. Roadmap from L1 → L5

1. L1 → L2 (3-6 tháng): Setup Git, CI pipeline test+lint, monitoring cơ bản.
2. L2 → L3 (6-12 tháng): Container hóa, K8s, IaC (Terraform), CD to staging, structured logging.
3. L3 → L4 (6-12 tháng): SLO, blameless postmortem, error budget, observability đầy đủ (3 pillars).
4. L4 → L5 (1-2 năm): CD to production, feature flags, progressive delivery, IDP, chaos engineering, FinOps.

11.3. Common pitfalls

• Tool-first thinking — buy K8s/ArgoCD trước khi văn hóa đúng.
• Skip culture — install monitoring nhưng không có blameless postmortem.
• Single hero — 1 senior làm hết DevOps, knowledge concentrated.
• "DevOps team" silo — tạo team mới mà không integrate Dev/Ops.
• Premature platform — build IDP trước có 10+ services dùng.
• Skip SLO — đo metric nhưng không có target → không actionable.

11.4. Final advice — vai trò của bạn

• Junior — focus master CLI + Git + 1 cloud + Docker. 12 tháng.
• Mid — K8s + Terraform + observability. Be on-call. 1-3 năm.
• Senior — system design, multi-region, chaos engineering, mentor. 3-7 năm.
• Staff/Principal — platform engineering, org-wide impact, technology strategy. 7+ năm.

"Đi nhanh thì đi một mình; đi xa thì đi cùng team. DevOps không phải cuộc đua kỹ năng cá nhân — là khả năng làm cho team của bạn nhanh hơn, ổn hơn, vui hơn."

12. Bài tập

1. Availability calculator: cho hệ thống 5 component nối tiếp 99.9%, tính availability tổng. Add redundancy 2× — tính lại.
2. SPOF audit: cho project hiện tại, list mọi component. Identify SPOF. Plan remediation.
3. Multi-AZ migration: nếu hệ thống single-AZ, migrate sang multi-AZ. Estimate cost increase + availability gain.
4. RTO/RPO definition: define cho service hiện tại. Match với DR strategy phù hợp. Estimate cost.
5. Backup drill: pick random backup, restore to staging. Verify data integrity. Time it. Document gaps.
6. Argo Rollouts canary: setup canary với 5% → 25% → 50% → 100%, analysis success rate qua Prometheus. Test với bad deploy → auto rollback.
7. Expand-contract migration: change schema (rename column) bằng pattern expand-contract. Document mỗi step.
8. Feature flag: implement với LaunchDarkly hoặc Unleash. Roll out feature 5% → 50% → 100%. Monitor metric per variant.
9. A/B test: design test (variant, sample size, metric, duration). Run nếu có app real. Analyze.
10. Backstage setup: cài Backstage local. Register 3 service catalog. Create software template cho Go service.
11. Maturity self-assessment: evaluate team theo 5-level model. Identify top 3 gaps. Plan 6-month roadmap.
12. DR runbook: viết runbook cho region failover. Game day với team, time the failover.
13. Compose all chapters: design end-to-end production-ready system với những gì đã học (chương 1-14). Whiteboard architecture, identify trade-offs.

13. Quiz