Performance & Scaling

1. Performance đo gì?

1.1. Key metrics

• Latency — thời gian xử lý 1 request (ms). Đo P50/P95/P99.
• Throughput — request/giây (RPS, QPS).
• Concurrency — số request đang xử lý cùng lúc.
• Error rate — % request fail.
• Resource utilization — CPU, RAM, disk, network usage.
• TTFB — Time To First Byte (web).

1.2. Little's Law

L = λ × W:

• L = số request đang trong hệ.
• λ = throughput (RPS).
• W = latency trung bình (s).

Ví dụ: app 100ms/request, 50 worker → throughput tối đa = 50 / 0.1 = 500 QPS.

1.3. Vì sao P99 quan trọng hơn average?

10 request: 9 cái 50ms, 1 cái 5000ms.

• Mean = 545ms — không user nào trải nghiệm.
• P50 = 50ms — median.
• P99 = 5000ms — worst case 1% user gặp.

Web app modern có many backend call. Tail latency amplification (Jeff Dean):

Mỗi service P99 = 1% chậm.
Request fan-out tới 10 service → xác suất hit P99 = 1 - (0.99)^10 ≈ 10%.
P99 tổng > P99 từng service rất nhiều.

→ giảm P99 ở mọi service quan trọng hơn cải thiện mean.

1.4. Premature optimization

"Premature optimization is the root of all evil." — Donald Knuth

Đừng tối ưu khi chưa đo. 90% code không cần fast — tối ưu wrong place = tốn thời gian + tăng complexity.

Quy trình: measure → identify bottleneck → optimize bottleneck → measure again.

2. Load Testing

2.1. Types of test

2.2. Tools

• k6 — modern, JS scripting, multiprotocol, output cloud.
• Locust — Python, distributed, web UI.
• JMeter — Java, GUI, classic, complex test.
• wrk / wrk2 — simple, high throughput.
• vegeta — Go CLI.
• Gatling — Scala, beautiful HTML report.
• Artillery — Node.js, simple.

2.3. Test environment

Load test phải như production:

• Same hardware specs (instance type).
• Same DB version + data volume.
• Same network topology.
• Realistic data (production-like).

Anti-pattern: test trên localhost, single user — meaningless.

3. k6 Deep Dive

3.1. Cài + script đầu tiên

brew install k6
# Hoặc Docker: docker run --rm -i grafana/k6 run - < test.js

// test.js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 10,           // virtual users
  duration: '30s',
};

export default function () {
  const res = http.get('https://api.example.com/users/1');
  check(res, {
    'status 200': (r) => r.status === 200,
    'duration < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

k6 run test.js

# Output:
#   http_req_duration..............: avg=145ms p(95)=320ms p(99)=520ms
#   http_reqs......................: 245    8.166/s
#   checks.........................: 100.00% ✓ 490 ✗ 0

3.2. Stages — ramp up/down

export const options = {
  stages: [
    { duration: '2m', target: 100 },   // ramp up to 100 VUs
    { duration: '5m', target: 100 },   // stay
    { duration: '2m', target: 200 },   // ramp up
    { duration: '5m', target: 200 },
    { duration: '2m', target: 0 },     // ramp down
  ],
  thresholds: {
    'http_req_duration{status:200}': ['p(95)<500', 'p(99)<1000'],
    'http_req_failed': ['rate<0.01'],   // < 1% errors
    'http_reqs': ['rate>100'],          // sustain > 100 RPS
  },
};

Threshold fail → exit code 1 → CI fail.

3.3. Realistic scenarios

import http from 'k6/http';
import { check, sleep, group } from 'k6';
import { SharedArray } from 'k6/data';

const users = new SharedArray('users', function () {
  return JSON.parse(open('./users.json'));
});

export const options = {
  scenarios: {
    browse: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '5m', target: 50 },
        { duration: '20m', target: 50 },
        { duration: '5m', target: 0 },
      ],
      exec: 'browseScenario',
    },
    checkout: {
      executor: 'constant-arrival-rate',
      rate: 10,                       // 10 checkouts/sec
      timeUnit: '1s',
      duration: '20m',
      preAllocatedVUs: 20,
      exec: 'checkoutScenario',
    },
  },
};

export function browseScenario() {
  group('Homepage', () => {
    const res = http.get('https://shop.example.com/');
    check(res, { 'status 200': (r) => r.status === 200 });
  });

  group('Product page', () => {
    const productId = Math.floor(Math.random() * 100);
    const res = http.get(`https://shop.example.com/product/${productId}`);
    check(res, { 'status 200': (r) => r.status === 200 });
  });

  sleep(2);
}

export function checkoutScenario() {
  const user = users[Math.floor(Math.random() * users.length)];

  // Login
  const loginRes = http.post('https://shop.example.com/api/login',
    JSON.stringify({ email: user.email, password: user.password }),
    { headers: { 'Content-Type': 'application/json' } }
  );
  check(loginRes, { 'login ok': (r) => r.status === 200 });
  const token = loginRes.json('token');

  // Add to cart
  http.post('https://shop.example.com/api/cart',
    JSON.stringify({ productId: 'abc', qty: 1 }),
    { headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' } }
  );

  // Checkout
  const checkoutRes = http.post('https://shop.example.com/api/checkout',
    JSON.stringify({ payment: 'mock' }),
    { headers: { 'Authorization': `Bearer ${token}`, 'Content-Type': 'application/json' } }
  );
  check(checkoutRes, { 'checkout ok': (r) => r.status === 201 });
}

3.4. CI integration

# .github/workflows/perf.yml
name: Performance Test

on:
  schedule:
    - cron: '0 2 * * *'    # nightly
  pull_request:
    paths: ['src/api/**']

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/k6-action@v0.3.1
        with:
          filename: tests/load.js
          flags: --out cloud   # k6 Cloud
        env:
          K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}

Track regression: nếu P95 latency tăng > 20% so baseline, fail CI.

4. Profiling

4.1. CPU profiling

Tìm function nào ăn CPU nhiều nhất:

• Node.js — --prof, Clinic.js, 0x.
• Python — cProfile, py-spy, scalene.
• Go — pprof (built-in).
• Java — JFR (Java Flight Recorder), async-profiler.
• Rust — cargo flamegraph, perf.
• OS-level — perf, eBPF.

4.2. Go pprof example

// main.go
import _ "net/http/pprof"
import "net/http"

func main() {
  go http.ListenAndServe(":6060", nil)
  // app code...
}

# CPU profile (30s)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
(pprof) top
(pprof) list functionName
(pprof) web                # SVG flame graph

# Heap profile
go tool pprof http://localhost:6060/debug/pprof/heap

# Goroutine
go tool pprof http://localhost:6060/debug/pprof/goroutine

4.3. Flame graph

Visualize CPU time stack: width = thời gian, x-axis = sample sorted, y-axis = stack depth. Wide function = hot path.

Tools: Brendan Gregg's FlameGraph, pprof web command.

4.4. Heap profiling — memory leak

# Node.js heap snapshot
node --inspect server.js
# Open chrome://inspect → Take heap snapshot, compare

# Python
pip install memory-profiler
python -m memory_profiler script.py

# Go
go tool pprof -alloc_objects http://localhost:6060/debug/pprof/heap

4.5. Distributed tracing for perf

Tracing (chương 10) là profiling cho microservice. P99 latency request = trace có waterfall view, identify slowest span.

4.6. Continuous profiling

Tools chạy continuous trong production:

• Pyroscope (Grafana) — open source.
• Parca — eBPF-based.
• Datadog Continuous Profiler, Pixie (acquired by NewRelic).
• Google Cloud Profiler.

Always-on profiling (1-2% overhead) — catch regression real-time.

5. Caching

Phil Karlton: "There are only two hard things in Computer Science: cache invalidation and naming things."

5.1. Caching layers

5.2. Cache patterns

5.3. Cache-Aside example

async function getUser(id: string): Promise<User> {
  // 1. Check cache
  const cached = await redis.get(`user:${id}`);
  if (cached) return JSON.parse(cached);

  // 2. Miss — fetch DB
  const user = await db.user.findById(id);
  if (!user) throw new Error('Not found');

  // 3. Populate cache (TTL 5 min)
  await redis.set(`user:${id}`, JSON.stringify(user), 'EX', 300);

  return user;
}

// Invalidation on update
async function updateUser(id: string, data: Partial<User>) {
  await db.user.update(id, data);
  await redis.del(`user:${id}`);
}

5.4. Cache invalidation

3 strategy:

1. TTL — expire sau N giây. Đơn giản, eventual consistent.
2. Write invalidate — delete cache khi update. Fresh nhưng race condition.
3. Pub/Sub invalidate — N service shared cache, broadcast invalidation event.

5.5. Eviction policies

• LRU (Least Recently Used) — đẩy ra cái lâu không dùng. Default Redis.
• LFU (Least Frequently Used) — đẩy ra cái ít dùng.
• FIFO — first in first out.
• Random — random.
• TTL-based — expire hardcoded.

5.6. Common pitfalls

5.6.1. Thundering herd / Cache stampede

Hot key expire → 1000 request cùng lúc miss → 1000 query DB → DB chết.

Fix:

• Lock: chỉ 1 request fetch DB, others wait.
• Probabilistic early refresh: refresh trước expire.
• Stale-while-revalidate: serve stale + async refresh.

5.6.2. Negative cache

Query "user 99999" không tồn tại → DB miss → app cache "not found"? Yes — tránh repeat DB query attack.

5.6.3. Cache penetration

Attacker query random ID → all miss cache → flood DB. Fix: bloom filter check existence trước.

5.7. Redis cluster

Single Redis instance limit ~10GB RAM, 100k QPS. Scale:

• Read replica — write primary, read replica.
• Redis Cluster — sharded across nodes.
• Managed — ElastiCache, Memorystore, Azure Cache.

6. CDN — Edge Caching

6.1. Vì sao CDN?

• Latency — user nhận content từ edge gần (~10ms thay vì cross-continent ~150ms).
• Egress cost — CDN cache reduce origin egress (cloud egress đắt).
• DDoS protection — CDN absorb attack.
• Origin scale — CDN serve majority traffic, origin chỉ handle uncached.

6.2. Providers

• Cloudflare — popular, free tier rộng, security focus.
• Fastly — developer-friendly, instant purge, edge compute.
• AWS CloudFront — tích hợp AWS.
• GCP Cloud CDN.
• Akamai — enterprise, high reach.
• Bunny.net — cheap, simple.

6.3. Cache-Control headers

Cache-Control: public, max-age=86400, s-maxage=86400, stale-while-revalidate=3600

# public            — both browser and CDN cache
# private           — only browser cache
# no-cache          — must revalidate before use
# no-store          — never cache
# max-age=N         — browser cache N seconds
# s-maxage=N        — CDN cache N seconds
# stale-while-revalidate=N — serve stale while async fetch fresh
# stale-if-error=N  — serve stale if origin error

6.4. Cache strategy by content type

6.5. Cache invalidation

# CloudFront invalidation
aws cloudfront create-invalidation \
  --distribution-id E123456 \
  --paths "/index.html" "/api/*"

# Cloudflare API
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/purge_cache" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"files":["https://example.com/index.html"]}'

Best practice: cache-busting qua filename (app.abc123.js) thay vì invalidation API. Build tool tạo hash filename mỗi build.

6.6. Edge compute

Modern CDN run code at edge:

• Cloudflare Workers — V8 isolate.
• Fastly Compute@Edge — WASM.
• AWS Lambda@Edge / CloudFront Functions.
• Vercel Edge Functions.

Use case: A/B testing, auth, header manipulation, redirect, edge cache logic.

7. Database Scaling

7.1. Vertical scaling (scale up)

Bigger instance — easier, cheaper to start. Cloud max: ~24 TB RAM, 192 vCPU. Đến giới hạn: 1 node Postgres ~50k QPS read, ~20k QPS write.

7.2. Read replicas

Async replication primary → replica. Route read sang replica, write vào primary.

// App routing
class DB {
  primary = new Pool({ host: 'primary.db' });
  replica = new Pool({ host: 'replica.db' });

  async write(query: string, params: any[]) {
    return this.primary.query(query, params);
  }

  async read(query: string, params: any[]) {
    // Async replica có thể stale (ms delay)
    return this.replica.query(query, params);
  }

  async readAfterWrite(query: string, params: any[]) {
    // Need read-your-write consistency? Use primary
    return this.primary.query(query, params);
  }
}

7.3. Sharding

Chia data ra nhiều DB instance theo key (user_id mod N, range, ...).

Pros: scale write, capacity. Cons: complexity (cross-shard query khó), hot shard, rebalance đau.

Tools: Vitess (YouTube), Citus (Postgres extension), CockroachDB (distributed SQL native).

7.4. NoSQL — built-in sharding

• DynamoDB / Cassandra — partition key sharding tự động.
• MongoDB — replica set + sharding.
• Redis Cluster — 16384 slots.

7.5. Connection pooling

Postgres mỗi connection = 1 process (~10MB). 1000 client direct → DB overload.

Solution: pooler:

• PgBouncer — Postgres pooler.
• RDS Proxy — managed.
• Supabase Pooler.

7.6. Caching tier

DB là expensive resource. Cache (Redis) absorb majority read → DB chỉ handle uncached + write.

7.7. Database for specific workload

"Right tool for right job" — không 1 DB cho mọi workload.

8. Autoscaling — K8s deep

8.1. HPA (Horizontal Pod Autoscaler)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 3
  maxReplicas: 100

  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

    - type: Pods                         # custom metric per Pod
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "100"             # 100 RPS per Pod

    - type: External                     # external metric (queue length)
      external:
        metric:
          name: sqs_messages
          selector:
            matchLabels:
              queue: orders
        target:
          type: Value
          value: "30"

  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100                      # double up to
          periodSeconds: 30
        - type: Pods
          value: 5                         # or +5 pods
          periodSeconds: 30
      selectPolicy: Max                   # use max of policies

    scaleDown:
      stabilizationWindowSeconds: 300     # 5 phút cool-down
      policies:
        - type: Percent
          value: 10                       # scale down 10% mỗi 60s
          periodSeconds: 60

8.2. KEDA (Event-driven)

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: order-processor
spec:
  scaleTargetRef:
    name: order-worker
  minReplicaCount: 0                     # scale-to-zero!
  maxReplicaCount: 50

  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka:9092
        consumerGroup: orders
        topic: orders
        lagThreshold: "100"

    - type: prometheus
      metadata:
        serverAddress: http://prometheus:9090
        metricName: pending_jobs
        threshold: "10"
        query: sum(rabbitmq_queue_messages_ready{queue="orders"})

KEDA scale-to-zero: khi không event, 0 pod (no cost). HPA min = 1.

8.3. VPA (Vertical Pod Autoscaler)

Adjust CPU/RAM request, không thay replica. Mode:

• Off — recommend only.
• Initial — set when pod creation.
• Auto — restart pod với resource mới (disruptive).

VPA + HPA cùng CPU = conflict. Use VPA Recommend mode + HPA on CPU.

8.4. Cluster Autoscaler / Karpenter

• Cluster Autoscaler (CA) — original, scale node group dựa trên unschedulable pods.
• Karpenter (AWS) — modern, faster (no node group), better bin-packing, mixed instance types, spot support.

# Karpenter NodePool
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-cpu
          operator: In
          values: ["4", "8", "16"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
      nodeClassRef:
        name: default
  limits:
    cpu: "1000"
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h                    # 30 days

8.5. PodDisruptionBudget

Đảm bảo không quá nhiều Pod down cùng lúc khi voluntary disruption (drain, autoscaler):

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 80%                      # hoặc maxUnavailable: 20%
  selector:
    matchLabels: { app: web }

9. Bottleneck Analysis

9.1. Common bottlenecks

9.2. USE Method

Brendan Gregg: cho mỗi resource, check:

• Utilization — % busy.
• Saturation — work queued.
• Errors.

Resource: CPU, RAM, network, disk, file descriptor, ...

9.3. Amdahl's Law

Speedup tối đa với parallelization: S = 1 / ((1-p) + p/n).

Nếu 90% code parallelizable, dùng 100 core: S = 1 / (0.1 + 0.9/100) = 9.17×, KHÔNG 100×.

Hệ quả: serial portion là wall — không break được. "Add more servers" có giới hạn.

9.4. Don't run > 80% utilization

Khi gần 100% capacity, queue overflow → latency tăng vọt 10-100×. Quy tắc: run < 80% (autoscale at 70%).

9.5. Real example: API chậm

Symptom: P95 latency 2000ms (was 200ms last week).

Step 1: Time series — when did it start?
  → Spike at 2024-01-15 10:00.

Step 2: Recent change?
  → Deploy at 09:55 — likely cause.

Step 3: Profile request flow (trace).
  → Slowest span: db.query "SELECT FROM orders".
  → 1500ms in DB!

Step 4: DB-side investigation.
  → EXPLAIN ANALYZE: Seq Scan instead of Index Scan.
  → Index dropped in migration!

Fix: Add back index. Latency back to 200ms.

Postmortem action: pre-deploy DB performance check.

10. Performance Budget

10.1. Concept

Define max acceptable latency / size, enforce trong CI:

# performance-budget.yml
api:
  endpoint: /api/users
  P50: 50ms
  P95: 200ms
  P99: 500ms
  error_rate: < 0.1%

frontend:
  bundle_size_js: 200 KB         # gzipped
  bundle_size_css: 50 KB
  Largest_Contentful_Paint: 2.5s
  Time_To_Interactive: 3.0s
  Cumulative_Layout_Shift: < 0.1

10.2. Enforce trong CI

- name: Build
  run: npm run build

- name: Bundle size check
  run: npx bundlesize           # fail if exceed budget

- name: Lighthouse CI
  uses: treosh/lighthouse-ci-action@v11
  with:
    configPath: './lighthouserc.json'
    # Lighthouse thresholds:
    # performance: 90, LCP < 2.5s, CLS < 0.1

- name: API perf test
  run: k6 run perf-test.js
  # Threshold fail → CI fail

10.3. Web vitals (Frontend)

Google ranking factor — performance ảnh hưởng SEO.

11. FinOps — Cost Optimization

11.1. Common waste

• Over-provisioned instances (high RAM but low usage).
• Unused resources (orphan EBS, unused EIPs).
• Forgotten dev/staging running 24/7.
• Egress trafic unnecessary cross-region.
• Unoptimized images / data transfer.
• NAT Gateway expensive.
• Reserved Instance không matching workload.

11.2. Right-sizing

Tools recommend instance size based on actual usage:

• AWS Compute Optimizer.
• GCP Recommender.
• K8s VPA recommend mode.

11.3. Spot / Preemptible

Discount 60-90% off on-demand. Use cho:

• Stateless worker.
• Batch job (replay if killed).
• CI runner.
• K8s dev/staging clusters.
• Mixed cluster: 30% on-demand baseline + 70% spot burst.

11.4. Reserved / Committed Use

Cam kết 1-3 năm cho discount 40-72%. Pattern:

• Reserved cho baseline (steady).
• Spot cho burst.
• On-demand cho overflow.

11.5. Auto stop dev/staging

# EventBridge rule schedule stop EC2 after 6pm
aws events put-rule --schedule-expression "cron(0 18 ? * MON-FRI *)"

# Auto resume morning
# Tag-based: tag instances "AutoStop: true"

Tools: AWS Instance Scheduler, GCP Instance Groups schedule.

11.6. Cost monitoring

• Tag everything — Owner, Project, Env, CostCenter.
• Budget alert — Slack notification 80% / 100%.
• Showback / chargeback — chia phí theo team.
• Anomaly detection — alert khi cost spike.

Tools: AWS Cost Explorer / Budget, Vantage, Infracost (preview cost của Terraform PR), CloudHealth.

11.7. Infracost trong CI

- name: Infracost diff
  uses: infracost/infracost-action@v3
  with:
    api_key: ${{ secrets.INFRACOST_API_KEY }}
    path: ./terraform
  # Comment cost difference vào PR

Result: PR comment "this PR will increase monthly cost by $1200" — review trước approve.

12. Bài tập

1. k6 first test: load test API với 50 VUs, 5 phút. Check P95 < 500ms threshold. CI integration.
2. Stress test: ramp up 0 → 1000 VUs trong 10 phút. Tìm breaking point. Plot RPS vs latency.
3. Profile: lấy CPU profile của Node.js / Go app trong 30s. Generate flame graph. Identify hot path.
4. Heap leak: tạo intentional memory leak. Reproduce với stress test. Take heap snapshot, find leak.
5. Cache implementation: thêm Redis cache-aside cho 1 endpoint. Đo cache hit rate. Compare latency before/after.
6. CDN setup: setup CloudFront/Cloudflare cho static site. Measure latency từ different region. Verify cache header.
7. DB optimization: identify N+1 query. Fix. Add index for slow query (EXPLAIN ANALYZE). Measure.
8. HPA: setup HPA cho web deployment. Stress test, watch scale up. Setup behavior policy (no flapping).
9. KEDA scale-to-zero: deploy worker với KEDA scale theo Redis queue length. Idle = 0 pod. Add msg = scale up.
10. Karpenter: trên EKS, deploy Karpenter. Schedule pod request more than node available, watch Karpenter spin up new node.
11. PodDisruptionBudget: setup PDB minAvailable 80% cho web. Drain node, verify K8s không violate PDB.
12. Performance budget: define + enforce: API P95 latency, JS bundle < 200KB, Lighthouse score > 90.
13. Cost analysis: review AWS cost 30 ngày qua. Top 5 categories. Identify 3 quick wins (vd: unused EBS, forgotten staging EC2).
14. Right-sizing: dùng Compute Optimizer / VPA recommend. Apply suggestions, đo cost saving.
15. Infracost CI: setup Infracost trong PR. Test bằng PR thêm large RDS — verify cost diff comment.

13. Quiz