Monitoring & Observability

1. Monitoring vs Observability — khác biệt

1.1. Monitoring

Theo dõi known unknowns: bạn biết câu hỏi cần trả lời (CPU usage, request rate, error count). Setup dashboard + alert dựa trên metric đã định trước.

1.2. Observability

Khả năng trả lời unknown unknowns: câu hỏi mới phát sinh từ dữ liệu chưa biết trước (vì sao request này chậm? bug này xảy ra với user nào?). Cần data đủ giàu (high cardinality, structured) để query ad-hoc.

Charity Majors (Honeycomb): "Monitoring is for known failure modes. Observability is for figuring out what new thing went wrong."

1.3. Tại sao cần Observability?

Microservice + cloud-native → vấn đề trở nên emergent: 1 user request chạy qua 20 service, fail ở đâu đó. Monitoring kiểu cũ (each service has its own dashboard) không đủ. Cần:

• Distributed tracing — follow request qua services.
• High-cardinality metrics — group by user_id, request_id, ...
• Structured logs — query as data.
• Correlation — link log + metric + trace.

2. Three Pillars of Observability

2.1. Khi nào dùng cái nào?

Three pillars complement nhau. Tool modern (Datadog, New Relic, Honeycomb) integrate cả 3.

3. Metrics & Prometheus

3.1. Prometheus — de-facto standard

Prometheus (SoundCloud 2012, CNCF graduated 2018):

• Pull-based — Prometheus scrape /metrics endpoint của targets.
• Time series DB — store metric với labels.
• PromQL — query language mạnh.
• Service discovery — auto-discover targets (K8s, Consul).
• Alertmanager — route alerts.

3.2. Architecture

3.3. Metric types

3.4. Naming convention

<namespace>_<subsystem>_<name>_<unit>_<total/sum/count>

# Examples:
http_requests_total                       # counter
http_request_duration_seconds            # histogram
process_cpu_seconds_total
node_filesystem_free_bytes               # gauge
mysql_global_status_uptime_seconds

3.5. Labels — high cardinality concern

http_requests_total{method="GET", status="200", path="/api/users"} 1234
http_requests_total{method="POST", status="201", path="/api/users"} 56
http_requests_total{method="GET", status="500", path="/api/users"} 7

Labels giúp filter/aggregate. NHƯNG: cardinality explosion. Mỗi unique combination = 1 time series. user_id label với 1M users = 1M series → Prometheus chết.

Quy tắc:

• Label cardinality < 100 (low cardinality).
• KHÔNG dùng user_id, request_id, email làm label.
• Dùng label cho thuộc tính bounded: status_code (5-10 value), method (5), path (template, không dynamic).

3.6. Instrument app — Node.js example

// npm i prom-client
import express from 'express';
import { register, Counter, Histogram } from 'prom-client';

const app = express();

const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
});

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10],
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const labels = { method: req.method, route: req.route?.path ?? 'unknown', status: res.statusCode };
    httpRequests.inc(labels);
    httpDuration.observe(labels, (Date.now() - start) / 1000);
  });
  next();
});

// Expose metrics
app.get('/metrics', async (_req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(3000);

3.7. Exporters — không sửa được app

Cho legacy/third-party app, dùng exporter expose metric:

• node_exporter — host metrics (CPU, RAM, disk, network).
• postgres_exporter, mysql_exporter — DB metric.
• redis_exporter.
• blackbox_exporter — probe HTTP/TCP/ICMP từ ngoài.
• kube-state-metrics — K8s object state.
• cAdvisor — container resource.

3.8. Prometheus config

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-us-east

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node1:9100', 'node2:9100']

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)

K8s pattern: pod annotation prometheus.io/scrape: "true" + prometheus.io/port: "9090" → Prometheus auto-discover.

4. PromQL — Query Language

4.1. Basic queries

# Instant value
http_requests_total

# Filter labels
http_requests_total{status="500"}
http_requests_total{status=~"5.."}              # regex match
http_requests_total{status!="200"}

# Range vector (last 5 min)
http_requests_total[5m]

4.2. Rate() — quan trọng nhất với counter

# Counter cumulative không hữu ích trực tiếp
# rate() = derivative trong time window

# Requests per second average over last 5 min
rate(http_requests_total[5m])

# Per status
rate(http_requests_total{status="500"}[5m])

# Total requests across all dimensions
sum(rate(http_requests_total[5m]))

# By status
sum by (status) (rate(http_requests_total[5m]))

4.3. Aggregation operators

sum(...)          # sum across labels
avg(...)
max(...), min(...)
count(...)        # count time series
topk(5, ...)      # top 5 series
bottomk(5, ...)
quantile(0.99, ...)
stddev(...)

# By / Without — group
sum by (instance) (rate(http_requests_total[5m]))
sum without (status) (rate(http_requests_total[5m]))

4.4. Histogram percentile

# P95 latency (last 5 min) overall
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# P99 by route
histogram_quantile(0.99,
  sum by (route, le) (rate(http_request_duration_seconds_bucket[5m]))
)

4.5. Common queries

# RPS
sum(rate(http_requests_total[5m]))

# Error rate (%)
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))
  * 100

# CPU usage (% per node)
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory used (%)
(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
  / node_memory_MemTotal_bytes
  * 100

# Disk used (%)
(node_filesystem_size_bytes{mountpoint="/"} - node_filesystem_free_bytes{mountpoint="/"})
  / node_filesystem_size_bytes{mountpoint="/"}
  * 100

# Pod restart count (last 1h)
increase(kube_pod_container_status_restarts_total[1h])

# Slow queries (postgres)
rate(pg_stat_database_slow_queries_total[5m])

4.6. Recording rules — pre-computed query

# rules.yml
groups:
  - name: app
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_requests:error_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

Recording rule chạy expression mỗi 30s, lưu kết quả thành metric mới. Dashboard dùng job:http_requests:rate5m nhanh hơn evaluate query phức tạp mỗi refresh.

5. Grafana — Dashboarding

Grafana là frontend visualization phổ biến nhất. Support 30+ data source: Prometheus, InfluxDB, Loki, ElasticSearch, MySQL, BigQuery, ...

5.1. Setup

helm repo add grafana https://grafana.github.io/helm-charts
helm install grafana grafana/grafana \
  --set adminPassword=admin \
  --set persistence.enabled=true

# UI
kubectl port-forward svc/grafana 3000:80

Stack phổ biến: kube-prometheus-stack (helm chart) — Prometheus + Grafana + Alertmanager + node-exporter + kube-state-metrics + dashboards có sẵn cho K8s.

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

5.2. Dashboard panel types

• Time series — line/bar chart over time.
• Stat — single big number.
• Gauge — speedometer style.
• Bar gauge — multi-stat bar.
• Heatmap — distribution over time.
• Table — query result as table.
• Logs — Loki/ES log panel.
• Geomap — geographic visualization.

5.3. Variables (template)

Dropdown chọn instance/env/namespace, query update theo:

$instance — query: label_values(node_cpu_seconds_total, instance)
$namespace — query: label_values(kube_pod_info, namespace)

# Sử dụng trong panel:
sum by (pod) (rate(container_cpu_usage_seconds_total{namespace="$namespace"}[5m]))

5.4. Dashboard as Code

# Provision dashboard từ ConfigMap (kube-prometheus-stack pattern)
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-dashboard
  labels:
    grafana_dashboard: "1"
data:
  my-dashboard.json: |
    {
      "title": "My Dashboard",
      "panels": [...]
    }

Tools:

• Grafonnet — Jsonnet library cho dashboard.
• grafana-operator — K8s CRD.
• Terraform Grafana provider.

5.5. RED Method

Tom Wilkie (Grafana) đề xuất: 3 metric quan trọng cho service:

• Rate — RPS.
• Errors — error rate.
• Duration — latency P50/P95/P99.

5.6. USE Method

Brendan Gregg đề xuất cho resource:

• Utilization — % busy.
• Saturation — work queued.
• Errors.

RED cho service (request-driven). USE cho resource (CPU, disk, network).

6. Logs — ELK / Loki

6.1. Structured Logging

Plain text log:

2024-01-15 10:30:21 INFO User 12345 logged in from 1.2.3.4

Structured (JSON):

{"timestamp":"2024-01-15T10:30:21Z","level":"info","msg":"login","userId":"12345","ip":"1.2.3.4","requestId":"abc-123"}

Lợi: query as data (filter level, group by, aggregate). Modern logging library: pino (Node), structlog (Python), zap/zerolog (Go), Logrus (Go).

6.2. ELK Stack

• Elasticsearch — search engine, store + index logs.
• Logstash / Fluentd / Fluent Bit / Vector — collect/parse/forward logs.
• Kibana — UI search/visualize.

Log flow: app stdout → DaemonSet log collector (Fluent Bit) → ES → Kibana.

6.3. Loki — Prometheus for logs

Loki (Grafana Labs 2018):

• Index labels only (không full-text index nội dung) → cheap.
• Compatible Grafana, syntax giống PromQL (LogQL).
• Tích hợp tốt với Prometheus + K8s labels.

# LogQL examples
{namespace="production", app="web"}                  # filter
{app="web"} |= "error"                                # contains "error"
{app="web"} |~ "ERROR|FATAL"                          # regex
{app="web"} | json | level="error" | line_format "{{ .msg }}"

# Aggregation
sum by (status) (count_over_time({app="web"}[5m]))    # log count
rate({app="web"} |= "error" [5m])                     # error rate

6.4. Log levels

Production: INFO mặc định, có thể tăng DEBUG runtime cho specific service khi troubleshoot. Log volume cost ~$0.50/GB ingest (CloudWatch) → đắt nếu DEBUG mọi nơi.

6.5. Log retention

• Hot (7-30 ngày, queryable nhanh) — ES, Loki.
• Warm (3-12 tháng) — S3 object storage.
• Cold (> 12 tháng) — Glacier, archive.

Compliance (HIPAA, SOC2, PCI-DSS) thường yêu cầu 12 tháng audit log retention.

7. Distributed Tracing

7.1. Vấn đề trong microservice

Request user qua API Gateway → Auth → User Service → Order Service → DB → Payment Gateway. Latency tổng 2s. Bottleneck ở đâu?

Tracing follow request qua services, mỗi step = 1 span:

7.2. Concepts

• Trace — root request (trace ID unique).
• Span — 1 unit work (span ID, parent span ID).
• Context propagation — pass trace ID giữa services qua HTTP header (W3C traceparent, B3).
• Tags / attributes — metadata trên span (user_id, http.method, db.statement).

7.3. Tools

7.4. Sampling

Trace mọi request = quá nhiều data, đắt. Sampling:

• Head-based (probabilistic) — quyết định trace hay không ở root, vd 10%.
• Tail-based — buffer hết, quyết định sau (giữ trace có error/slow). Phức tạp hơn nhưng hữu ích hơn.
• Practice: 100% trace error + slow (P99), 1-10% trace bình thường.

8. OpenTelemetry — Standard mới

OpenTelemetry (CNCF, merge OpenTracing + OpenCensus 2019) — standard duy nhất cho metric + log + trace:

• Vendor-neutral — instrument 1 lần, send tới Jaeger/Tempo/Datadog/...
• OTLP protocol — gRPC + HTTP standard.
• Auto-instrumentation — Java/Python/Node tự instrument framework phổ biến.
• OTel Collector — proxy receive + transform + send.

8.1. Architecture

8.2. Instrument Node.js

// otel.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
import { Resource } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';

const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: 'my-api',
    [SemanticResourceAttributes.SERVICE_VERSION]: '1.0.0',
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: 'http://otel-collector:4318/v1/metrics',
    }),
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

# Run với --import:
node --import ./otel.ts server.ts

Auto-instrument: HTTP client/server, Express, Fastify, MySQL, Postgres, Redis, ... — 0 code change for those.

8.3. Manual span

import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('my-api');

async function processOrder(orderId: string) {
  return tracer.startActiveSpan('processOrder', async (span) => {
    span.setAttribute('order.id', orderId);

    try {
      const order = await fetchOrder(orderId);
      span.setAttribute('order.value', order.total);

      await chargeCard(order);

      span.setStatus({ code: SpanStatusCode.OK });
      return order;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

9. Alerting

9.1. Alertmanager

# alerts.yml
groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total[5m]))
          > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate {{ $value | humanizePercentage }} for 5m"
          runbook: "https://wiki.example.com/runbooks/high-error-rate"

      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Pod {{ $labels.pod }} crash looping"

# alertmanager.yml
route:
  receiver: 'default'
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      continue: true
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'pagerduty'
    pagerduty_configs:
      - service_key: '<pagerduty-key>'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

9.2. Alert design — không alert fatigue

Quy tắc:

• Chỉ alert actionable — có việc làm, không phải info.
• Alert dựa trên symptom (user impact), không cause.
  • ❌ "CPU 90%" — có thể OK (batch job).
  • ✓ "P99 latency > 1s" — user impact.
• for: 5m tránh flap — alert chỉ fire khi sustained.
• Severity rõ ràng: P1 (page on-call ngay), P2 (Slack, fix giờ làm việc), P3 (ticket).
• Mọi alert có runbook link — what to do.
• Review alert quarterly — alert nào không actionable / không fire → xóa.

9.3. PagerDuty / Opsgenie

Tools route alert tới on-call rotation, escalation, SLA tracking. Setup integration với Alertmanager.

10. SLI / SLO / SLA — Đo độ tin cậy

Chương 11 SRE đi sâu hơn. Tóm tắt ở đây:

10.1. Definitions

• SLI (Service Level Indicator) — metric đo được. "P99 latency", "uptime %", "error rate".
• SLO (Service Level Objective) — target nội bộ. "P99 < 500ms 99.9% thời gian".
• SLA (Service Level Agreement) — cam kết với khách (refund nếu vi phạm). "Uptime 99.9% hoặc 10% credit".

SLO < SLA — luôn để buffer. Nếu SLA = 99.9%, SLO nội bộ = 99.95%.

10.2. SLI categories (Google SRE)

10.3. Error Budget

SLO 99.9% → 0.1% downtime cho phép = 43 phút/tháng → "error budget".

• Còn budget → dev có thể deploy, take risk.
• Hết budget → freeze deploy, focus reliability.

Đây là cách Google SRE manage tension giữa speed (dev) và stability (ops).

10.4. Burn rate

Alert dựa trên burn rate (tốc độ tiêu budget):

• Fast burn (1h burn 2% budget = 14× rate) → page ngay.
• Slow burn (24h burn 10%) → ticket, fix ngày làm việc.

11. Dashboard Design

11.1. Hierarchy

1. Service Dashboard (RED method) — RPS, error %, P50/P95/P99 latency.
2. Resource Dashboard (USE method) — CPU/memory/disk/network util.
3. Business Dashboard — orders/min, signups, revenue.
4. Drill-down — click panel → detailed view.

11.2. Best practices

• Time range default 1h, options 5m / 1h / 6h / 24h / 7d.
• Auto-refresh 30s.
• Annotation cho deploy event (link với CI).
• Threshold lines (red = SLO breach).
• Tooltip rõ ràng (chia sẻ scope giữa panels).
• Color consistent: green=good, yellow=warn, red=bad.
• Không quá 6-8 panel/dashboard — focus.
• "On-call dashboard" 1 trang — at-a-glance health.

11.3. Anti-patterns

• "Wall of charts" — 50 panel, không ai nhìn.
• Vanity metric — chart đẹp nhưng không actionable.
• Mismatched time — 1 panel 5m, 1 panel 1h.
• Hard-coded label — không variable, dashboard cho 1 service không reuse được.

12. Bài tập

1. Setup Prometheus + Grafana: cài kube-prometheus-stack trên minikube. Verify built-in dashboards.
2. Instrument Node.js: thêm prom-client vào app, expose /metrics. Custom counter cho login attempts. Verify Prometheus scrape.
3. PromQL drill: viết query cho:
   • RPS theo service.
   • P95 latency theo route.
   • Error rate %.
   • Top 5 slowest endpoint.
   • Pod memory usage compared to limit.
4. Grafana dashboard: build dashboard với 4 panel cho service: RPS / Error rate / P50-P95-P99 latency / CPU. Add variables cho service + env.
5. Loki log aggregation: cài Loki + Promtail/Fluent Bit. Ship K8s log vào Loki. Query trong Grafana.
6. OpenTelemetry: instrument app với OTel auto-instrumentation. Send trace tới Jaeger. Verify trace UI.
7. Manual span: thêm manual span cho business operation (process order, send email). Set attribute (order_id, user_id).
8. Alertmanager: setup alert "HighErrorRate" + "PodCrashLooping". Slack webhook receiver. Trigger alert (kill app), verify Slack message.
9. RED metrics: cho 1 microservice, identify R/E/D metrics. Build dashboard.
10. SLO: define SLO cho 1 service (vd availability 99.9%, P95 latency < 200ms). Tính error budget. Setup burn rate alert (fast + slow).
11. Anti-pattern review: review dashboard hiện tại của team. Identify wall-of-charts, vanity metric. Đề xuất simplify.
12. End-to-end debugging: tạo bug giả (slow query, high error). Use trace + log + metric để tìm root cause. Document.

13. Quiz