Configuration Management & Secrets

1. The Twelve-Factor App

2011, Heroku xuất bản 12-Factor App (12factor.net) — 12 nguyên lý xây cloud-native app. Vẫn relevant 2026.

1. Codebase — 1 codebase tracked in Git, nhiều deploy.
2. Dependencies — declare explicitly (package.json, requirements.txt), không phụ thuộc system-wide.
3. Config — store in environment variables, KHÔNG hardcode trong code.
4. Backing services — DB, cache, queue là attached resources, swappable qua URL/config.
5. Build, release, run — strictly separate stages.
6. Processes — execute as stateless processes (state ở backing service).
7. Port binding — service self-contained, expose via port (không dependent web server external).
8. Concurrency — scale out via process model.
9. Disposability — fast startup, graceful shutdown.
10. Dev/prod parity — keep environments similar (cùng OS, DB version).
11. Logs — treat as event streams, write to stdout/stderr.
12. Admin processes — run admin tasks (migration) as one-off processes.

Factor 3 (Config) là tâm điểm chương này.

2. Config tách khỏi code — Vì sao?

2.1. Anti-pattern: hardcode

// ❌ BAD
const db = new Pool({
  host: 'prod-db.example.com',
  port: 5432,
  user: 'admin',
  password: 'P@ssw0rd123!',                    // SECRET in code!
});

const apiKey = 'sk_live_4xPdL3...';            // SECRET in code!

Vấn đề:

• Secret leak qua git push (vĩnh viễn trong history).
• Cannot reuse code giữa env (dev/staging/prod).
• Developer không nên thấy prod password.
• Đổi password = redeploy.

2.2. Pattern đúng: Environment Variables

// ✓ GOOD
const db = new Pool({
  host: process.env.DB_HOST,
  port: parseInt(process.env.DB_PORT ?? '5432'),
  user: process.env.DB_USER,
  password: process.env.DB_PASSWORD,
});

const apiKey = process.env.STRIPE_API_KEY;

if (!apiKey) {
  throw new Error('STRIPE_API_KEY required');
}

Env vars set khi deploy:

# Local dev — .env file (gitignored)
DB_HOST=localhost
DB_USER=app
DB_PASSWORD=devpass
STRIPE_API_KEY=sk_test_xxx

# Production — set qua secret manager (xem section 7-9)

2.3. Config types

2.4. Config best practices

1. Validate config at startup — fail fast nếu thiếu.
2. Type-safe config (TypeScript, Zod, Pydantic).
3. Different env values: dev .env, staging via CI, prod via secret manager.
4. Document required vars (README, .env.example).
5. Rotate secret regularly (90 days).

3. Ansible — Configuration Management

3.1. Vì sao Ansible?

IaC (Terraform) tạo infrastructure (VM, network). Còn cài software trên VM (nginx, fluent-bit, monitoring agent), cấu hình OS (firewall, sysctl), deploy app? Đó là configuration management.

Ansible (Red Hat, 2012):

• Agentless — chỉ cần SSH, không cài daemon trên target.
• YAML — declarative playbook.
• Idempotent — chạy nhiều lần OK.
• 1500+ modules — apt, yum, file, service, user, ...

3.2. Cài

pip install ansible
ansible --version

# Hoặc
brew install ansible

3.3. Inventory — danh sách host

# inventory.ini
[web]
web1.example.com
web2.example.com
web3.example.com

[db]
db.example.com

[all:vars]
ansible_user=ubuntu
ansible_ssh_private_key_file=~/.ssh/id_ed25519

# inventory.yml — recommended (more features)
all:
  children:
    web:
      hosts:
        web1:
          ansible_host: 10.0.1.10
        web2:
          ansible_host: 10.0.1.11
    db:
      hosts:
        db1:
          ansible_host: 10.0.10.10
  vars:
    ansible_user: ubuntu

3.4. Ad-hoc commands

# Ping all hosts
ansible all -i inventory.ini -m ping

# Run command
ansible web -i inventory.ini -m shell -a "uptime"

# Copy file
ansible web -m copy -a "src=app.conf dest=/etc/app/app.conf owner=root mode=0644"

# Install package
ansible web -m apt -a "name=nginx state=present" --become

--become = sudo. -m = module name.

4. Playbooks & Roles

4.1. Playbook đầu tiên

# playbook.yml
---
- name: Setup web server
  hosts: web
  become: yes
  vars:
    nginx_version: "1.24"
    app_dir: /opt/myapp

  tasks:
    - name: Update apt cache
      apt:
        update_cache: yes
        cache_valid_time: 3600

    - name: Install dependencies
      apt:
        name:
          - nginx
          - curl
          - htop
        state: present

    - name: Create app user
      user:
        name: myapp
        system: yes
        shell: /bin/false

    - name: Create app directory
      file:
        path: "{{ app_dir }}"
        state: directory
        owner: myapp
        group: myapp
        mode: '0755'

    - name: Copy nginx config
      template:
        src: nginx.conf.j2
        dest: /etc/nginx/sites-available/myapp
        mode: '0644'
      notify: reload nginx

    - name: Enable nginx site
      file:
        src: /etc/nginx/sites-available/myapp
        dest: /etc/nginx/sites-enabled/myapp
        state: link
      notify: reload nginx

    - name: Start nginx
      service:
        name: nginx
        state: started
        enabled: yes

  handlers:
    - name: reload nginx
      service:
        name: nginx
        state: reloaded

# Run
ansible-playbook -i inventory.ini playbook.yml

# Dry-run (check mode)
ansible-playbook -i inventory.ini playbook.yml --check --diff

# Limit hosts
ansible-playbook playbook.yml --limit web1

4.2. Variables & Templates (Jinja2)

# templates/nginx.conf.j2
upstream backend {
{% for host in groups['web'] %}
    server {{ hostvars[host]['ansible_host'] }}:3000;
{% endfor %}
}

server {
    listen 80;
    server_name {{ domain_name | default('example.com') }};

    location / {
        proxy_pass http://backend;
        proxy_set_header Host $host;
    }

    {% if enable_ssl | default(false) %}
    listen 443 ssl;
    ssl_certificate {{ ssl_cert_path }};
    ssl_certificate_key {{ ssl_key_path }};
    {% endif %}
}

4.3. Roles — reusable playbook

roles/
├── nginx/
│   ├── defaults/main.yml      # default vars (lowest priority)
│   ├── vars/main.yml          # vars (higher priority)
│   ├── tasks/main.yml         # main task list
│   ├── handlers/main.yml      # handler list
│   ├── templates/             # jinja2 templates
│   ├── files/                 # static files
│   └── meta/main.yml          # dependencies
└── postgres/
    └── ...

# site.yml — top-level
- name: Configure web tier
  hosts: web
  become: yes
  roles:
    - common
    - nginx
    - { role: monitoring, agent_type: prometheus }

- name: Configure DB tier
  hosts: db
  become: yes
  roles:
    - common
    - postgres

4.4. Ansible Vault — encrypt secret in repo

# Tạo file encrypted
ansible-vault create secrets.yml
# Editor mở, type secrets:
# db_password: secret123

# View
ansible-vault view secrets.yml

# Edit
ansible-vault edit secrets.yml

# Encrypt existing file
ansible-vault encrypt vars.yml

# Decrypt
ansible-vault decrypt secrets.yml

# Run với password
ansible-playbook playbook.yml --ask-vault-pass
ansible-playbook playbook.yml --vault-password-file=~/.vault_pass

Pattern: vault password ở 1Password / env var, không commit. Secrets.yml encrypted commit OK.

4.5. Ansible Galaxy — community roles

# Cài role từ Galaxy
ansible-galaxy install geerlingguy.nginx
ansible-galaxy install geerlingguy.postgresql

# requirements.yml
ansible-galaxy install -r requirements.yml

5. Alternatives — Chef, Puppet, Salt

Trend modern: container-first → ít nhu cầu CM cho VM. Ansible vẫn relevant cho:

• VM bare metal / on-prem.
• Network device (Cisco, Juniper).
• Cloud day-1 setup (initial bootstrap K8s, install ops tools).
• Application deployment ngoài K8s.

5.1. Cloud-init — alternative cho cloud

#cloud-config
package_update: true
packages:
  - nginx
  - htop

write_files:
  - path: /etc/nginx/sites-available/myapp
    content: |
      server {
        listen 80;
        location / { proxy_pass http://localhost:3000; }
      }

runcmd:
  - ln -sf /etc/nginx/sites-available/myapp /etc/nginx/sites-enabled/
  - systemctl reload nginx

cloud-init chạy 1 lần khi VM boot lần đầu — bootstrap đơn giản. Sau đó dùng Ansible cho ongoing changes.

5.2. Image-based pattern (immutable)

Modern best practice: thay sửa server đang chạy → bake image với Packer + redeploy.

# packer/web.pkr.hcl
source "amazon-ebs" "web" {
  ami_name      = "myapp-web-{{ timestamp }}"
  instance_type = "t3.medium"
  region        = "us-east-1"
  source_ami    = "ami-..."
  ssh_username  = "ubuntu"
}

build {
  sources = ["source.amazon-ebs.web"]

  provisioner "ansible" {
    playbook_file = "ansible/web.yml"
  }
}

Workflow: Packer + Ansible → tạo AMI → ASG launch instance từ AMI mới → rolling update. Immutable infrastructure.

6. Secrets — Vấn đề thật

6.1. Common mistakes

• Hardcode trong code.
• Commit .env vào Git.
• Slack/Email với password.
• Lưu secret trong CI variable plaintext.
• Long-lived API key, không rotate.
• Cùng password cho nhiều env (dev = prod).
• Print secret vào CI log.

6.2. Hậu quả secret leak

• Cloud bill $100k qua đêm (crypto miner trên AWS với leaked key).
• DB exfiltrate.
• GDPR fine.
• Reputation damage.

2024 GitGuardian report: 12.8M secrets leak qua public GitHub repos. AWS keys, Stripe key, OpenAI key thường gặp.

6.3. Best practices

1. Never commit — gitleaks, trufflehog scan trong pre-commit + CI.
2. Centralized secret manager — Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault.
3. Short-lived — IAM Role + STS thay long-lived key. OIDC (xem chương 4).
4. Rotation — auto rotate every 90 days.
5. Audit — log mọi access secret.
6. Least privilege — mỗi service chỉ truy cập secret cần.
7. Encryption-at-rest + in-transit.
8. Detect leak — GitGuardian, GitHub Secret Scanning.

6.4. Phản ứng khi secret leak

1. Rotate immediately — invalidate key cũ, tạo mới.
2. Audit logs — kiểm tra dùng đến đâu.
3. Notify — security team, downstream services.
4. Fix root cause — script leak, dev workflow, ...
5. Postmortem.

7. HashiCorp Vault — Industry Standard

7.1. Vault concepts

• Secrets engines — KV (key-value), database (dynamic creds), AWS, PKI, transit (encryption-as-a-service), ...
• Auth methods — token, AppRole, K8s, AWS IAM, JWT/OIDC, LDAP, ...
• Policies — HCL, control access (path-based).
• Audit log — log mọi request.
• Seal/Unseal — Vault encrypted at rest, unseal qua Shamir's secret sharing.

7.2. Setup Vault dev

# Cài
brew install vault

# Dev mode (NEVER in prod — không persist)
vault server -dev
# Output:
# Root Token: hvs.xxx...
# Unseal Key: ...

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='hvs.xxx'

vault status

7.3. KV secrets

# KV v2
vault kv put secret/myapp/prod \
  db_password=secret123 \
  api_key=sk_live_xxx \
  jwt_secret=$(openssl rand -hex 32)

vault kv get secret/myapp/prod
vault kv get -field=db_password secret/myapp/prod
vault kv get -format=json secret/myapp/prod

# Versioning
vault kv get -version=1 secret/myapp/prod
vault kv rollback -version=2 secret/myapp/prod

# Delete + undelete
vault kv delete secret/myapp/prod
vault kv undelete -versions=3 secret/myapp/prod

7.4. Policies

# myapp-policy.hcl
path "secret/data/myapp/prod" {
  capabilities = ["read"]
}

path "secret/metadata/myapp/prod" {
  capabilities = ["list", "read"]
}

# Database dynamic secret
path "database/creds/myapp-readonly" {
  capabilities = ["read"]
}

vault policy write myapp myapp-policy.hcl
vault policy list
vault policy read myapp

7.5. AppRole auth (for app)

# Enable AppRole
vault auth enable approle

# Create role
vault write auth/approle/role/myapp \
  policies="myapp" \
  token_ttl=1h \
  token_max_ttl=24h

# Get role_id (commit OK, like username)
vault read auth/approle/role/myapp/role-id

# Get secret_id (DON'T commit — like password)
vault write -f auth/approle/role/myapp/secret-id

// App login + read secret
import vault from 'node-vault';

const client = vault({ endpoint: 'http://vault:8200' });

// Login
const { auth } = await client.approleLogin({
  role_id: process.env.VAULT_ROLE_ID,
  secret_id: process.env.VAULT_SECRET_ID,
});

client.token = auth.client_token;

// Read secret
const { data } = await client.read('secret/data/myapp/prod');
const dbPassword = data.data.db_password;

7.6. Database dynamic secret — game changer

# Enable database secrets engine
vault secrets enable database

# Configure Postgres
vault write database/config/myapp-db \
  plugin_name=postgresql-database-plugin \
  allowed_roles="myapp-readonly" \
  connection_url="postgresql://{{username}}:{{password}}@db:5432/myapp" \
  username="vault" password="vault_master_password"

# Define role
vault write database/roles/myapp-readonly \
  db_name=myapp-db \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
                        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" \
  max_ttl="24h"

# App đọc credentials → Vault tự CREATE USER với password random, TTL 1h
vault read database/creds/myapp-readonly
# Key            Value
# username       v-myapp-readonly-AbCd1234...
# password       random_strong_pass
# lease_duration 1h

Lợi ích: mỗi app instance có credential riêng, tự expire — không có credential static để leak.

8. Cloud Secret Managers

8.1. AWS Secrets Manager

# Tạo secret
aws secretsmanager create-secret \
  --name prod/myapp/db \
  --secret-string '{"username":"app","password":"secret123"}'

# Read
aws secretsmanager get-secret-value --secret-id prod/myapp/db

# Update
aws secretsmanager update-secret \
  --secret-id prod/myapp/db \
  --secret-string '{"username":"app","password":"newpass"}'

# Auto rotation (Lambda rotate secret, update DB password)
aws secretsmanager rotate-secret \
  --secret-id prod/myapp/db \
  --rotation-lambda-arn arn:aws:lambda:...

Pricing: $0.40/secret/month + $0.05/10k API call.

8.2. AWS Parameter Store (Systems Manager)

# String / SecureString
aws ssm put-parameter \
  --name /myapp/prod/db_password \
  --value secret123 \
  --type SecureString

aws ssm get-parameter --name /myapp/prod/db_password --with-decryption

Cheaper alternative: Standard parameter free, advanced $0.05/10k.

8.3. GCP Secret Manager

gcloud secrets create myapp-db-password \
  --data-file=- <<< "secret123"

gcloud secrets versions access latest --secret=myapp-db-password

8.4. Azure Key Vault

az keyvault create --name myKeyVault --resource-group myRG --location eastus

az keyvault secret set --vault-name myKeyVault \
  --name db-password \
  --value secret123

az keyvault secret show --vault-name myKeyVault --name db-password

8.5. Đọc secret từ K8s — External Secrets Operator

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa     # IRSA cho SA này

---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: ClusterSecretStore
  target:
    name: app-secrets                       # K8s Secret được tạo
  data:
    - secretKey: db-password
      remoteRef:
        key: prod/myapp/db
        property: password
    - secretKey: api-key
      remoteRef:
        key: prod/myapp/api

ESO sync từ Secrets Manager → tạo K8s Secret. Pod mount Secret. Source of truth ở Secrets Manager (audit, rotation).

9. Sealed Secrets / SOPS — Encrypt secret trong Git

GitOps yêu cầu mọi config trong Git. Nhưng plaintext secret trong Git = dead. Solution: encrypt secret trước khi commit.

9.1. Sealed Secrets (Bitnami)

# Cài controller trên cluster
helm install sealed-secrets sealed-secrets/sealed-secrets

# Cài CLI
brew install kubeseal

# Tạo Secret bình thường
echo -n 'secret123' | kubectl create secret generic db-pw \
  --dry-run=client --from-file=password=/dev/stdin -o yaml > secret.yaml

# Encrypt với public key của controller (trên cluster)
kubeseal -o yaml < secret.yaml > sealed-secret.yaml

# Now commit sealed-secret.yaml vào Git — safe!
# Khi apply lên cluster, controller decrypt với private key
kubectl apply -f sealed-secret.yaml

9.2. SOPS (Mozilla)

brew install sops

# Encrypt với KMS / age / PGP
export SOPS_KMS_ARN="arn:aws:kms:us-east-1:..."
sops -e secrets.yaml > secrets.enc.yaml

# Edit (decrypt + edit + encrypt)
sops secrets.enc.yaml

# Decrypt for use
sops -d secrets.enc.yaml

SOPS phổ biến vì: support YAML/JSON/ENV, integrate với Helm Secrets, Terraform, ArgoCD.

9.3. age — modern alternative GPG

age-keygen -o key.txt
# AGE-SECRET-KEY-1...
# Public: age1xxx...

# Encrypt
age -r age1xxx -o secrets.age secrets.txt

# Decrypt
age -d -i key.txt secrets.age

10. GitOps — ArgoCD & Flux

10.1. GitOps là gì?

GitOps (Weaveworks 2017) — pattern dùng Git làm single source of truth cho infrastructure + app:

1. Mọi desired state declarative trong Git (K8s YAML, Helm chart, Kustomize).
2. Agent trong cluster pull from Git → apply.
3. Drift detection: agent compare Git ↔ cluster, alert/auto-fix.
4. Rollback = git revert.

10.2. Push vs Pull

10.3. ArgoCD

# Cài ArgoCD
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# Get admin password
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

# Port-forward UI
kubectl port-forward svc/argocd-server -n argocd 8080:443
# https://localhost:8080

# Application manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/me/k8s-manifests.git
    targetRevision: main
    path: manifests/production
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true              # delete resource khi xóa từ Git
      selfHeal: true           # revert manual changes về Git state
    syncOptions:
      - CreateNamespace=true

Với automated.selfHeal: true: nếu ai kubectl edit trên cluster, ArgoCD revert về Git state trong vài giây.

10.4. App-of-apps pattern

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: bootstrap
spec:
  source:
    repoURL: https://github.com/me/k8s-manifests.git
    path: bootstrap/                   # folder chứa Application manifests khác
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated: {}

1 Application "bootstrap" deploy nhiều Application khác → manage cluster declaratively.

10.5. Flux v2

# Install
flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=fleet-infra \
  --branch=main \
  --path=clusters/prod \
  --personal

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/me/myapp-config
  ref:
    branch: main

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: myapp
  namespace: flux-system
spec:
  interval: 5m
  path: ./manifests
  prune: true
  sourceRef:
    kind: GitRepository
    name: myapp

10.6. ArgoCD vs Flux

Cả 2 đều CNCF graduated. ArgoCD UI tốt hơn → phổ biến hơn ở team có nhiều app. Flux đơn giản hơn cho infrastructure.

11. Bài tập

1. 12-Factor audit: review 1 app bạn maintain. Check 12 factors. Yếu tố nào violate? Plan fix.
2. Ansible inventory + ping: setup 3 VM (cloud hoặc local). Ansible inventory. ansible all -m ping.
3. Web server playbook: viết playbook install nginx + custom config + start service. Test idempotent (chạy 2 lần OK).
4. Roles: refactor playbook trên thành role. Tạo role thứ 2 (postgres). site.yml combine.
5. Ansible Vault: encrypt vars file chứa db password. Run playbook với --ask-vault-pass.
6. Vault dev: setup Vault dev mode. Store secret (db_password). Read qua API.
7. Vault AppRole: tạo role "myapp", policy chỉ read 1 secret. Đọc secret từ Node.js/Python qua AppRole login.
8. Dynamic DB credential: setup Vault database secrets engine với Postgres. Verify mỗi vault read tạo user mới với TTL.
9. AWS Secrets Manager: store DB credential. Lambda đọc secret. Setup auto-rotation Lambda.
10. External Secrets Operator: cài ESO trên K8s. Tạo ExternalSecret sync từ AWS Secrets Manager → K8s Secret.
11. Sealed Secrets: encrypt secret bằng kubeseal. Commit sealed-secret.yaml vào Git. Apply, verify Secret được tạo.
12. SOPS: encrypt secrets.yaml với SOPS + age. Decrypt khi cần. Integrate với Helm Secrets.
13. ArgoCD setup: cài ArgoCD, tạo Application trỏ Git repo. Push commit → verify ArgoCD auto-sync.
14. Self-heal demo: trong ArgoCD setup ở trên, kubectl edit deployment để thay đổi replicas. Watch ArgoCD revert về Git state.
15. Flux compare: setup tương tự với Flux. So sánh ergonomics.

12. Quiz