Lesson 13 — HA & Autoscaling

Scale the API under load. Survive node drains. Blue/green deploys with one Helm value.

The Module-5 arc

HPA in 30 seconds

HorizontalPodAutoscaler watches a metric and adjusts the Deployment’s replicas between minReplicas and maxReplicas to keep the metric at target. Two metric sources:

• resource — CPU or memory from metrics-server.
• external / pods — custom metrics (e.g., our http_requests_total from L12 via prometheus-adapter).

HPA on CPU (the boring default)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api }
spec:
  scaleTargetRef: { kind: Deployment, name: url-shortener-api }
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Resource
    resource: { name: cpu, target: { type: Utilization, averageUtilization: 70 } }

Target 70% CPU utilisation, scale 2–8. Requires requests.cpu set on the Deployment — otherwise “utilisation” is undefined and the HPA does nothing.

HPA on a custom metric (the right default)

  metrics:
  - type: Pods
    pods:
      metric: { name: http_requests_per_second }
      target: { type: AverageValue, averageValue: "100" }

“100 rps per pod” reads directly from real traffic. Requires the prometheus-adapter to expose http_requests_per_second via the Kubernetes metrics API. CPU lags traffic by 30–60 s on bursty workloads; request rate doesn’t.

PDB — protect capacity during drains

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api }
spec:
  minAvailable: 50%
  selector: { matchLabels: { app: api } }

When a node drains, kubectl drain honours the PDB and refuses to evict pods that would drop available count below 50%. Without a PDB, a rolling node upgrade can take you to 0 replicas.

topologySpreadConstraints — don’t co-locate replicas

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector: { matchLabels: { app: api } }

Prefer to put pods in different AZs; if not possible, still schedule (ScheduleAnyway vs DoNotSchedule). One zone failure no longer eats all your replicas.

Blue/green via Helm values

# values-prod.yaml
api:
  image:
    repository: ghcr.io/me/url-shortener-api
    tag: blue           # flip to green for cut-over
service:
  api:
    selector: { app: api, color: blue }   # also flipped

Two Deployments (api-blue, api-green), one Service whose selector picks one. The flip is --set api.color=green. Rollback is --set api.color=blue. No Ingress change; no DNS change.

Picking minReplicas and maxReplicas

• minReplicas: 2 minimum, always, in prod. One pod = one node failure = downtime.
• maxReplicas should cover 2× peak observed traffic with headroom — but not arbitrarily large. maxReplicas × requests.cpu is the upper bound on cluster capacity you can reserve.
• If scale-up hits cluster capacity before reaching maxReplicas, you have a cluster-autoscaler problem, not an HPA problem.

requests vs limits for the HPA

HPA reads CPU utilisation = used / requests. If requests.cpu = 100m and the pod uses 500m, utilisation = 500% — HPA scales out aggressively. If requests.cpu = 1000m and the pod uses 500m, utilisation = 50% — HPA may scale in.

Tune requests to the steady-state usage, then HPA’s averageUtilization is meaningful.

The L13 diff

sample-app/helm/templates/
  api-deployment.yaml    # + topologySpreadConstraints, + color label, split blue/green
  worker-deployment.yaml # + topologySpreadConstraints
  hpa.yaml               # NEW: HPA on api, CPU + request-rate metrics
  pdb.yaml               # NEW: PDB minAvailable=50% on api + worker
  api-service.yaml       # selector includes color: {{ .Values.api.color }}
sample-app/helm/values.yaml
  api: { color: blue, minReplicas: 2, maxReplicas: 8 }
sample-app/helm/values-prod.yaml
  api: { color: blue }   # flip here for cut-over

What we’re not doing

• Cluster autoscaling — Karpenter / cluster-autoscaler is the next layer up. HPA works within fixed node capacity.
• Stateful workloads — Postgres autoscaling requires storage-aware primitives (PVC resizing, primary/replica failover).
• Canary or A/B traffic splitting — blue/green is the simpler primitive. True canary (5% to green, metric-gated ramp) is covered in L13 homework via nginx-ingress canary-weight.

Production checklist

Recap & what’s next

• HPA on request rate beats HPA on CPU — traffic precedes CPU saturation by tens of seconds.
• PDB + topology spread is non-negotiable in prod — together they protect against both voluntary and zonal failures.
• Blue/green is a values flip — no Ingress change, no DNS change, 30-second rollback.