Lesson 13 HA & Autoscaling
Theme:

HA & Autoscaling

Lesson 13 · Cloud Infrastructure & DevOps · ~15 min read

1. From “running” to “highly available”

Lessons 09–11 got the URL shortener running on Kubernetes with TLS. One replica was enough for the demo. Production needs three properties that one replica cannot provide:

The Kubernetes primitives that deliver all three are: HorizontalPodAutoscaler, PodDisruptionBudget, topologySpreadConstraints, and two parallel Deployments controlled by a Service selector. Module 5 deploys them on top of L12’s observability stack so the autoscaler can read real traffic metrics, not just guess from CPU.

2. HPA: the control loop

HorizontalPodAutoscaler (autoscaling/v2) is a control loop that runs every 15 seconds by default. On each tick:

  1. Read the configured metric across all pods matching the Deployment’s selector.
  2. Compute desired replicas = current × (current_metric / target), rounded.
  3. Clamp to [minReplicas, maxReplicas].
  4. If desired ≠ current and the behavior policies allow it, patch the Deployment’s .spec.replicas.

Three metric sources exist:

The L13 sample HPA uses both CPU and http_requests_per_second. Kubernetes takes the max of the desired-replicas across all metrics, so whichever signal is more stressed drives scale-out. This is the right default: CPU catches “the pod is computing”; request rate catches “the pod is receiving traffic”, which precedes CPU saturation by tens of seconds.

3. behavior: — scale-up and scale-down policies

Default HPA behaviour scales aggressively up and conservatively down. For most APIs this is wrong:

Bound both with spec.behavior:

behavior:
  scaleUp:
    stabilizationWindowSeconds: 60
    policies:
      - type: Percent
        value: 50
        periodSeconds: 60
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 25
        periodSeconds: 60

This reads: “scale up at most 50% every 60 seconds, after smoothing over 60 seconds. Scale down at most 25% every 60 seconds, after smoothing over 5 minutes.” The 5-minute scale-down window prevents flapping when traffic oscillates near a threshold.

4. PDB — survival during voluntary disruptions

PodDisruptionBudget is the contract that says “no matter what you (the platform) do, keep at least N of my pods available.” It applies to voluntary disruptions: node drains for upgrades, evictions for cluster-autoscaler reclamation, etc. Involuntary disruptions like a node crash do not honour PDBs — that is what topologySpreadConstraints and replica count are for.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: api }
spec:
  minAvailable: 50%
  selector: { matchLabels: { app: api } }

Use either minAvailable (absolute or %) or maxUnavailable. Use minAvailable: 50% when replica count is variable (HPA-controlled); use absolute (minAvailable: 2) when you have a known fixed count.

Without a PDB, kubectl drain will evict all pods on a node. If your Deployment has 2 replicas and both happen to be scheduled on the same node, drain takes you to zero. PDB + topology spread together is the only correct shape.

5. topologySpreadConstraints — don’t put replicas in one basket

Without constraints, the Kubernetes scheduler picks the most resource-available node — often the same node, repeatedly. The fix:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector: { matchLabels: { app: api } }
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: ScheduleAnyway
  labelSelector: { matchLabels: { app: api } }

Two constraints: spread across zones (AZ failure tolerance) and across hosts (node failure tolerance). maxSkew: 1 means the count per zone or host can differ by at most 1. ScheduleAnyway is the right default — if the constraint cannot be honoured (cluster has 1 zone, 1 node), schedule anyway rather than block. DoNotSchedule is for hard constraints — use sparingly.

A kind cluster has one zone and one node; the constraints still parse and apply but have no effect. On real EKS, GKE, or AKS with ≥ 2 zones, they do exactly what you want.

6. Blue/green via Helm values

Two simultaneous Deployments, one Service. The Service’s selector picks one colour. The flip is a single Helm value:

# values-prod.yaml
api:
  color: blue          # the currently-serving colour
  image:
    blue:  { tag: 1.2.3 }
    green: { tag: 1.3.0 }

Template:

{{- range $color := list "blue" "green" }}
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: url-shortener-api-{{ $color }} }
spec:
  selector: { matchLabels: { app: api, color: {{ $color }} } }
  template:
    metadata: { labels: { app: api, color: {{ $color }} } }
    spec:
      containers:
      - name: api
        image: "{{ $.Values.api.image.repository }}:{{ (index $.Values.api.image $color).tag }}"
{{- end }}

And the Service:

spec:
  selector: { app: api, color: {{ .Values.api.color }} }

The cut-over is helm upgrade --set api.color=green. Rollback is helm upgrade --set api.color=blue. Both colours run concurrently — extra cost in exchange for a sub-30-second rollback. For low-traffic apps, accept the cost; for high-traffic, consider scaling down the inactive colour to 1 replica via a replicas.{{ $color }} value.

7. Where this stops and a service mesh begins

Blue/green is a discrete switch — all traffic flips at once. Sometimes you want canary: 5% to green, watch the metrics, ramp to 25%, 50%, 100%. That requires traffic-splitting at the L7 level — either via the ingress controller’s annotations (nginx canary-weight) or via a service mesh (Istio VirtualService, Linkerd TrafficSplit). Both are referenced in L13 homework but out of scope for the core lab.

Argo Rollouts is the productionised version of this idea — it ships with automated metric analysis (“if p95 latency goes up 10% during canary, abort”) and is the natural pair with the ArgoCD covered in L14. The pattern is shown in L14 theory as the next step beyond the blue/green primitive.

Common pitfalls

  1. HPA with no requests.cpu set → utilisation is undefined → HPA does nothing silently.
  2. HPA minReplicas: 1 in prod → one pod reschedule = downtime.
  3. PDB present but replicas: 1 → drain blocks forever because evicting the only pod violates the budget.
  4. topologySpreadConstraints with DoNotSchedule on a 1-zone cluster → pods Pending forever.
  5. Blue/green flip without scaling down the previous colour → permanent 2× cost for both Deployments at full maxReplicas.
  6. Forgetting that HPA + GitOps (L14) creates a conflict: git is the source of truth for replicas. ArgoCD will revert HPA’s scale-up patch. Solution: tell ArgoCD to ignore .spec.replicas (covered in L14).

References