Lesson 13 HA & Autoscaling
Theme:

Lab 13 — Drive traffic, watch HPA scale up; drain a node, watch PDB hold; flip blue/green with one Helm value

Lesson 13 · Cloud Infrastructure & DevOps · ~60 min

Prereqs

L12 lab complete: Prometheus, Grafana, and Loki running in the monitoring namespace on kind; the URL shortener observable (golden-signals dashboard live). You also need:

Step 1 — Upgrade the chart with HA values

helm upgrade --install url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener \
  -f cloud-devops/sample-app/helm/values-prod.yaml \
  --set autoscaling.customMetric.enabled=false   # turn back on after prometheus-adapter
kubectl -n url-shortener get deploy,hpa,pdb

Expected output:

NAME                                    READY   UP-TO-DATE   AVAILABLE
deployment.apps/url-shortener-api-blue  3/3     3            3
deployment.apps/url-shortener-api-green 3/3     3            3
deployment.apps/url-shortener-worker    2/2     2            2

NAME                                       MINPODS   MAXPODS   REPLICAS
horizontalpodautoscaler.autoscaling/...api   3        12        3

NAME                                        MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
poddisruptionbudget.policy/url-shortener-api     50%         N/A               1
poddisruptionbudget.policy/url-shortener-worker   1          N/A               1

Step 2 — Install prometheus-adapter and enable the custom metric

helm install prometheus-adapter prometheus-community/prometheus-adapter \
  -n monitoring \
  --set prometheus.url=http://kps-kube-prometheus-stack-prometheus.monitoring.svc \
  --set prometheus.port=9090 \
  --set-string rules.custom='[{"seriesQuery":"http_requests_total{namespace!=\"\",pod!=\"\"}","resources":{"overrides":{"namespace":{"resource":"namespace"},"pod":{"resource":"pod"}}},"name":{"matches":"^(.*)_total$","as":"${1}_per_second"},"metricsQuery":"sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)"}]'

After install, wait ~60 seconds then verify the metric is visible:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/url-shortener/pods/*/http_requests_per_second" | jq .

Then enable it in the chart:

helm upgrade url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener \
  -f cloud-devops/sample-app/helm/values-prod.yaml
kubectl -n url-shortener describe hpa url-shortener-api | grep -A2 "Metrics:"

Step 3 — Drive load and watch HPA scale up

Open two terminals.

Terminal 1 — start the load:

kubectl -n url-shortener port-forward svc/url-shortener-api 5000:80 &
hey -z 5m -c 50 http://localhost:5000/

Terminal 2 — watch the HPA react:

watch -n2 'kubectl -n url-shortener get hpa,deploy'

Within 60 seconds the HPA should show current usage above target and replicas climbing from 3 towards the cap (12). After load stops, watch replicas scale back down over the next ~5 minutes — that is the stabilizationWindowSeconds: 300 scale-down window preventing flapping.

Step 4 — Drain a node; watch PDB hold the line

On kind, simulate a node drain:

kubectl get nodes
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data

In a second terminal, watch pod evictions:

kubectl -n url-shortener get pods -o wide -w

With minAvailable: 50% and 3 replicas, Kubernetes will evict at most 1 pod at a time and wait for a replacement to reach Ready before evicting the next. You should never see two pods Terminating simultaneously.

Compare to the unprotected case: delete the PDB, drain again, and observe in your hey output that 5xx responses spike briefly during the drain window.

kubectl -n url-shortener delete pdb url-shortener-api
kubectl drain <NODE_NAME> --ignore-daemonsets --delete-emptydir-data --force
# ... observe 5xxs ...
kubectl uncordon <NODE_NAME>
# Reinstall PDB:
helm upgrade url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener -f cloud-devops/sample-app/helm/values-prod.yaml

Step 5 — Blue/green flip

Confirm only blue is serving:

kubectl -n url-shortener get pods -L color
kubectl -n url-shortener get svc url-shortener-api -o jsonpath='{.spec.selector}'
# {"app.kubernetes.io/component":"api","app.kubernetes.io/instance":"url-shortener","app.kubernetes.io/name":"shortener","color":"blue"}

Point green at a new image tag (or re-use the same tag for this lab):

helm upgrade url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener -f cloud-devops/sample-app/helm/values-prod.yaml \
  --set api.image.green.tag=0.4.1

Cut over to green:

helm upgrade url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener -f cloud-devops/sample-app/helm/values-prod.yaml \
  --set api.color=green
kubectl -n url-shortener get svc url-shortener-api -o jsonpath='{.spec.selector}'
# {"...", "color":"green"}
curl -s http://localhost:5000/   # served by green now

Rollback to blue:

helm upgrade url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener -f cloud-devops/sample-app/helm/values-prod.yaml \
  --set api.color=blue

Step 6 — Confirm both colours stayed up through the flip

Open the Grafana dashboard at http://localhost:3000 (admin / changeme). The “API — Golden Signals” dashboard’s request-rate panel should show a continuous line across the flip with no gap. The “5xx errors” panel should be flat zero. Both colours were running simultaneously — the flip is just a Service selector change.

Step 7 — Topology spread on a multi-node kind (optional)

Recreate kind with 3 worker nodes:

cat > kind-3nodes.yaml <<'EOF'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
kind create cluster --config kind-3nodes.yaml --name url-shortener-ha
# ... redeploy observability + url-shortener as in L12 ...
kubectl -n url-shortener get pods -o wide

Expect API pods spread across all three workers rather than clustered on one. Note: kind has a single zone label, so the topology.kubernetes.io/zone constraint has no effect — only the kubernetes.io/hostname constraint takes effect on kind.

Step 8 — Production checklist

Before claiming L13 done, confirm all of the following:

What you should leave with

A URL shortener that scales 3 → 12 under load, survives a node drain without 5xx errors, and can blue/green flip in 30 seconds. The entire cluster state is reproducible from helm upgrade alone — no manual patches, no state on the laptop.

Troubleshooting