Lesson 13 HA & Autoscaling
Theme:

Homework 13 — HA & Autoscaling

Lesson 13 · Cloud Infrastructure & DevOps · Three tasks

These three tasks deepen the HA and autoscaling foundation from the lab. Task 1 adds HPA behaviour policies for the worker. Task 2 implements canary traffic splitting as an alternative to blue/green. Task 3 stress-tests the PDB to understand when minAvailable: 100% is right and when it will block your node upgrades forever. You will need the working cluster from the lab.

Task 1 — Bound HPA behavior for the worker

The worker Deployment does not have an HPA today. Add one that scales on worker_jobs_processed_per_second rate: target 50 jobs/s per pod, minimum 1 replica, maximum 4 replicas.

  1. Add the HPA manifest (either as a new Helm template or as a plain kubectl apply YAML) with explicit behavior policies:
    • Scale-up: at most 100% of current replicas per 60 seconds.
    • Scale-down: at most 50% per 60 seconds.
    • Scale-down stabilisation window: 600 seconds (10 minutes — worker jobs are longer than API requests).
  2. Apply it and confirm the HPA is registered: kubectl get hpa -n url-shortener.
  3. Drive sustained worker load (your choice of mechanism; even kubectl exec + a loop creating jobs works). Watch the HPA scale up to maxReplicas: 4 during load and back down over ~10 minutes after load stops.

Self-check: kubectl get hpa -n url-shortener shows two entries (api and worker). Under sustained job load the worker HPA reaches maxReplicas: 4. After load stops, replicas drop over ~10 minutes, not immediately.

Task 2 — Canary instead of blue/green

Blue/green is a 100%-at-once flip. Read the nginx-ingress canary-weight annotation docs. Then implement canary traffic splitting:

  1. Create an additional Ingress with annotations nginx.ingress.kubernetes.io/canary: "true" and nginx.ingress.kubernetes.io/canary-weight: "10" that points at a Service selecting only the green colour. The main Ingress (unchanged) points at the blue-serving Service.
  2. Apply and verify: run 100 requests and count how many land on the green pods (check pod logs or the color label). Expect ~10 green, ~90 blue.
  3. Write a 6-line summary describing how this approach differs from the lab’s single-Service selector flip. What are the trade-offs? When would you choose canary over blue/green?

Self-check: 100 requests to url-shortener.example.com land on green ~10% of the time. Your summary cites at least one concrete scenario for each approach.

Task 3 — PDB stress test

  1. Set pdb.api.minAvailable: "100%" via helm upgrade --set pdb.api.minAvailable=100%. Then attempt to kubectl drain a node. Document what happens (expected: drain hangs and prints eviction-blocked events).
  2. Restore pdb.api.minAvailable: "50%" (the default) and repeat the drain. Observe that it proceeds without hanging.
  3. Write 5 bullets on when minAvailable: 100% is the right choice vs when it will cause problems. Be specific about workload types.

Self-check: you can reproduce the block and the resolution with the two helm upgrade invocations above. Your 5 bullets cite at least one concrete service class on each side of the trade-off.

View solutions →