Homework 13 Solutions — HA & Autoscaling

Lesson 13 · Cloud Infrastructure & DevOps

Solution to Task 1 — Worker HPA with behavior

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: url-shortener-worker
  namespace: url-shortener
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: url-shortener-worker
  minReplicas: 1
  maxReplicas: 4
  metrics:
  - type: Pods
    pods:
      metric:
        name: worker_jobs_processed_per_second
      target:
        type: AverageValue
        averageValue: "50"
  behavior:
    scaleUp:
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Percent
          value: 50
          periodSeconds: 60

The prometheus-adapter rule installed in L13 already exposes worker_jobs_processed_per_second because the source counter (worker_jobs_processed_total) follows the _total suffix naming convention that the rule matches.

Solution to Task 2 — Canary via nginx-ingress annotations

# Main Ingress — unchanged, points at the api Service (which selects the active colour)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: url-shortener-api
  namespace: url-shortener
spec:
  ingressClassName: nginx
  rules:
  - host: url-shortener.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: url-shortener-api
            port: {number: 80}
---
# Canary Ingress — same host, sends 10% to a green-only Service
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: url-shortener-api-canary
  namespace: url-shortener
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: url-shortener.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: url-shortener-api-green   # a Service selecting color=green only
            port: {number: 80}
---
# Green-only Service for the canary
apiVersion: v1
kind: Service
metadata:
  name: url-shortener-api-green
  namespace: url-shortener
spec:
  selector:
    app.kubernetes.io/component: api
    color: green
  ports: [{port: 80, targetPort: 8080}]

Six-line summary:
Blue/green flips 100% of traffic at once — fast, simple, but a bug on green hits every user before you notice. Canary sends a small fraction (e.g., 10%) to the new version — slower cut-over, more infrastructure, but observability gets a small-population signal before full rollout. Use canary when you have automated metric comparison (Argo Rollouts, Flagger) and a user pool large enough to make 10% statistically meaningful. Use blue/green for low-traffic internal services where 10% would only be ~1 user/minute and a fast rollback matters more than a gradual ramp. The key failure mode of canary without metrics analysis is the same as blue/green: you don’t notice the error until the dashboard pings you.

Solution to Task 3 — PDB stress test

What happens with minAvailable: 100%: kubectl drain prints:

error when evicting pods: Cannot evict pod as it would violate the pod's disruption budget.

and blocks indefinitely. The HPA may scale up to replace the pod that would be evicted, but because the PDB requires all current replicas to be available, the drain can never find a safe window to evict even one pod. kubectl get events --field-selector reason=EvictionBlocked -n url-shortener confirms the cause.

Bullets — when minAvailable: 100% is the right choice:

1. One-replica admin tooling where you would rather block the drain than risk a brief downtime (small batch coordinators, scheduler processes, webhook validators that back every API request).
2. Stateful jobs mid-operation that cannot be safely interrupted and cannot checkpoint — e.g., a long-running data migration that writes to an external system with no idempotency.
3. Compliance-driven setups where any availability dip is a reportable event regardless of duration (banking, healthcare, regulated financial APIs).

Bullets — when minAvailable: 100% is wrong:

1. Stateless web APIs with ≥ 2 replicas — blocking drain means you cannot upgrade cluster nodes without coordinated downtime, negating the value of rolling upgrades.
2. HPA-controlled workloads — 100% of a variable replica count is a moving target. At peak the HPA may hold 8 replicas; the drain can never proceed because it can never guarantee all 8 stay up.
3. Any service whose clients already retry on 5xx — a well-behaved internal API tolerates one pod restart; the user cost is one client retry, not a visible outage.