Solutions — Homework 09

Lesson 09 · Cloud Infrastructure & DevOps · Worked solutions with common mistakes

Task 1 — PodDisruptionBudget for the API

Create cloud-devops/sample-app/k8s/api-pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: shortener-api
  namespace: shortener
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: shortener-api

Apply:

kubectl apply -f cloud-devops/sample-app/k8s/api-pdb.yaml
kubectl get pdb -n shortener

Expected output:

NAME            MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
shortener-api   1               N/A               1                     5s

With 2 replicas running and minAvailable: 1, K8s allows 1 voluntary disruption at a time. If a node drain would evict the second pod while only 1 is already running, the drain blocks until the evicted pod is rescheduled and becomes Ready.

Common mistake — `minAvailable: 100%` on a 2-replica Deployment

Using minAvailable: 100% means all replicas must remain available at all times. This prevents any voluntary disruption — including node drains needed for cluster upgrades — which defeats the purpose of having a PDB. The standard for a 2-replica setup is minAvailable: 1 (one pod can be taken offline) or equivalently maxUnavailable: 1. Reserve minAvailable: 100% only for genuinely zero-disruption-tolerant workloads where you have enough replicas that removing one still leaves full capacity.

Task 2 — HorizontalPodAutoscaler

Install metrics-server on kind (required for HPA to read CPU metrics):

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Patch for kind: kubelet uses self-signed certs
kubectl -n kube-system patch deploy metrics-server --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

# Wait for metrics-server to become Ready (~30s)
kubectl -n kube-system rollout status deploy/metrics-server

Create cloud-devops/sample-app/k8s/api-hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: shortener-api
  namespace: shortener
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: shortener-api
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply and verify:

kubectl apply -f cloud-devops/sample-app/k8s/api-hpa.yaml
kubectl get hpa -n shortener
kubectl top pod -n shortener

Expected kubectl get hpa output (after metrics-server is ready):

NAME            REFERENCE                  TARGETS   MINPODS   MAXPODS   REPLICAS
shortener-api   Deployment/shortener-api   2%/70%    2         5         2

Common mistake — forgetting `--kubelet-insecure-tls` on `kind`

kind’s kubelet uses self-signed TLS certificates. Without the --kubelet-insecure-tls flag, metrics-server refuses the connection and its pod enters CrashLoopBackOff. As a result, kubectl top pod returns nothing and the HPA reports <unknown> for current CPU. This is a kind-specific quirk; managed clusters (EKS, GKE, AKS) use valid certs so the flag is not needed there.

Task 3 — Split readiness from liveness on the worker

Updated probe section in worker-deployment.yaml:

livenessProbe:
  exec:
    command: ["test", "-f", "/tmp/healthy"]
  initialDelaySeconds: 30
  periodSeconds: 30
readinessProbe:
  exec:
    command: ["test", "-f", "/tmp/ready"]
  initialDelaySeconds: 5
  periodSeconds: 5

The worker process should touch /tmp/healthy on startup (liveness signal: the event loop is running) and touch /tmp/ready only after it has successfully subscribed to its job queue (readiness signal: ready to consume work).

Verify both probes after apply:

kubectl describe deploy shortener-worker -n shortener | grep -A 5 Probe

# Simulate readiness failure
kubectl -n shortener exec deploy/shortener-worker -- rm /tmp/ready

# Watch READY column flip to 0/1 — no restart column change
kubectl -n shortener get pods -w

Justification (comment at top of the manifest): The worker’s liveness signal is “the process is alive and the event loop is not deadlocked” — a cheap, always-true check unless the process is genuinely broken. The readiness signal is “the worker has subscribed to its job queue and is ready to process messages.” Conflating the two means a transient queue connection blip would cause K8s to restart the worker (liveness failed) instead of merely de-routing it (readiness failed). Restart storms during dependency outages are a classic K8s reliability failure mode. The cost of maintaining two separate signals is one extra touch call in the worker startup path; the benefit is correct failure-mode isolation.

Common mistake — pointing both probes at the same endpoint

The temptation is to use /tmp/healthy for both probes “for simplicity.” This is exactly the antipattern this exercise exposes. If the queue connection drops, the liveness probe fails, K8s restarts the worker, the worker tries to reconnect, fails again because the queue is still down, and K8s restarts it again — exponential backoff aside, you get a restart storm that prevents any clean recovery. With separate probes: the readiness probe fails (worker is de-routed), the liveness probe stays green (process is alive), K8s does nothing but wait. When the queue recovers, readiness passes, the worker is re-routed. No restarts, no storm.

You have now closed the loop on Kubernetes day-one: raw manifests, rolling updates, probes, PDBs, and autoscaling. Lesson 10 — Helm — replaces every raw manifest you wrote in this lab with a single, parametrised chart. The probes, the PDB, the HPA all become values you flip per environment.