Lab 09 — Deploy the shortener to a local kind cluster

Lesson 09 · Cloud Infrastructure & DevOps · ~60 min

Goal: by the end of this lab you have a kind cluster running the API + worker + Postgres, you have used kubectl to inspect a misconfigured pod, you have watched a rolling update happen, and you have hit the API through kubectl port-forward.

Prerequisites

• Lesson 08’s lab done (the Terraform-managed VM exists; we keep it around as the previous state but won’t use it in this lab).
• Docker Desktop (or equivalent) running locally.
• kind — install with brew install kind (macOS/Linux) or download from kind.sigs.k8s.io.
• kubectl — install with brew install kubectl or sudo apt-get install kubectl.

Verify your tools:

kind --version       # kind v0.22 or newer
kubectl version --client  # v1.29 or newer

Step 1 — Create a kind cluster

kind create cluster --name shortener
kubectl cluster-info --context kind-shortener

Expected output: Kubernetes control plane is running at https://127.0.0.1:<port>. The cluster takes about 30 seconds to spin up.

kubectl get nodes
# NAME                      STATUS   ROLES           AGE   VERSION
# shortener-control-plane   Ready    control-plane   30s   v1.29.x

Step 2 — Tour the manifests in sample-app/k8s/

Open cloud-devops/sample-app/k8s/ in your editor. Eight files define the complete stack:

namespace.yaml — creates the shortener namespace. Apply this first; all other manifests reference it via metadata.namespace: shortener.

configmap.yaml — holds non-sensitive config: LOG_LEVEL, WORKER_INTERVAL_S, POSTGRES_DB, POSTGRES_USER. Injected as env vars into the API and worker pods via envFrom: configMapRef:.

secret.yaml — holds the full DATABASE_URL and POSTGRES_PASSWORD. Uses stringData (plaintext, cluster encodes it). This is safe for local kind; for any shared environment use External Secrets Operator instead.

postgres-statefulset.yaml + postgres-service.yaml — a one-replica StatefulSet for Postgres and a headless Service (clusterIP: None) that gives it a stable DNS name (postgres-0.postgres.shortener.svc.cluster.local). Key design choices:

• Why a StatefulSet? Stable network identity and ordered startup/shutdown. Even for a single replica the semantics matter for the API’s readiness probe, which checks database connectivity.
• Why clusterIP: None? A headless Service returns pod IPs directly in DNS instead of a virtual IP — the required pattern for StatefulSets so individual pods are addressable.

api-deployment.yaml — two replicas of the API pod with both probes:

• Readiness probe (/ready): checks whether the DB is reachable and migrations are complete. Pods fail readiness until Postgres is up, so they never receive traffic too early.
• Liveness probe (/healthz): checks whether the process is alive. Only restarts on a genuine crash, not on a DB outage.
• Why replicas: 2? The API is stateless; two replicas allow a rolling update with no downtime (one pod stays in service while the other is replaced).

api-service.yaml — a ClusterIP Service that makes the API reachable at http://shortener-api/ from other pods in the namespace.

worker-deployment.yaml — one replica of the worker with a liveness probe on /healthz.

• Why replicas: 1? The worker is a singleton task processor. Multiple replicas would process the same expiry events in parallel and produce duplicate work.

Step 3 — Apply the manifests

kubectl apply -f cloud-devops/sample-app/k8s/
kubectl -n shortener get all

Wait for Postgres to become Ready first. The API pods will show CrashLoopBackOff or 0/1 Running until Postgres is Ready — this is expected because the readiness probe checks database connectivity.

# Watch pods come up (Ctrl-C to stop)
kubectl -n shortener get pods -w

Expected final state:

NAME                              READY   STATUS    RESTARTS
postgres-0                        1/1     Running   0
shortener-api-<hash>-<hash>       1/1     Running   0
shortener-api-<hash>-<hash>       1/1     Running   0
shortener-worker-<hash>-<hash>    1/1     Running   0

Step 4 — Watch a rolling update

In one terminal, watch pods:

kubectl -n shortener get pods -w

In another terminal, trigger a rolling update by changing the image tag:

kubectl -n shortener set image deploy/shortener-api \
  api=ghcr.io/REPLACE_ME/shortener-api:0.4.1
kubectl -n shortener rollout status deploy/shortener-api

You will see new pods start while old pods are terminated one at a time — at least one pod stays in service throughout. To roll back:

kubectl -n shortener rollout undo deploy/shortener-api

Step 5 — Misconfigure a label, debug the breakage

This step exercises the most common K8s day-one debugging puzzle: a label-selector mismatch that silently empties a Service’s endpoints.

# Break the selector deliberately
kubectl -n shortener patch deploy shortener-api --type=json \
  -p='[{"op":"replace","path":"/spec/template/metadata/labels/app","value":"shortener-api-WRONG"}]'

# The Service now has no matching pods
kubectl -n shortener get endpoints shortener-api
# Endpoints will be empty: <none>

# Describe the Service to see selector vs endpoints
kubectl -n shortener describe svc shortener-api

Notice the mismatch between the Service’s Selector and the pods’ actual labels. Now revert:

kubectl -n shortener patch deploy shortener-api --type=json \
  -p='[{"op":"replace","path":"/spec/template/metadata/labels/app","value":"shortener-api"}]'

# Endpoints re-populate within a few seconds
kubectl -n shortener get endpoints shortener-api

Step 6 — Hit the API through port-forward

kubectl -n shortener port-forward svc/shortener-api 8080:80 &

curl -sS http://localhost:8080/healthz
# Expected: {"ok": true}

curl -sS -X POST http://localhost:8080/shorten \
  -d '{"url":"https://example.com"}' \
  -H 'Content-Type: application/json'
# Expected: {"short": "https://..."}

# Kill port-forward when done
kill %1

Step 7 — Tear down

kubectl delete -f cloud-devops/sample-app/k8s/
kind delete cluster --name shortener

All resources are removed. The kind cluster and its Docker containers are gone; your laptop returns to its previous state.

What you should leave with

• The shortener runs on a real Kubernetes cluster (locally). You can kubectl apply it, watch a rolling update, debug a label-selector mismatch, and curl through port-forward.
• Muscle memory for the four day-one objects: Deployment, Service, ConfigMap, Secret.
• Understanding of why liveness and readiness probes must point at different endpoints.
• Understanding of why Postgres uses a StatefulSet with a headless Service, while the API uses a regular Deployment with a ClusterIP Service.

Troubleshooting