Lesson 09 Kubernetes Fundamentals
Theme:

Kubernetes Fundamentals

Lesson 09 · Cloud Infrastructure & DevOps · ~14 min read

1. Why we leave the single VM behind

Modules 1–3 landed the URL shortener on a t3.micro managed by Terraform. docker-compose handles the multi-container stack; Caddy handles TLS; a single terraform apply from any workstation reproduces the whole setup in under two minutes. That is genuinely good, and for a low-traffic project it is enough.

It stops being enough the moment you need more than one box. Fault tolerance requires at least two instances behind a load balancer. A zero-downtime deploy means bringing new containers up before draining old ones — and docker-compose has no native mechanism for that across hosts. Adding a second VM manually reintroduces everything Terraform solved: click-driven setup, state stored on one machine, no review process for infrastructure changes.

The natural next step is not “two VMs with a load balancer” managed by hand. It is a cluster runtime that abstracts away the individual hosts entirely. Kubernetes is that runtime. It is the de-facto cluster standard across every major cloud and in the majority of large on-premises environments. This lesson is the on-ramp: enough K8s to host the URL shortener on your laptop and reason confidently about what is running.

2. The cluster mental model

A Kubernetes cluster has two layers. The control plane runs the API server, the scheduler, the controller manager, and etcd (the distributed key-value store that holds all cluster state). The worker nodes run the kubelet agent, a container runtime (typically containerd), and the kube-proxy network component. On a managed cluster (EKS, GKE, AKS) the control plane is invisible to you; on kind, everything runs inside Docker containers on your laptop.

You interact with the cluster by talking to the API server through kubectl. The interaction is always declarative: you submit a manifest describing the desired state (a Deployment with 2 replicas of the API, a Service pointing to those replicas), and the controllers in the control plane continuously reconcile actual state toward that desired state.

This reconciliation loop is the single mental model that explains every K8s behaviour. Why does a deleted pod come back? Because a Deployment controller owns it and sees that actual replicas (0) differs from desired replicas (2), so it creates a new pod. Why does changing an image tag trigger a rolling update? The Deployment controller detects the template hash changed, creates a new ReplicaSet, scales it up while scaling the old one down. Nothing in K8s is imperative in the way “restart this service” is imperative in systemd — even kubectl rollout restart just writes an annotation that the controller observes and acts on.

3. The four day-one objects

Pod. A Pod is a group of one or more containers that share a network namespace (they can reach each other on localhost) and a lifecycle (they start and stop together). It is the smallest deployable unit in K8s. In practice you almost never write a Pod manifest directly; a higher-level controller creates and owns Pods for you. If you create a naked Pod and the node it runs on fails, nothing restarts it — there is no owner that would notice the absence.

Deployment. A Deployment is a controller that owns a ReplicaSet, which in turn owns Pods. The Deployment manifest has two key fields: template (what one pod looks like — image, ports, env, probes) and replicas (how many copies to keep alive). When you change template (typically by bumping the image tag), the Deployment creates a new ReplicaSet and performs a rolling update: it brings new pods up a few at a time and tears old pods down only after the new ones pass their readiness probes. Rolling back means pointing the Deployment back at the previous ReplicaSet, which K8s does in seconds.

Service. A Service is a stable virtual IP address and DNS name that fronts a label-selected set of Pods. The DNS name is <service-name>.<namespace>.svc.cluster.local, but within the same namespace you can just use <service-name>. When the worker pod needs to call the API, it hits http://shortener-api/shorten — the cluster’s DNS resolves shortener-api to the Service’s virtual IP, and kube-proxy load-balances across the healthy backend Pods.

The critical insight about Services: they do not route by pod name or IP. They route by label selector. If a Deployment’s pod template carries labels: {app: shortener-api} and the Service’s selector is {app: shortener-api}, the Service will pick up exactly those pods. If you mis-type one label, the Service’s endpoint list is empty and every request gets a connection-refused. This is the single most common day-one debugging puzzle in K8s.

ConfigMap and Secret. Both are key-value stores that inject configuration into pods. A ConfigMap holds non-sensitive data: log levels, feature flags, timer intervals. A Secret holds credentials: DB passwords, API tokens, TLS private keys. Both can be consumed two ways: as environment variables (using envFrom: configMapRef: or secretRef:) or as mounted files (using volumes: + volumeMounts:). The cluster base64-encodes Secret values at rest — note that base64 is encoding, not encryption. For real encryption at rest you configure the cluster’s encryption provider separately, or use an external secrets manager.

4. Probes: liveness vs readiness

Common pitfall: wiring liveness to dependency checks

Kubernetes has two probe types with opposite failure semantics. A liveness probe answers “is this container still alive?” — if it fails, K8s restarts the container. Use liveness for crash detection: a deadlocked Python event loop, a hung file descriptor, a process that has stopped responding. Point it at a cheap, always-available endpoint like /healthz that returns 200 as long as the process itself is running.

A readiness probe answers “is this container ready to serve traffic?” — if it fails, K8s removes the pod from the Service’s endpoint list without restarting it. Use readiness for startup warmup (model loading, cache priming) and dependency checks (DB reachable? migrations done?).

Wiring liveness to a dependency check is a famous outage pattern: a transient database blip causes all pods to fail their liveness probe simultaneously; K8s restarts all of them; the restart storm prevents the DB from recovering; the cluster enters a crash loop. Always point the two probes at different endpoints with different semantics.

5. ConfigMaps and Secrets in depth

The usual first question about env-var injection vs file mounts is: which one should I use? The practical answer is: env vars for simple key-value config (log level, feature flags, hostnames); file mounts for multi-line config files, TLS certificates, or any value that benefits from hot-reload.

File mounts have a significant advantage: when you update a ConfigMap or Secret and the kubelet detects the change, it refreshes the mounted files without restarting the pod. Applications that watch their config files can pick up the new value live. Environment variables are set at container start and never change until the next restart — so to propagate an env-var config change you must roll the Deployment.

A real-world rule that applies to every environment: never commit Secret manifests with real credentials to git. The stringData field used in this lesson’s manifests writes plaintext values that the cluster encodes. That is fine for a local kind cluster where the only viewer is you. For staging and production, use External Secrets Operator (pulls from AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) or Sealed Secrets (encrypts the manifest so it can be stored in git). Both approaches are covered in a later module.

6. kind: Kubernetes in a Docker container

kind (Kubernetes IN Docker) runs a full, conformant Kubernetes cluster entirely inside Docker containers. The control plane node is a container; each worker node is a container; the cluster’s own container runtime (containerd) runs inside those containers. Spinning a cluster up takes about 30 seconds; tearing it down takes five. No cloud account, no VM, no persistent disk — the whole thing lives in Docker on your laptop.

The one catch that trips up every new kind user: kind does not share Docker’s image cache with your laptop. If you build an image with docker build, the image lives in your local Docker daemon but is invisible to the cluster. You have two options: push the image to a registry (GHCR, Docker Hub, a local registry) and let the kubelet pull it, or load it directly with kind load docker-image <name> --name <cluster>. For this lesson the API and worker images come from GHCR (already set up by Module 2’s CI), so pulling from the registry is the default path; kind load is only needed if you rebuild locally.

7. The shortener on Kubernetes

The end state of Lesson 09 is a single Kubernetes namespace (shortener) that contains everything the URL shortener needs to run:

kubectl apply -f sample-app/k8s/ brings the whole stack up in one command. kubectl port-forward svc/shortener-api 8080:80 -n shortener exposes the API on your laptop at localhost:8080. The total YAML is about 150 lines across eight files — readable, diff-able, review-able, and the same files Helm will template in Lesson 10.

Common pitfalls

  1. Label-selector mismatch. Service has zero endpoints; requests get connection-refused. Fix: compare Service.spec.selector with Deployment.spec.template.metadata.labels — they must be identical. Use kubectl describe svc <name> to see the Endpoints line.
  2. Probes pointing at the same endpoint. Wiring liveness to a dependency-check endpoint causes restart storms during dependency outages. Always use separate endpoints with different semantics.
  3. Image pull failure on kind. You forgot to push to GHCR or run kind load docker-image. Check kubectl describe pod <name> for ImagePullBackOff and read the events section.
  4. Forgetting imagePullPolicy: IfNotPresent when iterating locally. Without this, the kubelet tries to re-pull the image from the registry on every restart, even if the local build is newer.
  5. kubectl apply with no -n flag. Resources land in the default namespace and you can’t find them. All manifests in sample-app/k8s/ set metadata.namespace: shortener explicitly; apply the namespace manifest first so K8s accepts the rest.

References