Lesson 11 Ingress + cert-manager + DNS
Theme:

Ingress + cert-manager + DNS

Lesson 11 · Cloud Infrastructure & DevOps · ~15 min read

1. From port-forward to a real URL

kubectl port-forward is the dev-loop default — fast, secure-by-default, no infrastructure. It tunnels traffic from a local port directly to a pod over the Kubernetes API server connection. It does not scale to “share a link with a colleague” or “let a webhook in” — the tunnel is bound to your laptop and dies when you close the terminal. For a public endpoint we need something different.

On Kubernetes that means three new pieces of cluster software working together: an Ingress controller that terminates external HTTP/S and routes it to Services by hostname and path; a certificate manager (cert-manager) that automates the ACME TLS certificate lifecycle; and a DNS controller (ExternalDNS) that watches Ingress objects and writes matching DNS records at your provider. All three are optional — but all three together give you a fully automated path from a Helm values change to a live, TLS-secured public URL.

2. Ingress and the controller

The Ingress resource is just config — it has no built-in behaviour. The behaviour comes from an Ingress controller that watches Ingress resources cluster-wide and configures itself accordingly. This design separates the routing specification (the Ingress resource, written by the app team) from the routing implementation (the controller, operated by the platform team). A cluster can run multiple controllers simultaneously, each identified by an ingressClassName.

ingress-nginx is the most common open-source option; AWS/GCP/Azure also offer cloud-native alternatives (ALB Controller, GKE Ingress, Application Gateway Ingress Controller). For this course we standardise on ingress-nginx: it runs identically on kind and managed clusters, has excellent documentation, and supports the full range of annotations needed for cert-manager and ExternalDNS integration.

The Ingress controller installs as a Deployment + a Service of type LoadBalancer. On a cloud cluster the LoadBalancer Service auto-provisions a cloud LB (an AWS NLB, GCP TCP LB, etc.); on kind we use cloud-provider-kind to assign a loopback IP. Once the LB IP is reachable, anything matching an Ingress rule routes through the controller to your backing Service. The single LB serves every app in the cluster — each app gets its own Ingress resource with a host: rule; the controller routes by HTTP Host header.

When you write ingressClassName: nginx in the spec and install ingress-nginx, the controller watches all Ingress objects whose ingressClassName matches and configures its internal nginx configuration accordingly. Adding a new Ingress resource causes a hot config reload in the controller — no downtime, no restarts.

3. cert-manager and ACME

cert-manager is a Kubernetes-native ACME client packaged as a set of CRDs and a controller. You install it (CRDs + controller), then define an Issuer (namespace-scoped) or ClusterIssuer (cluster-wide) that specifies an ACME endpoint and a solver. When you create a Certificate resource — or when cert-manager infers one from an Ingress with the cert-manager.io/cluster-issuer annotation and a tls: block — it runs the ACME challenge: places a token at a well-known URL on your Ingress (http-01) or in a DNS TXT record (dns-01), and on success stores the issued cert in a Secret. The controller monitors expiry and renews automatically about 30 days before the cert expires — you never touch a certificate file manually again.

Always start with Let’s Encrypt staging. Production has hard rate limits (50 certs per registered domain per week; 5 failed validations per hostname per hour). Misconfiguration during the first end-to-end test will burn your prod quota and leave you locked out for a week. Staging certs are unlimited — the deliberate signal that you haven’t promoted yet is the browser warning, which you click through during testing.

http-01 is the default and easiest solver. It requires the ACME server to reach your Ingress at http://<host>/.well-known/acme-challenge/<token>. This means your DNS must already point at the Ingress controller’s LB, and ingress-nginx must accept HTTP (it does by default on port 80). cert-manager injects a temporary solver Pod + Ingress route for the challenge path, the ACME server fetches it, and on success the cert is stored. The solver Pod and temporary Ingress route are cleaned up automatically.

dns-01 is required for wildcards and lets you avoid exposing port 80; it needs API credentials for your DNS provider. You define the credentials in a Kubernetes Secret and reference them in the ClusterIssuer. The trade-off is more setup (per-provider configuration) in exchange for wildcard support and no HTTP-80 requirement.

4. ExternalDNS

Manually creating DNS records that point at Ingress IPs is the second-most-annoying part of cluster ops (after secrets). ExternalDNS automates it: watch Ingress resources, look at their host: and the LB IP/hostname behind the controller, write the matching A/CNAME records at your DNS provider. When an Ingress is deleted or a host changes, ExternalDNS updates or removes the record (with policy: sync).

Configuration is provider-specific: a Helm chart with provider: cloudflare (or aws, google, azure, cloudns, ~30 others), credentials via a Secret, a domainFilters list so it only touches your domain, and a policy: sync or policy: upsert-only toggle. Use upsert-only initially for production — it adds and updates records but never deletes them, which prevents accidental outages if an Ingress is temporarily removed or renamed.

Together the three components form a self-contained automation loop. You commit a new release of your chart with ingress.enabled: true and a host value:

  1. Helm applies the new Ingress.
  2. ExternalDNS sees it, creates shortener.example.com → <LB IP> at your DNS provider.
  3. cert-manager sees the tls: block + annotation, creates a Certificate, runs http-01 against the now-resolvable hostname, stores the cert in a Secret.
  4. ingress-nginx mounts the Secret, serves HTTPS.

Total elapsed time on a warm cluster with working DNS propagation: ~60–90 seconds.

5. Local development on kind

cloud-provider-kind is a small project that implements the cloud-controller-manager interface for kind — it watches LoadBalancer Services and binds them to host-loopback IPs (127.0.0.x). After sudo cloud-provider-kind & runs in the background, kubectl get svc -n ingress-nginx shows an EXTERNAL-IP you can curl from the host. The key steps: install cloud-provider-kind via go install, start it with sudo (it binds privileged ports), then install ingress-nginx and watch the Service flip from <pending> to a real IP.

ExternalDNS doesn’t work locally (no real DNS provider), so the lab uses /etc/hosts to point a fake domain (shortener.local.test) at the loopback IP. cert-manager’s http-01 challenge needs the ACME server to reach your cluster over the internet, which a kind cluster cannot satisfy; the cert will stay in Ready: False on kind unless you expose the cluster publicly (e.g., via a tunnel). The lab acknowledges this — the kind path validates the chart rendering and ingress routing; the full http-01 flow runs on a real cluster as the primary lab mode.

6. Promoting staging to prod issuer

Once staging works end-to-end — cert renews, ingress serves, ExternalDNS holds the record, and you have been watching it for at least a few hours — promote in two steps:

  1. Add a letsencrypt-prod ClusterIssuer with the production ACME server URL (https://acme-v02.api.letsencrypt.org/directory).
  2. Change the chart’s Ingress annotation to cert-manager.io/cluster-issuer: letsencrypt-prod and run helm upgrade. The old staging cert Secret becomes orphaned — delete it (kubectl delete secret shortener-tls -n shortener) and cert-manager re-issues the same name from the prod issuer within ~60 seconds.

Browser warnings disappear. Do not promote earlier — the prod rate limit will block you if you are still iterating on the Ingress config.

7. Common pitfalls

Common pitfalls

  1. Pointing DNS at the cluster before the Ingress controller is healthy. http-01 challenges fail repeatedly, staging quota exhausts, and you need to wait for the rate-limit window to reset. Always confirm kubectl -n ingress-nginx get pods shows all pods Ready before creating Ingress resources.
  2. ExternalDNS policy: sync deletes a record when an Ingress is renamed. A refactor that renames an Ingress triggers a delete + create at the DNS provider; during propagation your domain is unreachable. Use policy: upsert-only for production until you understand the full lifecycle. Switch to sync deliberately, with a maintenance window.
  3. Forgetting cert-manager.io/cluster-issuer annotation on the Ingress. cert-manager ignores Ingress objects without that annotation; the tls: block has no Secret behind it; ingress-nginx serves an empty cert. Add the annotation, or define a Certificate resource explicitly pointing at the ClusterIssuer.
  4. Promoting to prod issuer while still debugging. Each failed validation attempt counts against the prod rate limit. Stay on staging until everything is green for at least 24 hours, then promote once.

References