Lesson 12 Prometheus, Grafana & Loki
Theme:

Prometheus, Grafana & Loki

Lesson 12 · Cloud Infrastructure & DevOps · ~15 min read

1. Why observability comes before scale

Modules 1–4 got the URL shortener running on Kubernetes with TLS. That is the minimum to call something “deployed”. It is not enough to operate. Operation needs answers to three questions: (1) is the service healthy right now? (2) was it healthy yesterday at 14:00? (3) when it broke last week, what changed? Observability is how you get those answers without kubectl exec-ing into pods at 3 a.m.

Module 5 starts with observability because the next two lessons both depend on it. Lesson 13’s HorizontalPodAutoscaler needs a real traffic metric to decide when to scale — CPU alone lags by 30–60 s. Lesson 14’s GitOps loop needs alerts to surface drift before a human notices. You instrument before you scale, and you alert before you automate.

The concrete deliverable for this lesson: both URL-shortener services expose /metrics in Prometheus text format; Prometheus scrapes them; Grafana shows a six-panel dashboard; Loki holds the logs; and four alert rules fire when the service misbehaves. The cluster is not just deployed — it is observable.

2. The three pillars and the honest Pareto

The canonical model is metrics + logs + traces. The honest 2026 reality is that the value is distributed very unevenly:

Start with metrics + logs. Add tracing the day a real outage tells you it would have saved an hour. That day might never come.

3. The four golden signals

The Google SRE book distils every web-service dashboard into four numbers. Every panel in the L12 sample dashboard maps to one of these:

These four cover almost every “is the service okay?” question. When a dashboard panel does not fit one of the four, ask whether it should be on the dashboard at all. Most “nice to have” charts add noise without adding answers.

4. Instrumenting the Python API

The prometheus_client library is the standard Python client for Prometheus. For a FastAPI/Flask application, the instrumentation pattern is:

import time
from prometheus_client import Counter, Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware

REQ = Counter("http_requests_total", "Total HTTP requests", ["route", "code"])
LAT = Histogram("http_request_duration_seconds", "HTTP request latency", ["route"])

@app.before_request
def _obs_before():
    request._start = time.time()

@app.after_request
def _obs_after(resp):
    route = request.endpoint or "unknown"
    LAT.labels(route=route).observe(time.time() - getattr(request, "_start", time.time()))
    REQ.labels(route=route, code=str(resp.status_code)).inc()
    return resp

# Mount /metrics on the same WSGI tree
app.wsgi_app = DispatcherMiddleware(app.wsgi_app, {"/metrics": make_wsgi_app()})

Three things to notice. First, route is the Flask endpoint name, not the URL — /u/abc123 and /u/xyz789 both label as redirect, keeping cardinality bounded. A counter with one label value per URL would exhaust Prometheus memory within hours of public traffic. Second, code is the HTTP status string, needed to split successes from errors in PromQL. Third, /metrics is mounted as a WSGI application on the same port the API already exposes — no extra Service or Ingress required.

5. Instrumenting the worker

Workers do not serve HTTP, but Prometheus still scrapes over HTTP. The standard solution is start_http_server:

from prometheus_client import start_http_server, Counter, Histogram

JOBS = Counter("worker_jobs_processed_total", "Worker jobs", ["outcome"])
DUR  = Histogram("worker_job_duration_seconds", "Worker job duration")

def main():
    start_http_server(9090)          # metrics-only port
    while True:
        outcome = "ok"
        with DUR.time():
            try:
                process_one_expiry_batch()
            except Exception:
                outcome = "err"
                logger.exception("worker batch failed")
        JOBS.labels(outcome=outcome).inc()

The Helm chart adds prometheus.io/scrape: "true" and prometheus.io/port: "9090" annotations on the worker pod template. Prometheus’s Kubernetes service-discovery picks these up automatically: it watches all pods in the cluster, reads the annotations, and adds scrape targets without any manual configuration. No Service, no Ingress — Prometheus reaches the pod directly via the pod IP, which is routable inside the cluster.

6. The Loki + Promtail story

Loki indexes log lines by label set, not by body content. A query like {namespace="url-shortener", app="api"} |= "500" first narrows by label (cheap — O(label-set count)), then greps the matched stream (still cheap because the streams are pre-filtered). This is what makes Loki affordable at scale: you pay storage for the log bodies but not for an inverted index across them.

Promtail is the agent that ships logs to Loki. Deployed as a Kubernetes DaemonSet, it tails /var/log/pods/<pod-uid>/<container>/*.log, attaches Kubernetes labels (namespace, pod, container), and POSTs them to Loki. Three rules to keep cardinality bounded:

  1. Drop __path__. It changes on every pod restart and explodes the label space.
  2. Never promote high-cardinality values into labels. Request IDs, user IDs, and URL paths are content, not metadata. Query them with |= "..." inside the stream, not as labels.
  3. Drop log levels you never search in production. debug lines double your log volume; drop them in the Promtail pipeline stage before they reach Loki.

Application logs should be structured JSON ({"ts":..., "level":..., "route":..., "msg":...}) so Promtail’s json pipeline stage can extract fields for filtering, and Grafana can render them in a table panel with real columns.

7. Alert rules and the for: clause

PromQL is the alert language. A Prometheus rule is an expr (a PromQL expression that evaluates to a scalar or series) plus a for: duration, labels, and annotations. The rule fires when the expression is true and stays true for the entire for: window.

Two common mistakes with the for: clause:

  1. Omit it entirely. Every 15-second scrape that briefly crosses the threshold pages you. Pager fatigue arrives in three days. An alert without a for: is almost always wrong.
  2. Set it too long. A for: 1h on an error-rate alert means a real outage eats 59 minutes before anyone hears about it. The sweet spot is 5–10 minutes for error-rate and latency alerts, 30 minutes–1 hour for saturation alerts (which trend slowly).

The L12 sample ships four alerts: ApiHighErrorRate (5xx rate > 2% for 10 m), ApiHighLatency (p95 > 1 s for 10 m), WorkerJobBacklog (throughput < 0.1 jobs/s for 30 m), and PodRestartLoop (restart rate elevated for 10 m). Every alert carries a runbook_url annotation pointing at sample-app/observability/RUNBOOK.md. Every alert needs a runbook. If the on-call cannot find one, the alert is unactionable noise.

Common pitfalls

  1. Labelling http_requests_total with url (path) instead of route (endpoint name). Cardinality explodes within an hour of real traffic. Use the framework’s route pattern, not the actual URL.
  2. Exposing /metrics publicly without authentication. Competitors can scrape your traffic shape. Either restrict the endpoint with ingress auth, or run Prometheus purely inside the cluster and never expose /metrics externally.
  3. Watching mean(latency) on dashboards. The mean hides the long tail. Always p95 / p99. If you have only a mean, you have no visibility into your worst-served users.
  4. Alerts without a for: window. Pager fatigue arrives inside a week. Budget an hour to set for: correctly on every rule.
  5. Promoting request-id into a Loki label. Label cardinality explodes; the Loki ingester OOMs. Request IDs are content — grep for them, never label them.

References