Lesson 12 — Prometheus, Grafana & Loki

You can’t operate what you can’t see. Metrics, logs, and the four golden signals for the URL shortener.

From “it deploys” to “I’d take this pager”

The three pillars (the honest version)

• Metrics (Prometheus): cheap, queryable time-series, the backbone of alerts. ~70% of on-call value at ~10% of cost.
• Logs (Loki): expensive at scale, irreplaceable for one-off forensics. ~25% of value at ~60% of cost.
• Traces (Tempo/Jaeger): out of scope for this lesson — the first two pay for ~90% of the value. Add tracing when a real outage says you need it.

The four golden signals

• Latency — how long does a successful request take? Report p50/p95/p99, never means.
• Traffic — requests per second. The load.
• Errors — rate of 5xx responses and handled exceptions.
• Saturation — how full is the box: CPU, memory, queue depth.

Source: Google SRE Book. Every dashboard in this lesson populates exactly these four.

/metrics on the API

from prometheus_client import Counter, Histogram, make_wsgi_app

REQ = Counter("http_requests_total", "Requests", ["route", "code"])
LAT = Histogram("http_request_duration_seconds", "Latency", ["route"])

Counter + histogram + a WSGI /metrics route. The histogram buckets default to the Prometheus standard (5 ms → 10 s), which already covers a web API. route is the Flask endpoint name — not the URL — so cardinality stays bounded.

/metrics on the worker

from prometheus_client import start_http_server, Counter, Histogram

JOBS = Counter("worker_jobs_processed_total", "Jobs processed", ["outcome"])
DUR  = Histogram("worker_job_duration_seconds", "Job duration")

# In main():
start_http_server(9090)   # metrics-only sidecar port

Workers don’t have an HTTP API of their own, so they expose metrics on a dedicated port. Prometheus scrapes it via pod annotation prometheus.io/port: "9090" — no Service or Ingress needed.

Helm values — wiring the stack

observability:
  enabled: true   # false in dev; true in prod
  prometheus:
    scrapeInterval: 15s
  grafana:
    adminPassword: changeme   # override in prod
  loki:
    retention: 168h

A single observability.enabled: true gates the scrape annotations on both pods. In prod, swap to a managed Prometheus (AMP, Grafana Cloud) by adding remoteWrite config.

PromQL in 60 seconds

# 5xx error rate
rate(http_requests_total{code=~"5.."}[5m])

# p95 latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# request rate by route
sum by (route) (rate(http_requests_total[1m]))

Three queries covering three of the four golden signals. Saturation is container_cpu_usage_seconds_total and container_memory_working_set_bytes.

The alert rules

- alert: ApiHighErrorRate
  expr: sum(rate(http_requests_total{code=~"5.."}[5m]))
          / sum(rate(http_requests_total[5m])) > 0.02
  for: 10m

- alert: ApiHighLatency
  expr: histogram_quantile(0.95,
          rate(http_request_duration_seconds_bucket[5m])) > 1.0
  for: 10m

Two examples; the full file ships four. Alerts fire only after for: 10m — avoid pager-fatigue on transient blips. Never omit the for: window.

Loki + Promtail

Loki stores logs indexed by labels (not full-text), so you query by:

{namespace="url-shortener", pod=~"api-.*"} |= "500"

First narrow by label (cheap), then grep the matched stream (still cheap). Promtail is the DaemonSet that scrapes /var/log/pods/… and ships to Loki.

Three cardinality rules: (1) drop __path__ — it changes per pod restart; (2) never promote request-id into a label; (3) drop debug logs in production.

The Grafana dashboard — 6 panels

• Panel 1 — API request rate by route
• Panel 2 — 5xx error rate (alert threshold overlaid)
• Panel 3 — p50 / p95 / p99 latency
• Panel 4 — CPU + memory saturation per pod
• Panel 5 — worker throughput and job duration
• Panel 6 — raw log panel filtered by level=“error”

Grid layout: 2 columns × 3 rows. The dashboard JSON ships in observability/grafana-dashboards/api-golden-signals.json.

What we instrumented

sample-app/
  api/app.py             # + /metrics route, + REQ/LAT counters
  worker/worker.py       # + start_http_server(9090), + job counters
  helm/values.yaml       # + observability: { enabled: false } default
  helm/values-prod.yaml  # observability.enabled: true
  observability/
    prometheus-rules.yaml
    grafana-dashboards/api-golden-signals.json
    loki-values.yaml
    RUNBOOK.md
    README.md

What we’re not doing (yet)

• SLOs/SLIs as code — next step after dashboards are stable.
• Synthetic checks — blackbox-exporter for external health probes.
• Distributed tracing — Tempo / Jaeger / OpenTelemetry when you need end-to-end spans.
• Anomaly detection — ML-based alerting for seasonal traffic; not needed for two services.
• On-call rotation tooling — PagerDuty / Opsgenie integration deferred.

Recap & what’s next

• Metrics + logs is the 90/10 pair — traces come later, if ever.
• The four golden signals (latency, traffic, errors, saturation) fit every web service.
• Alerts must always carry a for: window — and a runbook link.