Lab 12 — Install Prometheus + Grafana + Loki, dashboard the API, fire one alert deliberately

Lesson 12 · Cloud Infrastructure & DevOps · ~60 min

Prereqs

Module 4 complete. A kind cluster running with the URL-shortener Helm chart deployed and reachable (Lesson 11 cluster is fine). You need:

• helm ≥ 3.13
• kubectl ≥ 1.28
• curl and optionally jq
• ~2 GiB free memory in the kind Docker container (kube-prometheus-stack is heavy)

Step 1 — Confirm the observability files are in place

All code was committed in the sample-app Task 4 update. Confirm the files exist before running anything:

ls cloud-devops/sample-app/observability/
# prometheus-rules.yaml  grafana-dashboards/  loki-values.yaml  RUNBOOK.md  README.md

ls cloud-devops/sample-app/observability/grafana-dashboards/
# api-golden-signals.json

Also confirm the /metrics route is in api/app.py and start_http_server is in worker/worker.py:

grep -n "metrics" cloud-devops/sample-app/api/app.py | head
grep -n "start_http_server" cloud-devops/sample-app/worker/worker.py

Step 2 — Confirm /metrics works locally

Run the API directly (no Docker) and curl /metrics to verify the Prometheus client is wired correctly before touching the cluster.

cd cloud-devops/sample-app/api
pip install -r requirements.txt -q
uvicorn app:app --port 5000 &
sleep 2

# Confirm /metrics returns Prometheus text format
curl -s localhost:5000/metrics | grep -E "^(http_requests_total|http_request_duration)" | head

# Generate one request so the counter appears
curl -s localhost:5000/health -o /dev/null

# Confirm the counter has incremented
curl -s localhost:5000/metrics | grep 'http_requests_total{' | head

kill %1
cd ../../..

You should see lines like:

# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{code="200",route="health"} 1.0

Step 3 — Deploy kube-prometheus-stack on kind

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword=changeme \
  --set prometheus.prometheusSpec.resources.requests.memory=512Mi

kubectl -n monitoring rollout status deploy kps-grafana --timeout=5m

Expect 10–12 pods running in monitoring after 2–3 minutes:

kubectl -n monitoring get pods

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraMounts: []
    kubeadmConfigPatches:
      - |
        kind: KubeletConfiguration
        systemReserved:
          memory: 512Mi

Step 4 — Deploy Loki + Promtail on kind

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki-stack \
  -n monitoring \
  -f cloud-devops/sample-app/observability/loki-values.yaml

kubectl -n monitoring rollout status statefulset loki --timeout=5m

Confirm Promtail DaemonSet is running:

kubectl -n monitoring get ds -l app.kubernetes.io/name=promtail

Step 5 — Apply the alert rules

kubectl apply -n monitoring -f cloud-devops/sample-app/observability/prometheus-rules.yaml
kubectl -n monitoring get prometheusrules

Confirm url-shortener-golden-signals appears in the list.

Step 6 — Redeploy the URL shortener with observability enabled

helm upgrade --install url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener --create-namespace \
  -f cloud-devops/sample-app/helm/values-prod.yaml \
  --set observability.enabled=true

kubectl -n url-shortener rollout status deploy url-shortener-api --timeout=2m
kubectl -n url-shortener rollout status deploy url-shortener-worker --timeout=2m

Confirm the scrape annotations are present on the API pods:

kubectl -n url-shortener get pods -o jsonpath='{.items[0].metadata.annotations}' | jq .

You should see prometheus.io/scrape: "true" and prometheus.io/port in the output.

Step 7 — Verify Prometheus is scraping both services

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 &
sleep 2

# Query the up metric for our namespace
curl -s 'http://localhost:9090/api/v1/query?query=up{namespace="url-shortener"}' \
  | jq '.data.result[] | {job: .metric.job, pod: .metric.pod, value: .value[1]}'

kill %1

Expect at least two entries with value: "1" — one for the API pod and one for the worker pod.

Step 8 — Import the Grafana dashboard

# Create the configmap from the dashboard JSON
kubectl -n monitoring create configmap api-golden-signals \
  --from-file=cloud-devops/sample-app/observability/grafana-dashboards/api-golden-signals.json

# Label it so the kube-prometheus-stack sidecar picks it up automatically
kubectl -n monitoring label configmap api-golden-signals grafana_dashboard=1

# Open Grafana
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 &
open http://localhost:3000   # admin / changeme

In the Grafana UI, navigate to Dashboards → API — Golden Signals. All 6 panels should populate within about 30 seconds. If panels show “No data”, check that the namespace variable is set to url-shortener.

Step 9 — Generate load and watch the four signals move

kubectl -n url-shortener port-forward svc/url-shortener-api 5000:80 &
sleep 1

# Generate ~20 rps
while true; do curl -s localhost:5000/health > /dev/null; sleep 0.05; done &

Switch to Grafana and watch panel 1 (request rate) climb, panel 3 (latency) stabilise, panel 2 (errors) remain flat, and panel 4 (CPU) tick upward. Leave the load running for the next step.

Step 10 — Fire ApiHighErrorRate deliberately

Force the API to return 500s on a fraction of requests. The simplest way is to temporarily set an env var that triggers error injection if your build supports it, or to scale the API down to 0 replicas (which causes 503s from the service):

# Option A: force HTTP 500 via kubectl exec (edit app.py at runtime in the container)
kubectl -n url-shortener exec -it deploy/url-shortener-api -- sh -c \
  "echo 'BREAK_RATE=0.5' >> /etc/environment"

# Option B: simpler — patch in a bad env var the app won't understand
kubectl -n url-shortener set env deploy/url-shortener-api INTENTIONAL_BREAK=true

Wait 10–12 minutes for the for: 10m window to elapse. Then check the alert state:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 &
sleep 1
curl -s 'http://localhost:9090/api/v1/alerts' \
  | jq '.data.alerts[] | select(.labels.alertname=="ApiHighErrorRate")'

Expect "state": "firing". Once confirmed, undo the breakage:

kubectl -n url-shortener set env deploy/url-shortener-api INTENTIONAL_BREAK-
# Watch the alert clear in Prometheus UI (state goes "firing" → "pending" → absent)
kill %1

Step 11 — Confirm logs flow to Loki

In Grafana, open Explore and select the Loki datasource. Run this query:

{namespace="url-shortener"} |= "GET"

You should see recent access-log lines from the API pods. Add a level filter to see the error lines emitted during step 10:

{namespace="url-shortener"} | json | level="error"

What you should leave with

• kind cluster running Prometheus + Grafana + Loki alongside the URL shortener.
• Both services emitting /metrics (API on port 8080, worker on port 9090).
• A working 6-panel Grafana dashboard showing all four golden signals.
• One fired-and-cleared ApiHighErrorRate alert in your history.
• Logs from both pods flowing through Promtail into Loki, queryable in Grafana Explore.