Lesson 12 Prometheus, Grafana & Loki
Theme:

Homework 12 — Prometheus, Grafana & Loki

Lesson 12 · Cloud Infrastructure & DevOps · Three tasks

These three tasks deepen the observability foundation from the lab. Task 1 adds a business-hours alert. Task 2 reduces Loki label cardinality. Task 3 builds a new Grafana panel for error log inspection. You will need the working cluster from the lab.

Task 1 — Add a fifth alert: ApiLowTraffic

The four shipped alerts cover the “something is broken” signals. This task adds a business-hours alert that fires when no one is using the service during core hours — a signal that the service may be unreachable, or that a deploy went wrong and traffic stopped flowing.

  1. Write a new Prometheus alert ApiLowTraffic in observability/prometheus-rules.yaml that fires when the request rate drops below 0.01 rps and the current hour is between 09:00 and 18:00 UTC. Use vector(1) combined with hour() to gate on business hours.
  2. The alert should use for: 30m and severity warn.
  3. Apply it to the cluster and confirm it appears in Prometheus: 
    kubectl apply -n monitoring -f cloud-devops/sample-app/observability/prometheus-rules.yaml
# Then in Prometheus UI (port-forward to :9090), navigate to Alerts
# and confirm ApiLowTraffic is listed.
  4. Add a three-bullet WHY.md file (or a comment block in prometheus-rules.yaml) explaining:
    • Why low traffic is worth alerting on.
    • Why the business-hours gate matters (why not alert at 3 a.m.?).
    • One edge case this alert could produce a false positive for.

✓ Self-check: ApiLowTraffic appears in the Prometheus Alerts UI. If you have promtool installed, promtool check rules observability/prometheus-rules.yaml exits 0.

Task 2 — Reduce Loki label cardinality

Label cardinality is the primary operational risk with Loki. This task has you measure what Promtail is currently exporting and drop the labels that scale with traffic or time.

  1. Inspect Promtail’s current label set: 
    kubectl -n monitoring exec -it ds/loki-promtail -- wget -qO- http://localhost:3101/targets
    Identify every label in the output. Note which ones are high-cardinality (their value changes with traffic or with pod restarts).
  2. Modify observability/loki-values.yaml to drop the two highest-cardinality labels via a labeldrop pipeline stage (beyond the ones already dropped in the shipped config).
  3. Upgrade Loki to apply the new config: 
    helm upgrade loki grafana/loki-stack \
  -n monitoring \
  -f cloud-devops/sample-app/observability/loki-values.yaml
  4. Confirm that loki_ingester_streams_total in the Prometheus UI is stable (not growing with each new request).

✓ Self-check: the metric loki_ingester_streams_total does not grow as you generate more traffic via curl. Document the two labels you dropped and a one-line reason for each in a comment in loki-values.yaml.

Task 3 — Build a 7th dashboard panel: error log table

The current dashboard’s panel 6 is a log-stream panel that scrolls. This task replaces or augments it with a structured table panel that shows the last 10 error log lines with typed columns.

  1. In Grafana, add a new panel of type Table to the API — Golden Signals dashboard.
  2. Use this Loki query: 
    {namespace="url-shortener", app="api"} | json | level="error" | line_format "{{.ts}} {{.route}} {{.msg}}"
  3. Configure the panel to show columns: time, route, msg. Set a max of 10 rows.
  4. Save the dashboard and export it as JSON (Dashboard settings → JSON Model). Copy the JSON back to observability/grafana-dashboards/api-golden-signals.json.

✓ Self-check: the exported api-golden-signals.json diff shows exactly one new panel of type table. The panel renders 0–10 rows depending on whether errors have occurred. python3 -c "import json; json.load(open('api-golden-signals.json'))" exits 0.

View solutions →