Solutions — Homework 12

Lesson 12 · Cloud Infrastructure & DevOps

Task 1 — ApiLowTraffic alert

Add this block to observability/prometheus-rules.yaml inside the existing rules: list:

- alert: ApiLowTraffic
  expr: |
    sum(rate(http_requests_total[5m])) < 0.01
    and on() hour() >= 9
    and on() hour() < 18
  for: 30m
  labels:
    severity: warn
  annotations:
    summary: "API receiving fewer than 0.01 rps during business hours for 30 minutes"
    description: "Traffic is {{ $value | humanize }} rps. Possible deploy failure or DNS issue."
    runbook_url: "https://github.com/REPLACE_ME/blob/main/cloud-devops/sample-app/observability/RUNBOOK.md#apilowtraffic"

How hour() works: hour() returns the hour of day (0–23) in UTC at the current evaluation timestamp. and on() is the binary operator that only passes through the left-hand side when both sides have a matching label set — using on() (empty label set) means “there is at least one element on both sides”, which for a scalar comparison is always true or false.

Common mistakes:

• Writing > 0.01 instead of < 0.01. The “low traffic” alert fires when traffic is less than the threshold.
• Forgetting the business-hours gate, so the alert pages at 3 a.m. every weekend when the service is legitimately idle.
• Setting the threshold too high (e.g., 1 rps) so the alert fires during normal low-traffic periods like lunch.
• hour() returns UTC by default. If your team is in a non-UTC timezone, document the offset and adjust the range accordingly.

Why three bullets (for the WHY.md or comment):

1. Low traffic is a signal something is wrong: the service may be unreachable (DNS failure, ingress misconfiguration, cert expired) even if no 5xx errors are firing, because no traffic reaches the service to generate errors.
2. Business-hours gate prevents weekend noise: a URL-shortener with internal users naturally receives zero traffic on nights and weekends. Alerting during those windows is pure pager fatigue.
3. One false-positive scenario: a planned maintenance window where traffic is intentionally drained. Fix with an alert inhibit rule or silence in Alertmanager during the window.

Task 2 — Loki label cardinality

Two common high-cardinality labels Promtail exports by default:

• filename — the full path on disk (/var/log/pods/<uid>/container/*.log). Changes on every pod restart, so cardinality grows as pods restart over time.
• stream — stdout vs stderr. Only two values, but it doubles the number of Loki streams per pod (one stream per label set = one per stdout + one per stderr).

Updated loki-values.yaml snippet:

promtail:
  config:
    snippets:
      common:
        # Drop high-cardinality labels (updated from shipped config)
        - action: labeldrop
          # __path__: changes per pod restart, explodes label space
          # filename: full disk path, changes per restart
          # stream: stdout/stderr doubles streams without adding query value
          regex: __path__|filename|stream

Common mistake: dropping pod or container — those are useful low-cardinality labels. They let you query {pod="api-xyz"} directly. The rule of thumb: any label whose distinct value count scales with time (filenames per restart, request IDs per request) is content, not metadata. Any label whose cardinality is bounded by the number of pods (pod name, container name, namespace) is safe.

Verifying with loki_ingester_streams_total:

kubectl -n monitoring port-forward svc/kps-kube-prometheus-stack-prometheus 9090:9090 &
# Generate 10 requests:
for i in $(seq 1 10); do curl -s localhost:5000/health -o /dev/null; done
# Check streams metric — should not change:
curl -s 'http://localhost:9090/api/v1/query?query=loki_ingester_streams_total' | jq '.data.result[0].value[1]'

The value should remain stable. Before the fix, each pod restart or new filename would increment it.

Task 3 — Error log table panel

Panel JSON skeleton to add to panels: in api-golden-signals.json:

{
  "id": 7,
  "type": "table",
  "title": "Recent API Error Logs",
  "gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
  "targets": [
    {
      "datasource": { "type": "loki" },
      "expr": "{namespace=\"url-shortener\", app=\"api\"} | json | level=\"error\" | line_format \"{{.ts}} {{.route}} {{.msg}}\"",
      "queryType": "range",
      "maxLines": 10
    }
  ],
  "options": {
    "showHeader": true,
    "sortBy": [{ "displayName": "Time", "desc": true }]
  },
  "transformations": [
    {
      "id": "extractFields",
      "options": {
        "source": "labels"
      }
    }
  ]
}

Common mistakes:

• Putting the table panel on the same row as the latency chart. Error tables grow long. Place it in its own row at the bottom (set y to a value below the last existing row) or move it to a linked “API — Errors Detail” dashboard.
• Using queryType: "instant" for a log panel — logs always need queryType: "range".
• Forgetting to set maxLines; without it the panel may load thousands of lines and slow the browser.

Closing tip: Lesson 13 plugs autoscaling and resilience into the metrics you just exposed. The HPA will read http_requests_total directly to scale on request rate, not just CPU. Everything you instrumented here feeds directly into the next lesson’s autoscaler configuration.