# Observability Runbook — URL Shortener

One section per Prometheus alert. On-call: read the matching section before touching anything.

---

## ApiHighErrorRate

**What it means:** The API is returning 5xx responses at more than 2% of total request volume, sustained for 10 minutes.

**First checks:**
```bash
# 1. Is the pod running?
kubectl -n url-shortener get pods -l app.kubernetes.io/component=api

# 2. Recent error logs
kubectl -n url-shortener logs -l app.kubernetes.io/component=api --tail=100 | grep -i error

# 3. PromQL — which route is failing?
# In Grafana > Explore: sum by (route) (rate(http_requests_total{code=~"5.."}[5m]))
```

**Likely causes:**
- A recent deploy introduced a bug (check `kubectl rollout history`).
- The Postgres connection pool is exhausted or Postgres is down.
- An upstream dependency (external URL redirect target) is timing out.

**Mitigations:**
- Roll back to the previous image: `helm rollback url-shortener -n url-shortener`
- Scale up to spread load: `kubectl -n url-shortener scale deploy url-shortener-api --replicas=5`
- If Postgres is the cause, check `kubectl -n url-shortener get pods -l app.kubernetes.io/component=postgres` and review its logs.

---

## ApiHighLatency

**What it means:** The p95 request latency is above 1 second, sustained for 10 minutes. 5% of users are waiting longer than 1 s per request.

**First checks:**
```bash
# 1. Pod resource usage
kubectl -n url-shortener top pods -l app.kubernetes.io/component=api

# 2. Is Postgres slow?
kubectl -n url-shortener exec -it deploy/url-shortener-api -- python -c \
  "import time, psycopg2, os; conn=psycopg2.connect(os.environ['DATABASE_URL']); t=time.time(); conn.cursor().execute('SELECT 1'); print(time.time()-t)"

# 3. Grafana panel 3 — Latency Percentiles
```

**Likely causes:**
- CPU throttling: `requests.cpu` is too low relative to actual usage — HPA hasn't scaled yet.
- Postgres queries are slow: missing index or table bloat after a data migration.
- A downstream redirect target is slow, causing the redirect handler to block.

**Mitigations:**
- Temporarily increase CPU requests: `kubectl -n url-shortener set resources deploy url-shortener-api --requests=cpu=200m`
- Run `ANALYZE` on the Postgres `urls` table to refresh query planner statistics.
- Roll back if a recent deploy caused the regression.

---

## WorkerJobBacklog

**What it means:** The worker is processing fewer than 0.1 expiry jobs per second on average over 30 minutes. The expiry backlog is growing.

**First checks:**
```bash
# 1. Is the worker running?
kubectl -n url-shortener get pods -l app.kubernetes.io/component=worker

# 2. Worker logs
kubectl -n url-shortener logs -l app.kubernetes.io/component=worker --tail=50

# 3. Worker metrics
kubectl -n url-shortener port-forward svc/url-shortener-worker 9090:9090 &
curl -s localhost:9090/metrics | grep worker_jobs
```

**Likely causes:**
- Worker pod is in CrashLoopBackOff due to a DB connection error.
- The worker is processing 0 jobs because the `urls` table has no expired rows (this alert may be a false positive in low-traffic environments — consider adding a business-hours gate).
- A code bug in the expiry sweep throws an exception on every batch.

**Mitigations:**
- Restart the worker: `kubectl -n url-shortener rollout restart deploy url-shortener-worker`
- Check Postgres connectivity from the worker pod.
- If this is a low-traffic environment, add `and on() hour() >= 9 and on() hour() < 18` to the alert expr to suppress off-hours paging.

---

## PodRestartLoop

**What it means:** A pod in the `url-shortener` namespace is restarting at more than 0.1 restarts per second (roughly once every 10 seconds), sustained for 10 minutes.

**First checks:**
```bash
# 1. Which pod?
kubectl -n url-shortener get pods --sort-by='.status.containerStatuses[0].restartCount'

# 2. Why is it crashing?
kubectl -n url-shortener describe pod <pod-name>
kubectl -n url-shortener logs <pod-name> --previous

# 3. OOMKilled?
kubectl -n url-shortener get events --field-selector reason=OOMKilling
```

**Likely causes:**
- OOMKilled: memory limit is too low for current traffic. Raise `limits.memory` in values.
- Liveness probe is too aggressive: `initialDelaySeconds` is too short for a slow-starting pod.
- Application bug introduced in a recent deploy causes a startup crash.

**Mitigations:**
- If OOMKilled: `helm upgrade url-shortener cloud-devops/sample-app/helm/ -n url-shortener --set api.resources.limits.memory=512Mi`
- Roll back the most recent deploy: `helm rollback url-shortener -n url-shortener`
- Temporarily increase `initialDelaySeconds` on the liveness probe to give the pod more time to start.