# Observability Stack — URL Shortener

This directory contains the observability configuration for the URL-shortener sample app, introduced in **Lesson 12** of the Cloud Infrastructure & DevOps course.

## What's here

| File | Purpose |
|---|---|
| `prometheus-rules.yaml` | Four PrometheusRule alerts (golden signals) |
| `grafana-dashboards/api-golden-signals.json` | Six-panel Grafana dashboard |
| `loki-values.yaml` | Loki + Promtail Helm values overlay for `kind` |
| `RUNBOOK.md` | One runbook section per alert |

## How it fits together

```
Prometheus ──scrapes──► api pods   (port 8080 /metrics)
            ──scrapes──► worker pods (port 9090 /metrics)
            ──reads──►  prometheus-rules.yaml ──fires──► Alertmanager

Promtail (DaemonSet) ──ships logs──► Loki

Grafana ──queries──► Prometheus  (panels 1–5)
        ──queries──► Loki        (panel 6 — error logs)
```

The Helm value `observability.enabled` (false by default) gates the `prometheus.io/scrape` pod annotations on both Deployments. Set it to `true` in `values-prod.yaml` or via `--set observability.enabled=true`.

## Install on `kind`

```bash
# 1. Add Helm repos
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# 2. Install kube-prometheus-stack (Prometheus + Grafana + Alertmanager)
helm install kps prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.adminPassword=changeme

# 3. Install Loki + Promtail
helm install loki grafana/loki-stack \
  -n monitoring \
  -f cloud-devops/sample-app/observability/loki-values.yaml

# 4. Apply alert rules
kubectl apply -n monitoring -f cloud-devops/sample-app/observability/prometheus-rules.yaml

# 5. Redeploy URL shortener with observability enabled
helm upgrade --install url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener --create-namespace \
  -f cloud-devops/sample-app/helm/values-prod.yaml

# 6. Import Grafana dashboard
kubectl -n monitoring create configmap api-golden-signals \
  --from-file=cloud-devops/sample-app/observability/grafana-dashboards/api-golden-signals.json
kubectl -n monitoring label configmap api-golden-signals grafana_dashboard=1
```

## Access Grafana

```bash
kubectl -n monitoring port-forward svc/kps-grafana 3000:80 &
open http://localhost:3000   # admin / changeme
```

The **API — Golden Signals** dashboard appears under Dashboards within ~30 seconds of the configmap label being applied (the kube-prometheus-stack sidecar reloads dashboards automatically).

## Managed Prometheus (production)

For real clusters, replace the in-cluster Prometheus with a managed service:

```yaml
# Add to values-prod.yaml
observability:
  enabled: true
  prometheus:
    remoteWrite:
      - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-xxx/api/v1/remote_write
        sigv4:
          region: us-east-1
```

The alert rules and dashboard PromQL work unchanged against any Prometheus-compatible backend.

## Custom metric for L13 HPA: `http_requests_per_second`

The L13 `HorizontalPodAutoscaler` can scale on `http_requests_per_second` per pod instead of CPU.
This requires the **prometheus-adapter** to bridge Prometheus → the Kubernetes custom-metrics API.

```bash
# Custom metric: http_requests_per_second per pod (consumed by L13's HPA)
helm install prometheus-adapter prometheus-community/prometheus-adapter \
  -n monitoring \
  --set prometheus.url=http://kps-kube-prometheus-stack-prometheus.monitoring.svc \
  --set prometheus.port=9090 \
  --set-string rules.custom='[{"seriesQuery":"http_requests_total{namespace!=\"\",pod!=\"\"}","resources":{"overrides":{"namespace":{"resource":"namespace"},"pod":{"resource":"pod"}}},"name":{"matches":"^(.*)_total$","as":"${1}_per_second"},"metricsQuery":"sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)"}]'
```

Verify the metric is visible:

```bash
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/url-shortener/pods/*/http_requests_per_second" | jq .
```

Then enable the custom metric path in the chart:

```bash
helm upgrade url-shortener cloud-devops/sample-app/helm/ \
  -n url-shortener \
  -f cloud-devops/sample-app/helm/values-prod.yaml
# values-prod.yaml already sets autoscaling.customMetric.enabled: true
```