# PKB-TS Production Runbook

## Service overview
- **Name:** pkb
- **Image:** built from `mcp-servers/sample-server/typescript/Dockerfile`
- **Transport:** HTTP+SSE on port 8080 (behind Caddy on 443)
- **State:** in-memory per-tenant note store + audit log file
- **Audit log:** `/var/log/pkb/audit.jsonl` (override with `PKB_AUDIT_LOG`)

## Healthcheck
- `GET /healthz` -> `{"ok": true, "service": "pkb"}` in < 50 ms.

## Common alerts

### "pkb is down" (compose says unhealthy)
1. `docker compose ps` — confirm container state.
2. `docker compose logs pkb --tail=200` — look for crash trace.
3. `docker compose logs caddy --tail=50` — confirm Caddy can still reach upstream.
4. If process restarted but is healthy now: capture logs to `incidents/<date>.log`, file a ticket.

### "high 401 rate from a single IP"
1. Grep audit log: `jq 'select(.outcome=="denied" and .err_type=="no_bearer")' /var/log/pkb/audit.jsonl | tail -100`.
2. Count by source IP using Caddy log: `jq 'select(.status==401) | .request.remote_ip' caddy.log | sort | uniq -c | sort -rn`.
3. If one IP dominates: add a Caddy deny line for that IP, reload Caddy (`docker compose exec caddy caddy reload`).

### "tenant X says 'rate limit exceeded' constantly"
1. Confirm with `jq 'select(.tenant=="X" and .outcome=="denied" and .err_type=="RateLimit")' /var/log/pkb/audit.jsonl | tail -20`.
2. Check capacity / refill env: `docker compose exec pkb env | grep PKB_RL`.
3. Raise quota if legitimate: edit compose env, `docker compose up -d --no-deps pkb`. Document the change in audit-ticket.
4. If illegitimate (abuse): quarantine the token via `PKB_TOKENS=` edit, restart.

### "audit log not writing"
1. `docker compose exec pkb ls -la /var/log/pkb/` — check perms.
2. If missing dir: mount a volume (`./audit:/var/log/pkb`) in compose, restart.
3. If disk full: `docker compose exec pkb df -h /var/log/pkb`. Rotate / ship.

## Threat model summary
The full threat model lives in `docs/threat-model.md` (TODO: create in production). One-page version:

| Threat | Control | Where |
|---|---|---|
| Unauthenticated access | Bearer auth at `/sse` | `src/auth.ts` |
| Cross-tenant data leak | Per-tenant store keyed by session-derived tenant | `src/store.ts` |
| Compute abuse | Per-tenant token-bucket rate limit | `src/rateLimit.ts` |
| Daily-budget overrun | Per-tenant daily quota (homework / production) | `src/rateLimit.ts` |
| Prompt-injection via tool result | Wrap note bodies in fenced tool-result block | `src/sanitize.ts` |
| Audit gap | Append-only JSONL audit log | `src/audit.ts` |
| Image tampering | Pinned base image + `USER node` + no devDeps | `Dockerfile` |
| TLS termination | Caddy auto-TLS (Let's Encrypt) | `Caddyfile` |

## Runbook ownership
This file is the canonical operational reference. Update it any time you change a control,
add a tool, or learn something during an incident. PRs that change server behaviour without
updating the runbook fail review.