Lesson 14 Multi-Tenant Remote MCP Servers
Theme:

Solutions — Lesson 14

Reference answers for Lesson 14's three homework tasks · per-tenant daily quota, session timeout, per-IP rate limit at Caddy

Each solution below shows the worked answer for one homework task followed by a "common mistake" callout that captures the failure modes we saw in earlier drafts. Your exact log lines, timer ids, and Caddy version strings will obviously differ; what matters is that the shape of the change matches the reference and the self-check evidence from the homework reproduces.

Solution to Task 1 — Per-tenant daily quota

The daily counter is a second bucket layered on top of the token-bucket rate limit. The token bucket caps burst (10 calls now) and short-term sustained rate (60 / minute); the daily quota caps long-term volume (1000 / day) so a tenant that paces itself exactly at the rate-limit ceiling still cannot run for 24 hours uncapped. We anchor the reset on UTC midnight, not on the first-call timestamp, so all tenants reset at the same calendar boundary — this makes capacity planning and billing reconciliation straightforward. 

// src/rateLimit.ts — append at the bottom:

interface DailyBucket {
  count: number;
  resetAt: number;  // epoch ms of next UTC midnight
}

const DAILY = new Map<string, DailyBucket>();
const DAILY_QUOTA = Number(process.env.PKB_DAILY_QUOTA ?? 1000);

function nextMidnightUTC(now: number): number {
  const d = new Date(now);
  d.setUTCHours(24, 0, 0, 0);  // rolls to next day at 00:00 UTC
  return d.getTime();
}

export function dailyCheck(tenant: string): {
  allowed: boolean;
  remaining: number;
  resetAt: string;
} {
  const now = Date.now();
  let b = DAILY.get(tenant);
  if (!b || now >= b.resetAt) {
    b = { count: 0, resetAt: nextMidnightUTC(now) };
    DAILY.set(tenant, b);
  }
  if (b.count >= DAILY_QUOTA) {
    return {
      allowed: false,
      remaining: 0,
      resetAt: new Date(b.resetAt).toISOString(),
    };
  }
  b.count += 1;
  return {
    allowed: true,
    remaining: DAILY_QUOTA - b.count,
    resetAt: new Date(b.resetAt).toISOString(),
  };
}

Wired into CallTool in src/server.ts after the per-second rate-limit check:

import { check as rlCheck, dailyCheck } from "./rateLimit.js";

server.setRequestHandler(CallToolRequestSchema, async (req, extra) => {
  const tenant = (extra as any)?.tenant ?? "anonymous";

  // 1. Per-second token-bucket (existing).
  const rl = rlCheck(tenant);
  if (!rl.allowed) {
    return {
      isError: true,
      content: [{ type: "text", text: `rate limit exceeded; retry after ${rl.retryAfterMs} ms` }],
    };
  }

  // 2. Per-day quota (new).
  const dq = dailyCheck(tenant);
  if (!dq.allowed) {
    return {
      isError: true,
      content: [{ type: "text", text: `daily quota exceeded; resets at ${dq.resetAt}` }],
    };
  }

  // ... dispatch as before ...
});

Self-check evidence on a successful run:

$ PKB_TRANSPORT=http \
    PKB_TOKENS="alice:tok_alice" \
    PKB_DAILY_QUOTA=5 \
    node dist/server.js &

# fire 6 calls from alice; the 6th should fail with quota exceeded:
$ for i in 1 2 3 4 5 6; do
    curl -s -H "Authorization: Bearer tok_alice" \
      -X POST localhost:8080/messages?sessionId=$SID \
      -d '{"jsonrpc":"2.0","id":'$i',"method":"tools/call","params":{"name":"pkb.search","arguments":{"q":"x"}}}'
    echo
  done
... (calls 1-5: normal result content) ...
{"isError":true,"content":[{"type":"text","text":"daily quota exceeded; resets at 2026-05-27T00:00:00.000Z"}]}

Reading this solution:

Common mistake

Resetting on first call (e.g., resetAt: now + 24 * 60 * 60_000) instead of UTC midnight. A tenant who first calls at 23:55 then again at 00:01 sees their quota window appear to "roll over" five minutes later, then again at midnight, then again 24 hours after first call — three different boundary semantics depending on which call you trace. Anchoring on calendar midnight gives one global reset everyone can reason about. The second common variant: forgetting now >= b.resetAt in the lazy-rotate branch, so the counter never actually resets and every tenant ends up permanently blocked one day later. Test the rotation by setting resetAt to Date.now() + 5_000 in dev, exhausting the quota, waiting 6 seconds, and confirming the next call succeeds.

Solution to Task 2 — Session timeout

Each SSE session gets a setTimeout that prunes the entry from the sessions map after PKB_SESSION_TTL_MS (default 30 minutes). The session value type grows a timer field so we can clearTimeout on the close path — without this, every closed connection leaks a timer that fires later and calls res.end() on an already-finished response. 

// src/server.ts

interface Session {
  transport: SSEServerTransport;
  tenant: string;
  timer: NodeJS.Timeout;
}

const sessions = new Map<string, Session>();
const SESSION_TTL_MS = Number(process.env.PKB_SESSION_TTL_MS ?? 30 * 60_000);

app.get("/sse", bearerAuth, async (req, res) => {
  const tenant = (req as any).tenant as string;
  const transport = new SSEServerTransport("/messages", res);

  // Prune this session after TTL. Keep the timer ref so we can clear it on close.
  const timer = setTimeout(() => {
    sessions.delete(transport.sessionId);
    try { res.end(); } catch { /* already closed */ }
    logger.info({
      phase: "session_expired",
      tenant,
      session_id: transport.sessionId,
    });
  }, SESSION_TTL_MS);

  sessions.set(transport.sessionId, { transport, tenant, timer });

  res.on("close", () => {
    clearTimeout(timer);                  // <-- the line everyone forgets
    sessions.delete(transport.sessionId);
  });

  await server.connect(transport);
});

The POST /messages handler doesn't change — it still looks the session up by sessionId, and a pruned session naturally returns 404 unknown session: 

app.post("/messages", express.json(), async (req, res) => {
  const sid = String(req.query.sessionId ?? "");
  const sess = sessions.get(sid);
  if (!sess) {
    res.status(404).send("unknown session");
    return;
  }
  (req as any).tenant = sess.tenant;
  await sess.transport.handlePostMessage(req, res, req.body);
});

Self-check evidence on a successful run:

$ PKB_TRANSPORT=http \
    PKB_TOKENS="alice:tok_alice" \
    PKB_SESSION_TTL_MS=10000 \
    node dist/server.js &

# Open SSE stream and capture the session id from the first event:
$ curl -N -H "Authorization: Bearer tok_alice" localhost:8080/sse &
event: endpoint
data: /messages?sessionId=01HXYZ... (this is the session id)

# Wait 11 seconds (TTL is 10s):
$ sleep 11

# Try to use the now-expired session:
$ curl -s -o /dev/null -w "%{http_code}\n" \
    -X POST "localhost:8080/messages?sessionId=01HXYZ..." \
    -H "Authorization: Bearer tok_alice" \
    -d '{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"pkb.search","arguments":{"q":"x"}}}'
404

# Server log shows the prune fired:
{"level":"info","phase":"session_expired","tenant":"alice","session_id":"01HXYZ..."}

Reading this solution:

Common mistake

Forgetting to clearTimeout on the close path. The session is deleted from the map on close, so functionally nothing breaks for a while — the pruner finds no entry and the sessions.delete call is a no-op. But the timer callback still runs later and calls res.end() on a closed stream, emitting ERR_STREAM_WRITE_AFTER_END once per orphaned session. In production this is the kind of bug that looks fine in dev (one connection, one timer) and reports thousands of "write after end" lines per minute under real traffic. The second variant: declaring the timer field on Session but leaving the res.on("close") handler from L14 step 5 unchanged, so the timer is set but never cleared — same bug, different line. Always do a grep -n "setTimeout" src/server.ts and confirm every setTimeout has a matching clearTimeout on every exit path (close, error, expire).

Solution to Task 3 — Per-IP rate limit at Caddy

The per-IP rate limit is a second layer above the per-tenant in-process rate limit. The in-process limit defends one tenant from itself (10 burst, 60 / minute, 1000 / day); the Caddy per-IP limit defends the process from an anonymous flood (no token, no tenant) that would otherwise saturate the auth middleware before it can issue 401s. Limits stack: each request must pass IP then token then tenant then daily.

Updated Caddyfile:

{
    # Caddy needs to know where `rate_limit` slots into the directive order.
    order rate_limit before reverse_proxy
}

pkb.example.com {
    rate_limit {
        zone per_ip {
            key    {remote_host}
            events 30
            window 1m
        }
        # Only rate-limit the auth-establishing route. The /messages POSTs
        # are already gated by sessionId + bearer at the app layer.
        match {
            path /sse*
        }
    }

    encode gzip
    reverse_proxy pkb:8080
    log {
        output stdout
        format json
    }
}

The rate_limit directive ships as a separate Caddy module (caddy-ratelimit) that isn't in the stock caddy:2-alpine image. The simplest production fix is to bake a custom Caddy image: 

# Caddyfile and image build, all-in-one.
FROM caddy:2-builder-alpine AS builder
RUN xcaddy build \
    --with github.com/mholt/caddy-ratelimit

FROM caddy:2-alpine
COPY --from=builder /usr/bin/caddy /usr/bin/caddy
COPY Caddyfile /etc/caddy/Caddyfile

Point the compose service at the local build:

services:
  caddy:
    build:
      context: .
      dockerfile: Caddyfile.Dockerfile
    depends_on:
      pkb:
        condition: service_healthy
    ports:
      - "443:443"
      - "80:80"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    restart: unless-stopped

Self-check evidence on a successful run:

$ docker compose build caddy
$ docker compose up -d

$ for i in $(seq 1 50); do
    curl -s -o /dev/null -w "%{http_code}\n" \
      -H "Authorization: Bearer tok_alice" \
      https://localhost/sse \
      -k --max-time 1
  done | sort | uniq -c
  30 200
  20 429

Reading this solution:

Common mistake

Adding the rate_limit block to the stock caddy:2-alpine image and watching caddy validate fail with unknown directive "rate_limit". The directive is in a third-party module that the stock image doesn't ship; you must build a custom image with xcaddy as shown above, or apply the limit at a different layer entirely (Cloudflare WAF rule, AWS WAF, fronting nginx with limit_req). If you can't build a custom Caddy image in your environment, document the gap explicitly in the runbook so the next operator doesn't assume the layer is in place when it isn't. The second common variant: forgetting order rate_limit before reverse_proxy in the global options — without it Caddy parses the directive but emits a warning and silently does nothing. Always confirm a 429 actually appears in the self-check; don't trust caddy validate alone.

What you learned

Lesson 15 closes the course: threat model, input fuzzer, prompt-injection-resistant tool result formatter, audit log, and a written runbook. The auth / store / rateLimit modules from L14 stay byte-identical — L15 layers audit.ts and sanitize.ts on top, plus a scripts/fuzz.ts harness that exercises every tool with pathological inputs and a RUNBOOK.md that documents on-call procedures.