Lab Guide

Format: Python script (lab.py)  ·  Time estimate: 60 minutes  ·  Prerequisites: Docker installed, Python 3.10+, anthropic, fastapi, uvicorn, redis, locust

Setup

Install the required dependencies and ensure Docker is running:

# Install Python dependencies
pip install fastapi uvicorn anthropic python-dotenv redis locust httpx

# Verify Docker is available
docker --version

# Start a Redis container for the caching exercise
docker run -d --name redis-lab -p 6379:6379 redis:7-alpine

Ensure your .env file contains a valid ANTHROPIC_API_KEY.

Exercise 1: Containerise a FastAPI LLM App

Goal

Build a production-ready Docker container for a FastAPI application that wraps the Claude API, including a health check endpoint and proper signal handling.

Steps

1. Create a minimal FastAPI app (app.py) with three endpoints:
   • GET /health — returns {"status": "ok"}
   • POST /v1/chat — accepts a JSON body with a message field, calls Claude, returns the response
   • GET /metrics — returns basic counters (total requests, total tokens, average latency)
2. Write a Dockerfile:

   FROM python:3.11-slim AS builder
   WORKDIR /app
   COPY requirements.txt .
   RUN pip install --no-cache-dir --prefix=/install -r requirements.txt
   
   FROM python:3.11-slim
   WORKDIR /app
   COPY --from=builder /install /usr/local
   COPY . .
   EXPOSE 8000
   HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
     CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"
   CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

3. Build and run:

   # Build the image
   docker build -t llm-app:v1 .
   
   # Run with your API key
   docker run -d --name llm-app -p 8000:8000 \
     -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
     llm-app:v1
   
   # Test the health endpoint
   curl http://localhost:8000/health
   
   # Test the chat endpoint
   curl -X POST http://localhost:8000/v1/chat \
     -H "Content-Type: application/json" \
     -d '{"message": "What is Docker in one sentence?"}'

Expected Output

$ curl http://localhost:8000/health
{"status":"ok"}

$ curl -X POST http://localhost:8000/v1/chat \
    -H "Content-Type: application/json" \
    -d '{"message": "What is Docker in one sentence?"}'
{"response":"Docker is a platform that packages applications and their
dependencies into lightweight, portable containers that run consistently
across any environment.","tokens_used":42,"latency_ms":1250}

Exercise 2: Add Exact-Match Caching

Goal

Implement an exact-match response cache using Redis to avoid redundant API calls for identical prompts.

Steps

1. Add a caching layer to your /v1/chat endpoint:

   import hashlib, json, redis, time
   
   r = redis.Redis(host="localhost", port=6379, decode_responses=True)
   CACHE_TTL = 3600  # 1 hour
   
   def get_cache_key(message: str, model: str) -> str:
       payload = json.dumps({"message": message, "model": model}, sort_keys=True)
       return f"llm_cache:{hashlib.sha256(payload.encode()).hexdigest()}"
   
   @app.post("/v1/chat")
   async def chat(body: ChatRequest):
       cache_key = get_cache_key(body.message, "claude-sonnet-4-20250514")
   
       # Check cache
       cached = r.get(cache_key)
       if cached:
           result = json.loads(cached)
           result["cache_hit"] = True
           return result
   
       # Call LLM
       start = time.time()
       response = client.messages.create(
           model="claude-sonnet-4-20250514",
           max_tokens=256,
           messages=[{"role": "user", "content": body.message}]
       )
       latency_ms = int((time.time() - start) * 1000)
   
       result = {
           "response": response.content[0].text,
           "tokens_used": response.usage.input_tokens + response.usage.output_tokens,
           "latency_ms": latency_ms,
           "cache_hit": False,
       }
   
       # Store in cache
       r.setex(cache_key, CACHE_TTL, json.dumps(result))
       return result

2. Test the cache by sending the same request twice:

   # First call — cache miss
   curl -s -X POST http://localhost:8000/v1/chat \
     -H "Content-Type: application/json" \
     -d '{"message": "Explain caching in one sentence."}' | python -m json.tool
   
   # Second call — cache hit (should be instant)
   curl -s -X POST http://localhost:8000/v1/chat \
     -H "Content-Type: application/json" \
     -d '{"message": "Explain caching in one sentence."}' | python -m json.tool

3. Verify the cache hit by checking the cache_hit field and comparing latencies.

Expected Output

# First call
{
    "response": "Caching stores frequently accessed data closer to where it's
    needed so future requests can be served faster without recomputing or
    re-fetching the original source.",
    "tokens_used": 48,
    "latency_ms": 1340,
    "cache_hit": false
}

# Second call
{
    "response": "Caching stores frequently accessed data closer to where it's
    needed so future requests can be served faster without recomputing or
    re-fetching the original source.",
    "tokens_used": 48,
    "latency_ms": 2,
    "cache_hit": true
}

Exercise 3: Load Testing with Locust

Goal

Use Locust to simulate concurrent users hitting your LLM service, measure throughput and latency under load, and identify bottlenecks.

Steps

1. Create a Locust test file (locustfile.py):

   from locust import HttpUser, task, between
   import random
   
   PROMPTS = [
       "What is Docker?",
       "Explain Kubernetes in one sentence.",
       "What is a load balancer?",
       "Define horizontal scaling.",
       "What is a reverse proxy?",
   ]
   
   class LLMUser(HttpUser):
       wait_time = between(1, 3)
   
       @task(3)
       def chat(self):
           """Send a chat request — mix of cached and uncached."""
           self.client.post("/v1/chat", json={
               "message": random.choice(PROMPTS)
           })
   
       @task(1)
       def health_check(self):
           """Hit the health endpoint."""
           self.client.get("/health")

2. Run Locust:

   # Start Locust with web UI
   locust -f locustfile.py --host http://localhost:8000
   
   # Or run headless for 60 seconds with 10 users
   locust -f locustfile.py --host http://localhost:8000 \
     --headless -u 10 -r 2 -t 60s

3. Open the Locust web UI at http://localhost:8089 and start a test with 10 users and a spawn rate of 2 users/second.
4. Observe:
   • Median response time — how fast are cached vs uncached requests?
   • Requests per second — does it plateau as users increase?
   • Failure rate — are any requests timing out or failing?
   • p99 latency — what is the worst-case response time?

Expected Output

Type     Name               # reqs   # fails  Avg    Min    Max   Median  req/s
-------- ------------------ ------- --------- ------ ------ ----- ------- ------
POST     /v1/chat              147     0(0%)   680     2    3420     45    2.1
GET      /health                52     0(0%)     3     1      12      2    0.9
-------- ------------------ ------- --------- ------ ------ ----- ------- ------
         Aggregated            199     0(0%)   492     1    3420     12    3.0

Finishing Up

Once you have completed all three exercises:

1. Stop the running containers:

   docker stop llm-app redis-lab
   docker rm llm-app redis-lab

2. Review the Locust results and note which metrics would be most important for your production use case.
3. Compare your implementation with lab_answer.py.

Self-Check

• ✓ Exercise 1: Your container builds, starts, and responds to both /health and /v1/chat.
• ✓ Exercise 1: The Docker image uses a multi-stage build and does not contain your API key.
• ✓ Exercise 2: The second identical request returns "cache_hit": true with latency under 10ms.
• ✓ Exercise 2: You can explain why exact-match caching requires temperature=0 to be effective.
• ✓ Exercise 3: You ran a Locust test and can interpret the median, p99, and req/s metrics.
• ✓ Exercise 3: You can identify the main bottleneck in your service under load.