Homework

Task 1: Production-Ready Dockerfile

Goal: Write a multi-stage Dockerfile that produces a minimal, secure production image for an LLM API service.

Instructions

Starting from the basic Dockerfile in the lab, enhance it to meet production standards:

1. Multi-stage build: Use a builder stage that installs all dependencies, and a runtime stage that copies only the installed packages and application code.
2. Non-root user: Create a dedicated appuser and run the application as that user (not as root).
3. Health check: Add a HEALTHCHECK instruction that pings your /health endpoint every 30 seconds.
4. Metadata labels: Add LABEL instructions for maintainer, version, and description.
5. Graceful shutdown: Ensure uvicorn receives SIGTERM properly (use exec form of CMD, not shell form).

Deliverable

A complete Dockerfile that builds successfully. Test by running:

docker build -t llm-app:prod .
docker run --rm -e ANTHROPIC_API_KEY=test -p 8000:8000 llm-app:prod
# In another terminal:
curl http://localhost:8000/health

Self-check

• ✓ The final image is under 200 MB (check with docker images llm-app:prod).
• ✓ Running docker inspect llm-app:prod | grep -i healthcheck shows a health check is configured.
• ✓ The container runs as a non-root user (verify with docker exec <container> whoami).
• ✓ The container shuts down within 10 seconds when you run docker stop <container>.

Task 2: Canary Deployment Pipeline Design

Goal: Design a canary deployment pipeline for an LLM-powered chat service, either as a diagram or as a CI/CD configuration file.

Instructions

Your pipeline should cover the full lifecycle from code push to production rollout. Include the following stages:

1. Build & Test: Lint, type-check, unit tests with mocked LLM responses.
2. Eval Gate: Run an eval suite of 50+ test prompts against the live model. Define pass/fail criteria (e.g., accuracy ≥ 90%, format compliance ≥ 95%, no safety violations).
3. Canary Deploy: Deploy the new version alongside the current version. Route 10% of traffic to the canary.
4. Monitoring Window: Monitor for 30 minutes. Track: error rate, p50/p99 latency, eval score on live traffic, user feedback signals.
5. Promotion or Rollback: Define automated rollback triggers (e.g., error rate > 2x baseline, p99 > 150% of baseline). If no triggers fire, promote to 100%.

Deliverable

Choose one of the following formats:

• A GitHub Actions YAML workflow file (.github/workflows/deploy.yml)
• A written description with a text-based diagram showing stages, gates, and decision points
• A GitLab CI/CD or similar pipeline configuration

Self-check

• ✓ Your pipeline includes an eval gate that runs before deployment (not just unit tests).
• ✓ The canary stage specifies a traffic split percentage and a monitoring duration.
• ✓ You have defined at least 3 automated rollback triggers with specific thresholds.
• ✓ Prompt changes are treated as deployable artefacts (versioned and tested).

Task 3: vLLM vs TGI Comparison

Goal: Research and compare the two leading open-model serving frameworks to understand when to use each.

Instructions

Research vLLM and Text Generation Inference (TGI) using their official documentation, benchmark results, and community discussions. Write a comparison covering:

1. Architecture: How does vLLM's PagedAttention differ from TGI's approach to memory management? Why does this matter for throughput?
2. Throughput: Find or estimate benchmark numbers for serving a 7B-parameter model (e.g., Llama 3.1 8B) on a single A100 GPU. Compare requests/second and tokens/second for both frameworks.
3. Ecosystem: Which framework integrates better with your existing tools? Consider: model format support, API compatibility, monitoring, multi-GPU support.
4. Recommendation: In one paragraph, state when you would choose vLLM over TGI and vice versa.

Deliverable

A written comparison (300-500 words) with a summary table:

Criterion	vLLM	TGI
Throughput (tok/s)	...	...
API compatibility	...	...
Multi-GPU	...	...
Quantisation support	...	...
Ease of deployment	...	...

Self-check

• ✓ You cited specific benchmark numbers (even if approximate) rather than vague claims.
• ✓ Your comparison covers at least 4 dimensions (throughput, ease of use, ecosystem, GPU support).
• ✓ Your recommendation is specific — it names concrete scenarios, not just "it depends".
• ✓ You used official documentation or reputable benchmarks as sources.