Deployment Patterns

Taking LLM applications from prototype to production — architecture patterns, hosting options, scaling strategies, and cost optimization.

Learning Objectives

• Choose the right architecture pattern (API gateway, queue-based, streaming) for your use case
• Compare hosting options: serverless, containers, and managed services
• Deploy and serve open-weight models with vLLM, TGI, and llama.cpp
• Implement scaling strategies: horizontal scaling, load balancing, rate limiting
• Apply caching (semantic and exact-match) to reduce latency and cost
• Set up CI/CD pipelines with canary deployments and rollback for LLM apps

Architecture Patterns

LLM applications typically follow one of three request-handling patterns, each suited to different latency and throughput requirements.

Hosting Options

Where you run your LLM application depends on cost, scale, and whether you are calling an external API or self-hosting a model.

Self-Hosting Open Models

Three production-grade serving frameworks dominate the self-hosting landscape.

# Launch vLLM with OpenAI-compatible API
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 --max-model-len 4096

# Launch TGI via Docker
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.1-8B-Instruct

# Launch llama.cpp server
./llama-server -m llama-3.1-8b-instruct-q5_k_m.gguf \
  --port 8080 --ctx-size 4096

Scaling: Load Balancing & Rate Limiting

LLM inference is GPU-bound and slow compared to typical web APIs. Scaling strategies must account for long request durations and high per-request cost.

Horizontal Scaling

• Replicate your inference service behind a load balancer (NGINX, ALB, Traefik)
• Use least-connections routing (not round-robin) — LLM requests vary wildly in duration
• Autoscale on GPU utilisation or queue depth, not CPU

Rate Limiting

• Token-bucket or sliding-window per API key / user
• Limit both requests/minute and tokens/minute (a single long request can be as expensive as 100 short ones)
• Return 429 Too Many Requests with a Retry-After header

# Example: FastAPI rate limiter with slowapi
from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/v1/chat/completions")
@limiter.limit("20/minute")
async def chat(request: Request, body: ChatRequest):
    return await generate_response(body)

Caching Strategies

LLM inference is expensive. Caching identical or semantically similar requests can cut costs by 30-80% in many production workloads.

import hashlib, json, redis

r = redis.Redis()

def cached_completion(messages, model, **kwargs):
    cache_key = hashlib.sha256(
        json.dumps({"messages": messages, "model": model}, sort_keys=True).encode()
    ).hexdigest()

    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)

    response = client.messages.create(model=model, messages=messages, **kwargs)
    r.setex(cache_key, 3600, json.dumps(response.to_dict()))
    return response

CI/CD, Canary Deploys & Rollback

LLM applications need special CI/CD considerations because behaviour changes can be subtle and hard to test with unit tests alone.

CI Pipeline for LLM Apps

1. Lint & type-check — standard code quality gates
2. Unit tests — mock LLM responses, test parsing and orchestration logic
3. Eval suite — run a curated set of prompts against the real model; assert on quality metrics (accuracy, format compliance, latency)
4. Build & push container — tag with git SHA for traceability

Canary Deployment

• Route 5-10% of traffic to the new version
• Compare metrics: latency p50/p99, error rate, eval scores, user feedback signals
• Promote to 100% only if metrics hold for 30+ minutes

Rollback Triggers

• Error rate exceeds baseline by >2x
• p99 latency increases by >50%
• Eval regression detected on safety or accuracy metrics
• Keep the previous container image tagged :rollback for instant revert

Lab & Homework