Theory

Reading time: ~20 minutes (~1500 words)  ·  Prerequisites: Familiarity with Docker, basic networking, REST APIs

1. The Deployment Gap

Building an LLM prototype is fast. A few API calls wrapped in a Flask or FastAPI app, and you have something to demo. But the gap between a working demo and a production deployment is wide. Production means handling concurrent users, surviving traffic spikes, recovering from failures, managing costs at scale, and shipping updates without breaking existing functionality.

This lesson covers the architectural decisions and operational practices that bridge that gap. Whether you are wrapping a hosted API (Claude, GPT) or self-hosting an open model (Llama, Mistral), the deployment patterns are broadly similar — the differences lie in infrastructure requirements.

2. Architecture Patterns

Most LLM applications fall into one of three architectural patterns, though many production systems combine elements of all three.

API Gateway Pattern

The simplest production architecture places a reverse proxy or API gateway in front of your LLM service. The gateway handles authentication, rate limiting, request routing, and retry logic. Your application server sits behind it and communicates with the LLM provider (or your self-hosted model). This pattern works well for synchronous request-response workloads where the client waits for the full response before proceeding.

Tools like Kong, NGINX, AWS API Gateway, or even a simple FastAPI middleware layer serve this role. The gateway is also the right place to inject logging and observability hooks — every request and response passes through it.

Queue-Based Processing

For workloads that do not require immediate responses — document summarisation pipelines, batch classification, nightly report generation — a message queue decouples the request from the processing. The client submits a job and receives a job ID. Workers pull jobs from the queue and process them at a sustainable rate, writing results to a database that the client can poll or subscribe to.

This pattern is essential when your processing rate is limited by GPU capacity or API rate limits. It also provides natural backpressure: when the queue grows, you can autoscale workers or alert operators, rather than dropping requests or overwhelming the inference service.

Streaming

Chat interfaces and real-time applications benefit enormously from streaming. Instead of waiting 5-20 seconds for a complete response, the user sees tokens appear as they are generated, typically via Server-Sent Events (SSE) or WebSockets. Both the Anthropic and OpenAI APIs support streaming natively. When self-hosting with vLLM or TGI, streaming is also supported out of the box.

Streaming introduces complexity in error handling (what happens if the stream breaks mid-response?) and in caching (you cannot cache a streamed response until it completes), but the UX improvement for interactive applications is substantial.

3. Hosting Options

Serverless

AWS Lambda, Google Cloud Functions, and Azure Functions are attractive for low-traffic LLM wrappers. You pay only for invocations, and scaling is automatic. However, serverless has significant limitations for LLM workloads: cold start latency (up to 10 seconds), execution time limits (15 minutes on Lambda), no GPU access, and limited memory. Serverless is best suited for lightweight API-wrapping services that call a hosted LLM and do minimal processing.

Containers

Docker containers running on ECS, Cloud Run, or Kubernetes give you full control over the runtime environment. You can install any dependency, attach GPUs, and run long-lived processes. For self-hosted models, containers are the standard deployment unit. A typical setup uses a multi-stage Dockerfile to keep the production image small: a build stage installs dependencies and compiles any C extensions, and a runtime stage copies only the necessary artefacts.

Managed ML Platforms

SageMaker Endpoints, Vertex AI Prediction, and Azure ML Managed Endpoints provide autoscaling GPU inference with minimal operational overhead. You provide a model artefact and a serving container, and the platform handles scaling, health checks, and traffic splitting. The cost is higher than self-managed infrastructure, but the operational burden is much lower — no Kubernetes clusters to maintain, no GPU drivers to update.

GPU Cloud Providers

Providers like RunPod, Modal, and Replicate offer on-demand GPU instances with pay-per-second billing. Modal in particular offers a serverless-like developer experience for GPU workloads: you write a Python function, annotate it with the GPU type you need, and the platform handles containerisation and scaling. These are excellent for bursty inference workloads and for teams that do not want to manage GPU infrastructure.

4. Self-Hosting Open Models

If you need to serve an open-weight model (for data privacy, cost control, or customisation), three frameworks dominate the production landscape.

vLLM is the throughput champion. Its PagedAttention mechanism manages GPU memory like a virtual memory system, avoiding the waste that comes from reserving contiguous memory blocks for each request. Combined with continuous batching (adding new requests to an in-flight batch), vLLM achieves 2-4x higher throughput than naive implementations. It exposes an OpenAI-compatible API, making it a drop-in replacement for many applications.

Text Generation Inference (TGI) from Hugging Face is battle-tested in production on the Hugging Face Inference Endpoints service. Written in Rust with Python bindings, it supports tensor parallelism for multi-GPU setups and has tight integration with the Hugging Face model hub. If your model is on the Hub, TGI is often the fastest path to deployment.

llama.cpp and its server mode enable inference on CPUs with optional GPU offloading. Using quantised GGUF model formats (4-bit, 5-bit, 8-bit), it runs models that would normally require 40+ GB of VRAM on consumer hardware with 8-16 GB of RAM. Throughput is lower than GPU-based solutions, but the cost is dramatically lower, making it viable for edge deployments, local development, and cost-constrained applications.

5. Scaling Strategies

Horizontal Scaling

LLM inference is primarily GPU-bound. Vertical scaling (a bigger GPU) has a ceiling. Horizontal scaling means running multiple replicas of your inference service behind a load balancer. The critical nuance is the load balancing algorithm: round-robin is a poor choice because LLM request durations vary enormously (a 50-token response takes 1 second; a 2000-token response takes 40 seconds). Use least-connections or weighted least-connections to distribute load more evenly.

Rate Limiting

Rate limiting for LLM services should operate on two dimensions: requests per minute and tokens per minute. A single request generating 4000 output tokens consumes far more resources than 20 requests generating 50 tokens each. Implement token-bucket or sliding-window algorithms per API key, and return 429 Too Many Requests with a Retry-After header so clients can back off gracefully.

6. Caching

Caching is the single highest-leverage optimisation for most LLM applications. Two approaches are commonly used.

Exact-match caching hashes the full prompt (system message, user message, model name, and relevant parameters) and stores the response in a fast key-value store like Redis. This works best for deterministic tasks — classification, extraction, structured data generation — where the same input always produces the same desired output. Set temperature=0 for these workloads to maximise cache hit rates.

Semantic caching embeds the prompt into a vector and searches for similar previous prompts using approximate nearest-neighbour search. If a cached prompt is above a similarity threshold (typically 0.95+), the cached response is returned. This is powerful for FAQ-like workloads where users ask the same question in different words, but requires careful threshold tuning — too low and you return irrelevant cached answers.

7. CI/CD for LLM Applications

Traditional CI/CD pipelines test code correctness. LLM application pipelines must also test behaviour — the quality of the model's outputs. A robust pipeline includes:

• Standard gates: linting, type-checking, unit tests (with mocked LLM responses)
• Eval suite: a curated set of test prompts run against the live model, with assertions on accuracy, format compliance, safety, and latency
• Container build: tagged with the git SHA for full traceability
• Canary deployment: 5-10% of traffic routed to the new version, with automated comparison of latency, error rate, and eval metrics against the baseline
• Promotion or rollback: automated promotion to 100% if metrics hold, or instant rollback to the previous container image if any metric degrades

Treat prompts as versioned artefacts. A prompt change can alter application behaviour as dramatically as a code change. Store prompts in version control, include them in your eval suite, and gate prompt changes behind the same canary process as code changes.

8. Key Concepts Summary

Further Reading

• vLLM Documentation — Architecture overview, supported models, and deployment guides.
• TGI Documentation — Setup, configuration, and performance tuning for Text Generation Inference.
• llama.cpp on GitHub — Build instructions, GGUF format details, and server mode documentation.
• Anthropic Prompt Caching — How to use API-level prompt caching to reduce costs on repeated prefixes.
• Locust — Python-based load testing framework used in the lab.

Next lesson: Lesson 33 — Custom Training — scaling laws, pre-training decisions, and when to train your own model.