Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Lessons 01–23 (API usage, prompt engineering, RAG fundamentals)

1. The Customisation Spectrum

Every team building with LLMs eventually faces the same question: the base model is close to what we need, but not quite right. The output drifts from our brand voice. It occasionally hallucinates domain terminology. The JSON it produces is valid 90% of the time — but 90% is not production-grade.

At this point, three options are on the table: refine your prompts, add retrieval (RAG), or fine-tune the model. These are not competing approaches — they are points on a customisation spectrum, each with different trade-offs in cost, complexity, and capability.

Prompt Engineering (Tier 1)

Prompt engineering is the cheapest and fastest way to steer model behaviour. A well-crafted system prompt with clear instructions and a few examples (few-shot prompting) can handle the vast majority of use cases. The advantages are obvious: zero upfront cost, instant iteration, and no infrastructure beyond the API. The limitation is equally clear: every call carries the full prompt, so long system prompts increase latency and token costs. More importantly, there is a ceiling — no matter how detailed your instructions, the model's underlying distribution of responses remains unchanged.

Retrieval-Augmented Generation (Tier 2)

RAG addresses a specific limitation: the model lacks knowledge that exists in your documents. By retrieving relevant chunks at query time and injecting them into the prompt, you get grounded, up-to-date answers without modifying the model. RAG is the right choice when the problem is what the model knows. It is the wrong choice when the problem is how the model responds — retrieval does not change the model's writing style, output format, or domain-specific reasoning patterns.

Fine-Tuning (Tier 3)

Fine-tuning modifies the model's weights. It changes the probability distribution over tokens, shifting the model's default behaviour toward your desired output patterns. After fine-tuning, the model does not need lengthy instructions or examples in every prompt — the behaviour is baked in. This makes fine-tuning uniquely suited to problems of style, format, and domain-specific reasoning.

2. When Fine-Tuning Helps

Fine-tuning is most valuable when you need consistent, repeatable behaviour that prompt engineering cannot reliably achieve. Here are the strongest use cases.

Style and Tone Consistency

Consider a company that needs every customer-facing message — support emails, help articles, chatbot responses — to sound like the same person wrote them. A system prompt can describe the desired tone, but over thousands of outputs, drift is inevitable. Some responses will be too formal, others too casual. Fine-tuning on a corpus of approved communications teaches the model the exact voice, and that voice becomes its default.

Domain-Specific Terminology

In fields like medicine, law, and finance, precision of language is not optional. A radiology report must use specific anatomical terms and structured findings. A legal brief must cite cases in the correct format. Base models approximate this language but make subtle errors — using a synonym where only the canonical term is acceptable. Fine-tuning on domain-specific corpora fixes these systematic errors.

Structured Output Reliability

If your application requires the model to consistently produce valid JSON, XML, or a custom schema, prompt engineering gets you most of the way — but edge cases persist. A fine-tuned model that has seen thousands of correctly formatted outputs becomes dramatically more reliable, especially for complex nested structures.

Latency and Cost Reduction

A common pattern: you build a system that works well with a large model and a 3000-token system prompt. It is accurate but slow and expensive. Fine-tuning a smaller model (e.g., Llama 3 8B or GPT-4o-mini) to replicate this behaviour lets you drop the system prompt entirely and use a model that runs 5-10x faster. At 10,000+ calls per day, the savings are substantial.

3. When NOT to Fine-Tune

Fine-tuning is one of the most over-applied techniques in production LLM work. Teams reach for it prematurely, spending weeks curating data and training models when a better prompt would have solved the problem in an afternoon. Here are the anti-patterns.

Insufficient Data

Fine-tuning with fewer than 100 high-quality examples almost always produces a model that memorises the training set rather than generalising from it. The model will perform well on inputs that look like the training data and poorly on everything else. If you have fewer than 500 examples, invest your time in better prompts and few-shot examples instead.

Rapidly Changing Knowledge

Fine-tuning bakes information into model weights. If your knowledge base changes weekly — product catalogues, pricing, news, regulations — the model becomes stale the moment training finishes. RAG is designed for exactly this scenario: update the retrieval index, and the model immediately reflects the new information.

General-Purpose Tasks

Summarisation, translation, basic classification, Q&A — base models are already optimised for these tasks through their pre-training and RLHF alignment. Fine-tuning for general tasks typically yields marginal improvements that do not justify the cost. The exception is when you need these tasks performed in a very specific way (e.g., summarisation that always produces exactly three bullet points in a specific format).

Premature Optimisation

The most common anti-pattern is skipping prompt engineering entirely. Many teams assume fine-tuning is necessary without first spending a few days systematically iterating on their prompts. A structured approach — testing different system prompts, adding few-shot examples, using chain-of-thought reasoning — often achieves 90%+ of the desired behaviour at zero additional cost.

4. Fine-Tuning Approaches

If the decision framework points toward fine-tuning, the next question is which approach to use. The landscape has four main options.

Full Fine-Tuning

Full fine-tuning updates every weight in the model. It offers the highest potential quality but requires significant compute (multiple high-end GPUs), large datasets (tens of thousands of examples), and careful hyperparameter tuning to avoid catastrophic forgetting — where the model loses its general capabilities while learning your specific task. Full fine-tuning is typically reserved for organisations with dedicated ML teams and substantial compute budgets.

LoRA (Low-Rank Adaptation)

LoRA freezes the original model weights and trains small adapter matrices (typically 1-5% of the original parameter count) that are added to the model's attention layers. This dramatically reduces memory requirements and training time while achieving quality that is often indistinguishable from full fine-tuning. LoRA adapters are small files (10-100 MB) that can be swapped in and out, enabling multiple task-specific adaptations from a single base model.

QLoRA (Quantised LoRA)

QLoRA combines LoRA with 4-bit quantisation of the base model. This reduces the GPU memory needed to fine-tune a 7B-parameter model from ~28 GB to ~6 GB, making it feasible on consumer GPUs or free Colab T4 instances. The quality trade-off is minimal for most practical applications. QLoRA is the recommended starting point for individual developers and small teams.

API-Based Fine-Tuning

OpenAI, Together AI, Fireworks, and other providers offer fine-tuning as a managed service. You upload training data in JSONL format, specify hyperparameters, and receive a fine-tuned model endpoint. No GPU management, no training infrastructure. The trade-offs are reduced control (you cannot inspect or modify the training process), vendor lock-in, and ongoing per-token inference costs. However, for validating whether fine-tuning helps at all, API-based approaches offer the fastest path to an answer.

5. Data Requirements and Quality

The single most important factor in fine-tuning success is data quality. A model fine-tuned on 500 excellent examples will outperform one trained on 5,000 mediocre examples. Each training example should be an (input, output) pair that demonstrates exactly the behaviour you want.

Quality Criteria

Correctness: every output must be something you would be happy to ship to production
Diversity: examples should cover the full range of inputs the model will encounter, including edge cases
Consistency: all examples should follow the same format and style conventions
Representativeness: the distribution of examples should match the distribution of production traffic

A practical data curation workflow: start by generating candidate outputs with the base model using your best prompt, then have domain experts review and correct them. This is faster than writing examples from scratch and ensures the training data is realistic.

6. Cost Analysis Framework

Fine-tuning involves two cost categories: the one-time training cost and the ongoing inference cost. The decision to fine-tune should be grounded in a break-even analysis.

Calculate your current cost per call (tokens x price) and multiply by your daily volume to get your monthly baseline. Then estimate the fine-tuned model's per-call cost (typically lower due to shorter prompts and smaller models) and add the training cost. The break-even point is where cumulative savings from cheaper inference exceed the training investment.

For high-volume applications (10,000+ calls per day), the break-even point is often measured in days, not months. For low-volume applications (fewer than 100 calls per day), the training cost may never be recouped — and prompt engineering remains the better choice.

7. Key Takeaways

Principle	Detail
Start simple	Always try prompt engineering before fine-tuning
Knowledge vs. behaviour	RAG fixes knowledge gaps; fine-tuning fixes behavioural gaps
Data quality over quantity	500 excellent examples beat 5,000 mediocre ones
Cost justification	Run a break-even analysis before committing to training
Start with LoRA or API	Full fine-tuning is rarely the right first step