Theory: Open Source Models

Understanding the open-source LLM ecosystem, running models locally, and choosing between open and closed models.

The Rise of Open-Source LLMs

The open-source LLM revolution began in earnest when Meta released the original Llama weights in February 2023. Within months, researchers and hobbyists demonstrated that relatively small open models could achieve surprisingly strong performance, especially after fine-tuning. This triggered a wave of open releases from companies like Mistral AI, the Technology Innovation Institute, MosaicML, and Microsoft.

Today, the open-source ecosystem is vibrant and fast-moving. New models appear weekly on HuggingFace Hub, and the gap between the best open models and the best closed APIs continues to narrow. For many practical tasks, an open 7B or 13B model can match or exceed what GPT-3.5 delivered just a year ago.

It is important to distinguish between truly “open source” and “open weights.” Most models release only the trained weights, not the full training data or training code. Licences vary widely: some models (Mistral 7B, Falcon) use Apache 2.0 with no restrictions, while others (Llama) have custom licences that restrict commercial use above certain thresholds.

Major Model Families

Llama (Meta)

Meta’s Llama family is the most influential open model series. Llama 2 (July 2023) proved that open models could be commercially viable, with 7B, 13B, and 70B parameter variants. Llama 3 (April 2024) brought a significant quality jump, trained on 15 trillion tokens with a new tokenizer supporting 128K vocabulary. Llama 3.1 extended context to 128K tokens and introduced a 405B parameter variant — the first open model competitive with GPT-4 on many benchmarks.

Llama’s significance extends beyond the models themselves. Its architecture became the de facto standard for open LLMs, meaning that tooling built for Llama (fine-tuning scripts, inference engines, quantization tools) works across many other model families.

Mistral / Mixtral

Mistral AI, a Paris-based startup, released Mistral 7B in September 2023. It outperformed Llama 2 13B despite being half the size, thanks to architectural innovations like sliding-window attention and grouped-query attention (GQA). Mixtral 8x7B introduced a Mixture-of-Experts (MoE) architecture: the model has 46.7 billion total parameters but activates only 12.9 billion per token, achieving near-70B quality at a fraction of the inference cost. Mistral models use a permissive Apache 2.0 licence with no usage restrictions.

Falcon

The Technology Innovation Institute (TII) in Abu Dhabi released Falcon in May 2023. Its distinguishing feature was the RefinedWeb dataset — a carefully filtered and deduplicated web crawl that demonstrated the importance of data quality over quantity. Falcon 180B was briefly the largest open model before Llama 3.1 405B surpassed it.

MPT (MosaicML)

MosaicML (now part of Databricks) released MPT-7B and MPT-30B, built using their Composer training framework. MPT used ALiBi positional encodings, which allow the model to generalise to longer sequences than it was trained on. The commercial-friendly Apache 2.0 licence and focus on enterprise deployment made MPT popular for business applications.

Phi (Microsoft)

Microsoft’s Phi series takes a different approach: training smaller models on extremely high-quality data. Phi-3-mini (3.8B parameters) achieves performance comparable to Llama 3 8B on many benchmarks, despite being less than half the size. This makes Phi models ideal for edge deployment, mobile applications, and scenarios where compute is limited. The MIT licence is fully permissive.

HuggingFace Hub and Transformers

HuggingFace Hub is the central repository for the open-source ML community. It hosts over 500,000 models, 100,000 datasets, and thousands of interactive demos (Spaces). For LLMs, the Hub is where you find model weights, model cards (documentation including benchmarks, intended use, and limitations), and community discussions.

The transformers library provides a unified Python interface for loading and running models. The key pattern is AutoModelForCausalLM and AutoTokenizer, which automatically select the correct model architecture based on the model ID. The apply_chat_template method handles converting messages into the model-specific prompt format, which varies between model families.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "mistralai/Mistral-7B-Instruct-v0.3"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful coding assistant."},
    {"role": "user", "content": "Write a Python function to check if a number is prime."},
]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, temperature=0.7, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

When browsing HuggingFace, pay attention to model cards. They document the training data, evaluation results, known limitations, and licence terms. The Open LLM Leaderboard provides standardised benchmark comparisons, though you should always test on your own tasks — benchmark performance does not always predict real-world usefulness.

Running Models Locally

Ollama

Ollama is the easiest way to run open-source models locally. It provides a Docker-like experience: ollama pull llama3.1 downloads a quantized model, and ollama run llama3.1 starts an interactive chat. Behind the scenes, Ollama uses llama.cpp for inference and manages models, quantization, and hardware acceleration automatically.

Critically, Ollama exposes an OpenAI-compatible REST API on port 11434. This means you can point any OpenAI SDK client at your local Ollama instance by changing the base_url. This makes it trivial to swap between a local model and a cloud API in your application code.

# Install Ollama (macOS)
curl -fsSL https://ollama.com/install.sh | sh

# Pull and run a model
ollama pull llama3.1
ollama run llama3.1

# List available models
ollama list

# Serve the API (runs automatically after install)
ollama serve

llama.cpp

llama.cpp is the C/C++ inference engine that powers most local LLM tools. Written by Georgi Gerganov, it supports CPU inference, Apple Metal (GPU), NVIDIA CUDA, and Vulkan. It reads models in GGUF format (a binary format designed for efficient loading and memory mapping). If you need maximum control over inference parameters, batch processing, or custom server configurations, llama.cpp gives you that control. For most users, Ollama provides a friendlier wrapper around the same engine.

Quantization

Full-precision LLMs store each parameter as a 16-bit floating point number (FP16 or BF16). A 7B parameter model therefore needs approximately 14 GB of memory just for the weights. Quantization reduces this by storing parameters at lower precision — 8-bit, 4-bit, or even 2-bit — with surprisingly little quality loss.

GGUF (llama.cpp)

GGUF is the standard format for llama.cpp and Ollama. Models are available in multiple quantization levels, indicated by names like Q4_K_M (4-bit, K-quant method, Medium quality) or Q8_0 (8-bit). The K-quant methods are more sophisticated: they use different bit widths for different layers based on sensitivity analysis, preserving quality where it matters most. For most use cases, Q4_K_M offers the best balance of quality and size.

GPTQ and AWQ

GPTQ and AWQ are GPU-focused quantization methods. GPTQ uses a one-shot weight quantization algorithm based on approximate second-order information. AWQ (Activation-aware Weight Quantization) improves on GPTQ by identifying which weights are most important based on activation patterns, then protecting those weights from aggressive quantization. AWQ generally offers faster inference than GPTQ at similar quality levels.

bitsandbytes

The bitsandbytes library integrates directly with HuggingFace Transformers, allowing you to load models in 4-bit or 8-bit precision with a single parameter. This is particularly useful for fine-tuning with QLoRA, where the base model is loaded in 4-bit and only the LoRA adapters are trained in full precision.

Open vs Closed Models: Trade-offs

The choice between open-source models and proprietary APIs involves multiple trade-offs across cost, privacy, customization, capability, and safety.

When to Use Open-Source Models

• Data privacy requirements: Healthcare, finance, government, or any domain where data cannot leave your infrastructure
• High-volume, cost-sensitive: When per-token API costs would be prohibitive at your scale
• Offline or edge deployment: Devices without reliable internet, embedded systems, mobile apps
• Deep customization: When you need to fine-tune on domain-specific data or modify the model architecture
• Experimentation: Research, education, or prototyping where you want to understand model internals

When to Use Closed API Models

• Maximum capability needed: Tasks requiring frontier reasoning, where the best closed models significantly outperform open alternatives
• Fast time-to-market: No infrastructure to provision; start making API calls immediately
• Low or variable volume: When usage is unpredictable and you do not want idle hardware costs
• Safety-critical applications: When you want the provider’s safety team handling alignment and content filtering

Further Reading

• HuggingFace Model Hub
• Ollama
• llama.cpp on GitHub
• HuggingFace Transformers Documentation
• Meta Llama
• Mistral AI