The open-source LLM ecosystem has exploded. This lesson covers the major model families, how to run them locally, quantization techniques, and when to choose open-source over proprietary APIs.
Since Meta released Llama in early 2023, the open-source ecosystem has grown rapidly. Today there are dozens of competitive model families.
| Model | Creator | Sizes | Licence |
|---|---|---|---|
| Llama 3.1 | Meta | 8B, 70B, 405B | Llama 3.1 Community |
| Mistral / Mixtral | Mistral AI | 7B, 8x7B, 8x22B | Apache 2.0 |
| Falcon | TII (UAE) | 7B, 40B, 180B | Apache 2.0 / TII |
| MPT | MosaicML | 7B, 30B | Apache 2.0 |
| Phi-3 | Microsoft | 3.8B, 7B, 14B | MIT |
Key trend: Smaller models are catching up. Phi-3-mini (3.8B) and Llama 3.1 8B rival GPT-3.5 on many benchmarks while running on a single consumer GPU.
Meta’s Llama models are the most widely adopted open-source LLMs, forming the backbone of the ecosystem.
Why Llama matters: Its open weights created an entire ecosystem of fine-tuned models, tooling, and research. Most open-source LLM infrastructure was built around Llama first.
HuggingFace Hub is the central repository for open-source models, datasets, and spaces. The Transformers library is the standard interface for loading and running models.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
messages = [{"role": "user", "content": "Explain transformers in 3 sentences."}]
inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to("cuda")
outputs = model.generate(inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))Model cards: documentation, benchmarks, licence info. Spaces: hosted demos. Datasets: training and evaluation data. Leaderboards: standardised benchmarks (Open LLM Leaderboard).
You do not need a cloud GPU to run open-source models. Quantized versions of 7B–13B models run well on modern laptops.
ollama pull llama3.1ollama run llama3.1# Using Ollama's OpenAI-compatible API
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
model="llama3.1",
messages=[{"role": "user", "content": "What is quantization?"}],
)
print(response.choices[0].message.content)Quantization reduces model precision from 16-bit to 8-bit, 4-bit, or lower — shrinking memory requirements and speeding up inference with minimal quality loss.
| Format | Bits | Tool | Use Case |
|---|---|---|---|
| GGUF | 2–8 bit | llama.cpp | CPU / Apple Silicon inference |
| GPTQ | 4 bit | AutoGPTQ | GPU inference (CUDA) |
| AWQ | 4 bit | AutoAWQ | GPU inference, faster than GPTQ |
| bitsandbytes | 4/8 bit | HuggingFace | Training and inference via Transformers |
A model with N billion parameters in FP16 needs roughly 2N GB of VRAM. At 4-bit quantization, that drops to roughly 0.5N GB. So a 7B model needs ~14 GB at FP16 but only ~3.5 GB at Q4, fitting easily on a laptop.
Q4_K_M = 4-bit quantization, K-quant method, Medium quality. Higher numbers = better quality but more memory.No GPU required: Ollama handles quantization automatically. A modern laptop with 8 GB+ RAM can run 7B models comfortably.