Theory

Reading time: ~20 minutes (~1500 words) · Prerequisites: Familiarity with LLM API calls (Lessons 02–03), basic RAG concepts (Lesson 10)

1. The Case for Orchestration Frameworks

When you first start building with LLMs, the workflow is simple: construct a prompt, send it to an API, parse the response. A few dozen lines of Python and the anthropic or openai SDK are all you need.

As applications grow more complex, patterns recur. You need to chain multiple LLM calls together, where the output of one becomes the input of the next. You need to manage conversation history across turns. You need to let the model call external tools and decide which tool to use. You need to load documents, split them into chunks, embed them, store them in a vector database, retrieve relevant chunks at query time, and synthesise an answer.

You could build all of this from scratch — and for simple applications, you should. But orchestration frameworks like LangChain and LlamaIndex exist to provide tested, reusable implementations of these common patterns. They save you from re-inventing the wheel, but they also add abstraction layers that can make debugging harder and hide what the model actually receives.

The central question is not "which framework should I use?" but rather: does my application's complexity justify the overhead of a framework?

2. LangChain: The General-Purpose Toolkit

LangChain is the most widely adopted LLM orchestration framework. Originally released in late 2022, it has evolved significantly. The current recommended approach is LangChain Expression Language (LCEL), which replaced the older LLMChain and SequentialChain classes.

Core Abstractions

LangChain is built around the Runnable interface. Every component — prompt templates, chat models, output parsers, retrievers, tools — implements this interface, which provides four methods: invoke() for single inputs, batch() for multiple inputs, stream() for streaming output, and their async equivalents. This uniform interface is what makes the pipe operator work.

Chains with LCEL

The simplest LCEL chain connects a prompt template to a model to a parser:

from langchain_anthropic import ChatAnthropic
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    ("system", "Summarise the following text in {style} style."),
    ("human", "{text}")
])
model = ChatAnthropic(model="claude-sonnet-4-20250514")
parser = StrOutputParser()

chain = prompt | model | parser
result = chain.invoke({"style": "bullet-point", "text": "...long document..."})

You can extend this by piping the result into another chain. For example, after summarising, you could extract keywords, translate the summary, or format it as JSON. Each step is a composable Runnable.

Memory

LangChain provides several memory strategies for multi-turn conversations:

ConversationBufferMemory — stores every message verbatim. Simple but grows unbounded.
ConversationBufferWindowMemory — keeps only the last k exchanges. Bounded but loses older context.
ConversationSummaryMemory — uses the LLM itself to periodically summarise the conversation history. More expensive but handles long conversations well.
ConversationTokenBufferMemory — keeps messages up to a token budget, then drops the oldest. Good for cost control.

In practice, many teams implement their own memory management because the built-in options do not always align with their specific requirements. This is one area where frameworks sometimes add complexity without sufficient payoff.

Agents and Tools

A LangChain agent is an LLM that can decide which tools to call and in what order. You define tools — functions the model can invoke — and the agent runtime handles the loop: the model proposes a tool call, the runtime executes it, the result is fed back to the model, and the model decides whether to call another tool or return a final answer.

from langchain_core.tools import tool
from langchain.agents import create_tool_calling_agent, AgentExecutor

@tool
def search_database(query: str) -> str:
    """Search the product database for items matching the query."""
    # ... actual database search logic ...
    return results

tools = [search_database]
agent = create_tool_calling_agent(model, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools)
response = executor.invoke({"input": "Find all products under $50"})

3. LlamaIndex: The Data Framework

LlamaIndex takes a different philosophy. While LangChain is a general-purpose toolkit, LlamaIndex is purpose-built for one thing: connecting LLMs to your data. If your primary pattern is "load documents, index them, query them with natural language," LlamaIndex provides a more streamlined experience.

The Data Pipeline

LlamaIndex defines a clear pipeline for data-augmented LLM applications:

Loading — Ingest data from files, databases, APIs, or web pages using connectors. SimpleDirectoryReader handles local files; LlamaHub provides hundreds of community connectors.
Indexing — Parse documents into nodes (chunks), generate embeddings, and store them in an index. VectorStoreIndex is the most common choice for semantic similarity search.
Querying — Convert the index into a query engine that retrieves relevant nodes and synthesises an answer using the LLM.

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.anthropic import Anthropic
from llama_index.embeddings.openai import OpenAIEmbedding

Settings.llm = Anthropic(model="claude-sonnet-4-20250514")
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

documents = SimpleDirectoryReader("./company_docs").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(similarity_top_k=3)
response = query_engine.query("What is our refund policy?")

Index Types

LlamaIndex offers several index types, each suited to different query patterns:

Index Type	How It Works	Best For
VectorStoreIndex	Embeds nodes, retrieves by cosine similarity	Semantic search, most RAG use cases
SummaryIndex	Scans every node sequentially	Summarisation over entire corpus
TreeIndex	Builds a hierarchical summary tree	Multi-level summarisation
KeywordTableIndex	Extracts keywords, matches by keyword overlap	Keyword-based retrieval

Response Synthesis

After retrieval, LlamaIndex must synthesise an answer from the retrieved nodes. The framework provides several strategies:

compact (default) — Stuffs as many nodes as possible into a single LLM call. Fast and cheap.
refine — Processes nodes one at a time, refining the answer iteratively. Better for detailed answers from many sources.
tree_summarize — Builds a bottom-up summary tree. Good for synthesising over large numbers of nodes.

4. When to Use What

Here is a practical decision framework:

Use direct API calls when:

Your application makes single LLM calls (chat, summarisation, classification)
You need maximum control over prompts and can inspect every token sent to the model
You want to minimise dependencies and keep your stack simple
Performance and latency are critical — frameworks add overhead

Use LangChain when:

You are building complex agent workflows with multiple tools
You need to chain multiple LLM calls with branching logic
Provider-agnostic design is a requirement (you may switch models later)
You want built-in support for memory, callbacks, and tracing

Use LlamaIndex when:

Your primary pattern is RAG: load, index, query
You need to ingest data from diverse sources (PDFs, databases, APIs)
You want to experiment with different index types and retrieval strategies quickly
Document QA is the core feature of your application

These are not mutually exclusive. Some teams use LlamaIndex for their retrieval layer and LangChain for the agent/orchestration layer on top. Others use neither and build everything with direct API calls plus a vector database client. The right answer depends on your team's needs, your application's complexity, and how much abstraction you are comfortable debugging through.

5. Common Pitfalls

Abstraction blindness — When something goes wrong, you need to see exactly what prompt was sent to the model. Both frameworks provide tracing (LangSmith for LangChain, callbacks for LlamaIndex), but many developers forget to enable it.
Version churn — Both frameworks iterate rapidly. LangChain in particular has undergone major API changes (the migration from chains to LCEL). Pin your versions and read changelogs before upgrading.
Over-engineering — Do not reach for a framework to make a single API call. The anthropic SDK is simpler, faster, and easier to debug for straightforward use cases.
Default chunking — LlamaIndex's default chunk size and overlap may not suit your documents. Always experiment with chunking parameters; they have a large impact on RAG quality.
Ignoring cost — Agent loops and refine-mode synthesis can make many LLM calls behind the scenes. Monitor token usage carefully, especially during development.

6. Key Concepts Summary

Concept	What It Means	Why It Matters
LCEL	LangChain's pipe-based composition syntax	The modern way to build LangChain applications
Runnable	Universal interface for all LangChain components	Enables composability via the pipe operator
VectorStoreIndex	LlamaIndex's primary index for semantic search	The default choice for most RAG applications
Query Engine	Combines retrieval and synthesis in LlamaIndex	The main interface for asking questions of your data
Response Synthesizer	Strategy for combining retrieved nodes into an answer	Affects quality, cost, and latency of RAG responses
Framework vs. Direct API	The decision to add an abstraction layer or not	Impacts debugging, dependencies, and development speed