Lesson 03 — Policy-Based Routing

How Linux picks between multiple routing tables.

The routing table is just one table

Lesson 02 showed how the kernel consults the main routing table for every packet. But Linux actually ships with three built-in tables:

• local (table 255) — kernel-managed; host addresses and broadcasts. Always consulted first.
• main (table 254) — what ip route show displays. Your static routes live here.
• default (table 253) — historically for postrouting; almost never populated today.

Plus up to 252 user-defined tables, each with its own independent prefix set. The question is: who decides which table to use?

Two lookups, not one

When the kernel needs to route a packet, it performs two distinct lookups in sequence:

This two-step process lets you route different traffic classes through entirely different forwarding paths — without touching the main table at all.

Anatomy of an ip rule

Every rule has three parts: a priority, a selector, and a target table.

ip rule add pref 100 from 10.10.0.0/24 lookup 200

• pref 100 — priority; lower numbers are evaluated first
• from 10.10.0.0/24 — selector: match packets whose source address falls in this subnet
• lookup 200 — target: if this rule matches, consult routing table 200

Selectors at a glance

A single rule can combine multiple selectors — all must match for the rule to fire.

Use case 1 — Source-based routing

A server has two uplinks (eth0 → ISP-A, eth1 → ISP-B). Traffic from subnet 10.10.0.0/24 must always leave via ISP-B, regardless of the destination.

# Table 200 gets its own default route via ISP-B
ip route add default via 203.0.113.1 dev eth1 table 200

# Rule: any packet sourced from 10.10.0.0/24 uses table 200
ip rule add from 10.10.0.0/24 lookup 200 pref 100

# All other traffic falls through to table main → ISP-A
ip route add default via 198.51.100.1 dev eth0

Use case 2 — fwmark-driven routing

Netfilter's MARK target paints a 32-bit tag onto a packet in the mangle table. The ip rule fwmark selector then routes by that tag.

# Mark packets from VPN clients with 0x42
iptables -t mangle -A PREROUTING -i tun0 -j MARK --set-mark 0x42

# Route all marked packets through table 200
ip rule add fwmark 0x42 lookup 200 pref 100

# Table 200's default: force through the secondary WAN
ip route add default via 10.99.0.1 dev eth1 table 200

Persisting marks across a flow — CONNMARK

# Three-rule mangle chain for CONNMARK
# 1. Restore saved mark for ESTABLISHED/RELATED packets
iptables -t mangle -A PREROUTING -m conntrack --ctstate ESTABLISHED,RELATED \
  -j CONNMARK --restore-mark

# 2. Mark NEW packets
iptables -t mangle -A PREROUTING -m conntrack --ctstate NEW \
  -i tun0 -j MARK --set-mark 0x42

# 3. Save the mark back to conntrack for future packets
iptables -t mangle -A PREROUTING -j CONNMARK --save-mark

Use case 3 — Multi-WAN with metrics

Two default routes with different metrics provide automatic failover. Lower metric = preferred. When the primary link goes down, the kernel falls back to the higher-metric route automatically.

# Primary default (metric 100 — preferred)
ip route add default via 10.99.0.1 dev eth0 metric 100

# Backup default (metric 200 — used only if eth0 fails)
ip route add default via 10.99.1.1 dev eth1 metric 200

# Optional: source-policy overlay to guarantee return-path symmetry
ip rule add from 10.99.0.0/24 lookup isp-a pref 100
ip rule add from 10.99.1.0/24 lookup isp-b pref 110

The rp_filter gotcha

When you add policy routing, return packets may arrive on an interface that doesn't match the route the kernel would use to reach the sender. The Reverse Path Filter will silently drop these packets.

# Check current rp_filter values
sysctl net.ipv4.conf.all.rp_filter
sysctl net.ipv4.conf.eth0.rp_filter

# 0 = off  |  1 = strict (default)  |  2 = loose
# For PBR, use loose mode (2) on the relevant interface
sysctl -w net.ipv4.conf.eth0.rp_filter=2
sysctl -w net.ipv4.conf.all.rp_filter=2

# Persist in /etc/sysctl.d/
echo "net.ipv4.conf.all.rp_filter=2" >> /etc/sysctl.d/99-pbr.conf

Under the hood — iproute2 internals

Understanding the data structures behind ip rule and ip route helps you reason about performance and design rule lists correctly.

Toolkit

Five commands cover 90% of PBR debugging:

# 1. Show the rule list (and which table each rule points to)
ip rule show

# 2. Show routes in a specific table
ip route show table 200

# 3. Show all tables at once (filter noise)
ip route show table all | grep -v "^broadcast\|^local\|^multicast"

# 4. Simulate the full two-step lookup for a given source/destination
ip route get 8.8.8.8 from 10.10.0.5

# 5. Watch conntrack marks on live traffic
conntrack -L -o extended | grep mark

# Bonus: trace the netfilter path and routing decision together
iptables -t mangle -A PREROUTING -j LOG --log-prefix "[mangle] "
# (remember to delete after debugging)

Common mistakes

• Forgetting the return route. Adding a rule for outbound traffic (src-based) without also ensuring replies come back on the same interface creates asymmetric routing. Always add a matching rule for the source subnet of each WAN gateway.
• rp_filter=1 in strict mode. Strictly correct packets dropped silently. Check with sysctl net.ipv4.conf.all.rp_filter first before spending hours in tcpdump.
• Rule list explosion. Each rule is walked linearly per packet. 50+ rules cause measurable latency on high-pps routers. Consolidate by using fwmark instead of many source-IP ranges.
• Forgetting CONNMARK for stateful flows. MARK alone only marks a single packet. If you need all packets in a TCP flow to use the same path, add --save-mark / --restore-mark.
• Rules not persistent across reboots. ip rule add is ephemeral like ip route add. Write rules to /etc/iproute2/rt_rules or your distro's network configuration (NetworkManager, systemd-networkd, ifupdown).

Recap

• Two-step routing — ip rule picks a table; that table is consulted for the longest prefix match.
• Selectors — from, to, fwmark, iif, oif, tos; combine freely, lower pref wins.
• CONNMARK — --save-mark / --restore-mark makes marks sticky to the entire conntrack flow.
• Watch out for rp_filter (use loose mode=2), return-route symmetry, and rule list length.