VXLAN: stretching a layer 2 network across a routed one

Some workloads need to be on the same layer 2 segment, in the same broadcast domain and subnet, even when they sit in different racks: virtual machines that move between hosts, clustered applications that rely on broadcast or on a shared subnet, or legacy systems that were never designed for routing. The traditional answer is to stretch a VLAN across the switches between them. That makes the whole path one big layer 2 network, with the scaling and stability problems that come with it, and a VLAN ID is only 12 bits, so there are at most 4,094 of them.

Modern data center networks are routed instead: every link is layer 3, and traffic is spread across many equal-cost paths. VXLAN (Virtual Extensible LAN) lets you keep the routed network and still give those workloads their layer 2 segment. It carries Ethernet frames inside UDP packets, so a layer 2 segment becomes an overlay that runs over any IP network. The segment ID is 24 bits, which allows about 16 million of them.

This post explains how VXLAN works and builds a small lab with Linux bridges to show it: three hosts on one subnet, connected through a routed network, with the actual packets on the wire.

How VXLAN works

Encapsulation. A VXLAN device takes an Ethernet frame, adds an 8-byte VXLAN header, and sends the result as a UDP packet to port 4789:

┌──────────┬──────────┬──────────┬─────────┬───────────────────┐
│ Outer    │ Outer IP │ UDP      │ VXLAN   │ Original Ethernet │
│ Ethernet │ header   │ header   │ header  │ frame             │
│          │ 20 bytes │ 8 bytes  │ 8 bytes │ (14-byte header   │
│          │          │ dst port │ carries │ + payload)        │
│          │          │ 4789     │ the VNI │                   │
└──────────┴──────────┴──────────┴─────────┴───────────────────┘

The headers added around the original frame are the outer IP header (20 bytes), the UDP header (8) and the VXLAN header (8), which is 36 bytes. The original frame also keeps its own 14-byte Ethernet header inside the tunnel, so an IP packet that crosses VXLAN grows by 50 bytes. The underlay therefore has to carry packets 50 bytes larger than the biggest IP packet the hosts send.

VNI. The VXLAN Network Identifier is the 24-bit segment ID in the VXLAN header. Frames with the same VNI belong to the same layer 2 segment. It plays the role a VLAN ID plays on a switch, with far more room.

VTEP. A VXLAN tunnel endpoint is a device that encapsulates frames into VXLAN and decapsulates them again. It has an IP address on the routed network, and that address is the source and destination of the tunnel packets. On a leaf switch or a Linux host, the VTEP is a virtual device attached to a bridge, so the local ports and the tunnel are ports of the same layer 2 segment.

The underlay. The routed IP network that carries the tunnel packets is called the underlay. It only needs to deliver IP packets between the VTEP addresses. It never looks at the frames inside, and it doesn’t know the VNI.

Broadcast, unknown unicast and multicast (BUM). A layer 2 segment depends on frames that go to everyone: ARP requests, for example, are broadcasts. The VTEP has to deliver these to every other VTEP in the segment. There are two ways to do it: send them to an IP multicast group that all the VTEPs join, or head-end replication, where the VTEP keeps a list of the other VTEPs and sends each one its own unicast copy of the frame. The lab uses head-end replication, because it needs nothing from the underlay.

Flood and learn. A VTEP learns the way a switch does. When an encapsulated frame arrives, the VTEP records its source MAC address together with the source IP of the VTEP that sent it. Later frames to that MAC go straight to that one VTEP. Until a destination is learned, frames to it are flooded to every VTEP, like unknown unicast on a normal switch. The table holding all this is the forwarding database, the FDB.

Learning from traffic means the VTEPs need no other way to know where MAC addresses live, which keeps this lab small. In larger deployments a control plane, such as BGP EVPN, usually distributes that information ahead of time instead of leaving it to flooding.

The lab

A spine router connects three leaves over a routed IPv4 network. Each leaf is a VTEP with one host attached, and the three hosts share a single subnet, 192.168.100.0/24, in VNI 100.

                   spine
           ┌─────────┼─────────┐
         leaf1     leaf2     leaf3     ← VTEPs, VNI 100
           │         │         │
          h1        h2        h3       ← 192.168.100.0/24

The leaves’ eth2 ports have no address because they are bridge ports. Each leaf’s VTEP address is the address on its eth1. The underlay is plain static IP: each leaf has a default route to the spine, which has a connected route to every leaf. There is no routing protocol.

The nodes are containers deployed with containerlab. You need containerlab and podman. They run a small image built from the official Alpine Linux image with the iproute2 and tcpdump packages. The lab contains its two-line Containerfile, and you build the image once:

sudo podman build -t localhost/vxlan-node:latest image/

Get the files from the lab repository and cd into vxlan-basics.

The configuration

The topology file gives every node the same image, and the three leaves each get a small set of variables and the same script. This is the top of the file and one leaf:

name: vxlan

topology:
  kinds:
    linux:
      image: localhost/vxlan-node:latest
      sysctls:
        net.ipv6.conf.all.disable_ipv6: 1
        net.ipv6.conf.default.disable_ipv6: 1

  nodes:
    leaf1:
      kind: linux
      binds:
        - configs/leaf.sh:/leaf.sh
      env:
        VTEP_IP: 10.0.1.2
        GATEWAY: 10.0.1.1
        REMOTE_VTEPS: "10.0.2.2 10.0.3.2"
      exec:
        - sh /leaf.sh

The sysctls turn IPv6 off on every node. With IPv6 on, each node announces itself with multicast listener reports as soon as an interface comes up. VXLAN floods them like any other multicast, and every VTEP would learn the senders’ MAC addresses before we have sent a single packet. Turning IPv6 off keeps the only traffic in the lab the traffic we generate.

The script, leaf.sh, does the VXLAN work:

#!/bin/sh
set -e

# Underlay: an address on the spine-facing link, and a route to the other leaves
ip addr add "$VTEP_IP/24" dev eth1
ip route add default via "$GATEWAY"

# Overlay: a bridge that joins the host-facing port and the VXLAN device
ip link add br0 type bridge mcast_snooping 0
ip link add vxlan100 type vxlan id 100 local "$VTEP_IP" dstport 4789
ip link set vxlan100 master br0
ip link set eth2 master br0
ip link set br0 up
ip link set vxlan100 up
ip link set eth2 up

# Flood list: send broadcast, unknown-unicast and multicast frames to every remote VTEP
for vtep in $REMOTE_VTEPS; do
  bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst "$vtep"
done

Deploy

sudo containerlab --runtime podman deploy -t topology.clab.yml

The underlay is ordinary IP. Check that leaf1 can reach leaf2’s VTEP address across the spine:

sudo podman exec clab-vxlan-leaf1 ping -c 2 10.0.2.2

--- 10.0.2.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.057/0.096/0.136 ms

The VXLAN device and the bridge ports on leaf1:

sudo podman exec clab-vxlan-leaf1 ip -d link show vxlan100
sudo podman exec clab-vxlan-leaf1 bridge link show

4: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ca:15:1f:2c:20:09 brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 65535 
    vxlan id 100 local 10.0.1.2 srcport 0 0 dstport 4789 ttl auto ageing 300 reserved_bits 0xf7ffffff000000ff 
797: eth2@if798: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 master br0 state forwarding priority 32 cost 2 
4: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100 

The vxlan line shows the VNI (id 100), the VTEP address (local 10.0.1.2) and the port (dstport 4789). The bridge has two ports: eth2 toward the host and vxlan100, the tunnel.

Before any traffic

The VXLAN device’s forwarding database on leaf1:

sudo podman exec clab-vxlan-leaf1 bridge fdb show dev vxlan100

ca:15:1f:2c:20:09 vlan 1 master br0 permanent
ca:15:1f:2c:20:09 master br0 permanent
00:00:00:00:00:00 dst 10.0.2.2 self permanent
00:00:00:00:00:00 dst 10.0.3.2 self permanent

The first two lines are the bridge’s own address. The last two are the flood list we configured: any frame that has to be flooded goes to both 10.0.2.2 and 10.0.3.2. There is no entry for any host yet.

The first ping

Ping from h1 to h2, which are in the same subnet but separated by the routed network. To see what happens inside the tunnel, capture on leaf1’s underlay link first. The capture ends by itself after seven packets:

sudo podman exec clab-vxlan-leaf1 tcpdump -nn -i eth1 udp port 4789 -v -c 7

Capture on h3 too, in a second terminal, then start the ping from a third:

sudo podman exec clab-vxlan-h3 tcpdump -nn -i eth1
sudo podman exec clab-vxlan-h1 ping -c 2 192.168.100.2

PING 192.168.100.2 (192.168.100.2): 56 data bytes
64 bytes from 192.168.100.2: seq=0 ttl=64 time=0.220 ms
64 bytes from 192.168.100.2: seq=1 ttl=64 time=0.058 ms

--- 192.168.100.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.058/0.139/0.220 ms

It works, and the TTL is still 64: no router decremented it, because from h1’s point of view h2 is on the same link. The capture on leaf1 shows what carried the packets:

21:01:48.460788 IP (tos 0x0, ttl 64, id 23145, offset 0, flags [none], proto UDP (17), length 78)
    10.0.1.2.58000 > 10.0.3.2.4789: VXLAN, flags [I] (0x08), vni 100
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.100.2 tell 192.168.100.1, length 28
21:01:48.460793 IP (tos 0x0, ttl 64, id 40985, offset 0, flags [none], proto UDP (17), length 78)
    10.0.1.2.58000 > 10.0.2.2.4789: VXLAN, flags [I] (0x08), vni 100
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.100.2 tell 192.168.100.1, length 28
21:01:48.460894 IP (tos 0x0, ttl 63, id 15278, offset 0, flags [none], proto UDP (17), length 78)
    10.0.2.2.58000 > 10.0.1.2.4789: VXLAN, flags [I] (0x08), vni 100
ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.100.2 is-at aa:c1:ab:ad:55:a5, length 28
21:01:48.460916 IP (tos 0x0, ttl 64, id 40986, offset 0, flags [none], proto UDP (17), length 134)
    10.0.1.2.57209 > 10.0.2.2.4789: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 27593, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.100.1 > 192.168.100.2: ICMP echo request, id 3, seq 0, length 64
21:01:48.460939 IP (tos 0x0, ttl 63, id 15279, offset 0, flags [none], proto UDP (17), length 134)
    10.0.2.2.57209 > 10.0.1.2.4789: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 65350, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.100.2 > 192.168.100.1: ICMP echo reply, id 3, seq 0, length 64
21:01:49.460985 IP (tos 0x0, ttl 64, id 40989, offset 0, flags [none], proto UDP (17), length 134)
    10.0.1.2.57209 > 10.0.2.2.4789: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 27787, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.100.1 > 192.168.100.2: ICMP echo request, id 3, seq 1, length 64
21:01:49.461009 IP (tos 0x0, ttl 63, id 15480, offset 0, flags [none], proto UDP (17), length 134)
    10.0.2.2.57209 > 10.0.1.2.4789: VXLAN, flags [I] (0x08), vni 100
IP (tos 0x0, ttl 64, id 65438, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.100.2 > 192.168.100.1: ICMP echo reply, id 3, seq 1, length 64

Each tunnel packet is an outer IP and UDP packet between two VTEP addresses, on port 4789 with vni 100, and the line below it is the original frame. Reading the seven packets in order:

1. The ARP request is sent twice, once to 10.0.3.2 and once to 10.0.2.2, at the same instant. h1 doesn’t know where 192.168.100.2 is, so its ARP broadcast is flooded, and head-end replication sends a copy to every VTEP on the flood list.
2. The ARP reply comes back once, from leaf2 (10.0.2.2) to leaf1. It is unicast, because leaf2 had learned from the request which VTEP h1 is behind.
3. Both pings go only to 10.0.2.2. After the ARP exchange, leaf1 knows where h2 is, so the frames are no longer flooded. leaf3 never sees them.

The sizes show the overhead. The ICMP packet inside is 84 bytes long (length 84), and the tunnel packet carrying it is 134 bytes (length 134): 50 bytes more. The ARP packets work the same way, 28 bytes inside and 78 outside. The UDP source port differs between the ARP frames (58000) and the pings (57209): implementations choose it from a hash of the inner headers (RFC 7348), so that when the underlay spreads traffic over several equal-cost paths, different flows can take different ones.

h3’s capture shows the flooding from the other side:

21:01:48.460880 ARP, Request who-has 192.168.100.2 tell 192.168.100.1, length 28

1 packet captured
1 packet received by filter

h3 received the broadcast ARP request, because it is on the same segment and the request was flooded to it. It received nothing else. The pings between h1 and h2 never reached it.

After the traffic

The leaves have learned where the hosts are:

sudo podman exec clab-vxlan-leaf1 bridge fdb show dev vxlan100
sudo podman exec clab-vxlan-leaf2 bridge fdb show dev vxlan100

aa:c1:ab:ad:55:a5 master br0 
ca:15:1f:2c:20:09 vlan 1 master br0 permanent
ca:15:1f:2c:20:09 master br0 permanent
aa:c1:ab:ad:55:a5 dst 10.0.2.2 self 
00:00:00:00:00:00 dst 10.0.2.2 self permanent
00:00:00:00:00:00 dst 10.0.3.2 self permanent

aa:c1:ab:e4:3c:b7 master br0 
aa:da:3f:02:ba:ad vlan 1 master br0 permanent
aa:da:3f:02:ba:ad master br0 permanent
aa:c1:ab:e4:3c:b7 dst 10.0.1.2 self 
00:00:00:00:00:00 dst 10.0.1.2 self permanent
00:00:00:00:00:00 dst 10.0.3.2 self permanent

Each leaf has one new entry. On leaf1, aa:c1:ab:ad:55:a5 is h2’s MAC address, and it now points at dst 10.0.2.2, leaf2’s VTEP address. On leaf2, h1’s MAC (aa:c1:ab:e4:3c:b7) points at 10.0.1.2. Neither leaf learned anything about h3, which hasn’t sent a frame. The flood entries haven’t changed. They are still there for the next frame to an unknown destination.

Tear it down

sudo containerlab --runtime podman destroy -t topology.clab.yml --cleanup