EVPN: a BGP control plane for VXLAN

VXLAN stretches a layer 2 segment across a routed network. A VXLAN tunnel endpoint (VTEP) wraps each Ethernet frame in a UDP packet and sends it to another VTEP, and a 24-bit segment ID, the VNI, says which segment the frame belongs to. What VXLAN does not define is how a VTEP finds out two things: which other VTEPs belong to a segment, and which VTEP a given MAC address lives behind.

Without a control plane there are only two answers. You can configure every VTEP with a list of all the others, which means touching every device each time one is added. And VTEPs can learn MAC addresses the way a switch does, from the frames that arrive, which means flooding every frame whose destination hasn’t been seen yet.

EVPN (Ethernet VPN) gives VXLAN a control plane. It uses BGP to distribute both pieces of information: VTEPs announce themselves, and announce the MAC addresses behind them. Adding a VTEP means configuring it and its BGP sessions, and the others find out on their own. EVPN is widely used as the control plane for VXLAN in data center fabrics.

This post explains how EVPN works and builds a small lab with FRR to show it: three leaves that discover each other and exchange MAC addresses over BGP, with no flood list configured anywhere.

How EVPN works

A new BGP address family. EVPN is carried in BGP as its own address family, l2vpn evpn. VTEPs peer with each other (directly, or through a route reflector) and exchange EVPN routes over those sessions. These routes describe layer 2 information, not IP prefixes.

Route types. EVPN defines several kinds of route. The lab uses two of them:

Type 3 builds the flood list. When a VTEP has a VNI, it sends a type-3 route for it. Every other VTEP that has the same VNI adds the sender to its flood list. This replaces the list you would otherwise configure by hand, and a new VTEP appears on every other VTEP as soon as its route arrives.

Type 2 replaces learning from traffic. When a leaf learns a host’s MAC address on a local port, it sends a type-2 route for that MAC with its own VTEP address as the next hop. The other leaves install the MAC in their forwarding database, pointing at that VTEP. The VXLAN device is set to not learn from the tunnel at all, so remote MAC addresses come from BGP only. Until a MAC’s route has arrived, frames to it are still flooded as unknown unicast.

Route distinguishers and route targets. Two BGP attributes keep the routes organized. A route distinguisher (RD) makes a route unique, so the same MAC in two segments, or announced by two VTEPs, doesn’t collide. A route target (RT) says which segment a route belongs to, so each VTEP imports only the routes for VNIs it has. FRR generates both for you: the RD from the router ID, and the RT from the AS number and the VNI. A third attribute, ET:8, marks the encapsulation as VXLAN.

What EVPN doesn’t change. Broadcast frames, such as ARP requests, still have to reach every VTEP in the segment, and they still do: they go to the flood list that the type-3 routes built. EVPN changes how the VTEPs know where things are, not how a broadcast is delivered.

The lab

A spine router connects three leaves over a routed IPv4 network. Each leaf is a VTEP with one host attached, and the three hosts share a subnet, 192.168.100.0/24, in VNI 100. The leaves run FRR and peer with each other over iBGP in AS 65000.

                   spine
           ┌─────────┼─────────┐
         leaf1     leaf2     leaf3     ← VTEPs (FRR), iBGP AS 65000, VNI 100
           │         │         │
          h1        h2        h3       ← 192.168.100.0/24

The leaves’ eth2 ports have no address because they are bridge ports. Each leaf’s VTEP address is its eth1 address, and the leaves peer with each other using those addresses, across the spine. With three leaves, each one peers directly with the other two, so no route reflector is needed. The underlay is static IP: each leaf has a default route to the spine, which only forwards IP and doesn’t run BGP.

The nodes are containers deployed with containerlab. You need containerlab and podman. The leaves run the official FRR image. The spine and the hosts run a small image built from the official Alpine Linux image with the iproute2 and tcpdump packages. The lab contains its two-line Containerfile, and you build the image once:

sudo podman build -t localhost/evpn-node:latest image/

Get the files from the lab repository and cd into evpn-basics.

The configuration

Each leaf has two parts: a script that builds the Linux bridge and VXLAN device, and an FRR configuration that runs EVPN. Compare the script with a VXLAN setup that uses a static flood list. There is no list here.

#!/bin/sh
set -e

# Underlay: an address on the spine-facing link, and a route to the other leaves
ip addr add "$VTEP_IP/24" dev eth1
ip route add default via "$GATEWAY"

# Overlay: a bridge that joins the host-facing port and the VXLAN device
ip link add br0 type bridge mcast_snooping 0
ip link add vxlan100 type vxlan id 100 local "$VTEP_IP" dstport 4789 nolearning
ip link set vxlan100 master br0
ip link set eth2 master br0
ip link set br0 up
ip link set vxlan100 up
ip link set eth2 up

The FRR configuration is the same on each leaf except for the router ID and the neighbor addresses. This is leaf1’s:

frr defaults traditional
hostname leaf1
!
router bgp 65000
 bgp router-id 1.1.1.1
 no bgp default ipv4-unicast
 neighbor 10.0.2.2 remote-as 65000
 neighbor 10.0.3.2 remote-as 65000
 !
 address-family l2vpn evpn
  neighbor 10.0.2.2 activate
  neighbor 10.0.3.2 activate
  advertise-all-vni
 exit-address-family
exit

The topology also turns IPv6 off on every node. Otherwise the nodes’ own background IPv6 traffic is learned as MAC addresses, and the leaves advertise routes for them before we have sent anything. With it off, the tables only contain what we generate.

FRR also needs the BGP daemon enabled, which the lab’s daemons file does with bgpd=yes.

Deploy

sudo containerlab --runtime podman deploy -t topology.clab.yml

The BGP sessions take a few seconds to establish. Check them on leaf1:

sudo podman exec clab-evpn-leaf1 vtysh -c "show bgp l2vpn evpn summary"

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.0.2.2        4      65000         4         4        2    0    0 00:00:06            1        1 N/A
10.0.3.2        4      65000         4         5        2    0    0 00:00:06            1        1 N/A

Both sessions are up, and each peer has already sent one prefix and received one. Nobody has sent any traffic yet. The single prefix is each leaf’s type-3 route.

VTEP discovery

Ask leaf1 what it knows about VNI 100:

sudo podman exec clab-evpn-leaf1 vtysh -c "show evpn vni 100"

VNI: 100
 Type: L2
 Vlan: 1
 Bridge: br0
 Tenant VRF: default
 VxLAN interface: vxlan100
 VxLAN ifIndex: 4
 SVI interface: br0
 SVI ifIndex: 3
 Local VTEP IP: 10.0.1.2
 Mcast group: 0.0.0.0
 Remote VTEPs for this VNI:
  10.0.3.2 flood: HER
  10.0.2.2 flood: HER
 Number of MACs (local and remote) known for this VNI: 0
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 0
 Advertise-gw-macip: No
 Advertise-svi-macip: No

leaf1 knows both remote VTEPs, 10.0.2.2 and 10.0.3.2, and for each it records flood: HER, head-end replication: broadcast frames will be copied to each of them. Nobody configured these. They came from the type-3 routes:

sudo podman exec clab-evpn-leaf1 vtysh -c "show bgp l2vpn evpn route type multicast"

   Network          Next Hop            Metric LocPrf Weight Path
                    Extended Community
Route Distinguisher: 1.1.1.1:2
 *>  [3]:[0]:[32]:[10.0.1.2] RD 1.1.1.1:2
                    10.0.1.2                           32768 i
                    ET:8 RT:65000:100
Route Distinguisher: 2.2.2.2:2
 *>i [3]:[0]:[32]:[10.0.2.2] RD 2.2.2.2:2
                    10.0.2.2                      100      0 i
                    RT:65000:100 ET:8
Route Distinguisher: 3.3.3.3:2
 *>i [3]:[0]:[32]:[10.0.3.2] RD 3.3.3.3:2
                    10.0.3.2                      100      0 i
                    RT:65000:100 ET:8

Displayed 3 prefixes (3 paths) (of requested type)

(FRR prints a legend of route-type formats above this table, which I left out.) There are three type-3 routes, one per leaf: leaf1’s own, with weight 32768 because it originated it, and one from each of the others. In [3]:[0]:[32]:[10.0.2.2], the 3 is the route type and 10.0.2.2 is the originating VTEP. Each route has its own RD (the router ID plus a local number), the route target 65000:100 that carries the VNI, and ET:8 for VXLAN.

The forwarding database shows the result in the kernel:

sudo podman exec clab-evpn-leaf1 bridge fdb show dev vxlan100
sudo podman exec clab-evpn-leaf1 ip -d link show vxlan100

b6:65:ff:e2:5e:51 vlan 1 master br0 permanent
b6:65:ff:e2:5e:51 master br0 permanent
00:00:00:00:00:00 dst 10.0.2.2 self permanent
00:00:00:00:00:00 dst 10.0.3.2 self permanent

    vxlan id 100 local 10.0.1.2 srcport 0 0 dstport 4789 ttl auto ageing 300 reserved_bits 0xf7ffffff000000ff nolearning 

The first two lines are the bridge’s own address. The two lines with 00:00:00:00:00:00 are the flood entries, one per remote VTEP. They are the same entries you would add by hand with a static list, installed by FRR instead. The last line confirms the VXLAN device is in nolearning mode. There are no MAC addresses for hosts yet, because no host has sent anything.

The first ping

Ping from h1 to h2, which are in the same subnet but separated by the routed network. Capture on h3 in a second terminal first, to see what reaches the host with nothing to do with this conversation:

sudo podman exec clab-evpn-h3 tcpdump -nn -i eth1
sudo podman exec clab-evpn-h1 ping -c 2 192.168.100.2

PING 192.168.100.2 (192.168.100.2): 56 data bytes
64 bytes from 192.168.100.2: seq=0 ttl=64 time=0.235 ms
64 bytes from 192.168.100.2: seq=1 ttl=64 time=0.124 ms

--- 192.168.100.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.124/0.179/0.235 ms

21:03:09.883029 ARP, Request who-has 192.168.100.2 tell 192.168.100.1, length 28

1 packet captured
1 packet received by filter

The ping works. h3 received exactly one packet, h1’s ARP request. That is a broadcast, so it went to every VTEP on the flood list that the type-3 routes built, and leaf3 delivered it to h3. Nothing else from the conversation reached h3.

MAC advertisement

While that conversation was happening, each leaf learned its own host’s MAC address on a local port and advertised it. Look at the type-2 routes on leaf3, which has no connection to either host:

sudo podman exec clab-evpn-leaf3 vtysh -c "show bgp l2vpn evpn route type macip"

   Network          Next Hop            Metric LocPrf Weight Path
                    Extended Community
Route Distinguisher: 1.1.1.1:2
 *>i [2]:[0]:[48]:[aa:c1:ab:48:bc:35] RD 1.1.1.1:2
                    10.0.1.2                      100      0 i
                    RT:65000:100 ET:8
Route Distinguisher: 2.2.2.2:2
 *>i [2]:[0]:[48]:[aa:c1:ab:1e:7b:79] RD 2.2.2.2:2
                    10.0.2.2                      100      0 i
                    RT:65000:100 ET:8

There are two type-2 routes: aa:c1:ab:48:bc:35 is h1’s MAC with next hop 10.0.1.2 (leaf1), and aa:c1:ab:1e:7b:79 is h2’s with next hop 10.0.2.2 (leaf2). The [48] is the length of the MAC address in bits. FRR keeps a per-segment view of the same information:

sudo podman exec clab-evpn-leaf1 vtysh -c "show evpn mac vni 100"
sudo podman exec clab-evpn-leaf3 vtysh -c "show evpn mac vni 100"

MAC               Type   Flags Intf/Remote ES/VTEP                     VLAN  Seq #'s
aa:c1:ab:48:bc:35 local        eth2                                          0/0
aa:c1:ab:1e:7b:79 remote       10.0.2.2                                      0/0

MAC               Type   Flags Intf/Remote ES/VTEP                     VLAN  Seq #'s
aa:c1:ab:48:bc:35 remote       10.0.1.2                                      0/0
aa:c1:ab:1e:7b:79 remote       10.0.2.2                                      0/0

On leaf1, h1’s MAC is local, learned on eth2, and h2’s is remote, behind 10.0.2.2. On leaf3 both are remote, because neither host is attached to it. Last, the entries in leaf3’s kernel forwarding database:

sudo podman exec clab-evpn-leaf3 bridge fdb show dev vxlan100

aa:c1:ab:48:bc:35 vlan 1 extern_learn master br0 
aa:c1:ab:1e:7b:79 vlan 1 extern_learn master br0 
aa:c1:ab:1e:7b:79 extern_learn master br0 
aa:c1:ab:48:bc:35 extern_learn master br0 
ae:38:b3:99:2e:ee vlan 1 master br0 permanent
ae:38:b3:99:2e:ee master br0 permanent
aa:c1:ab:48:bc:35 dst 10.0.1.2 self extern_learn 
aa:c1:ab:1e:7b:79 dst 10.0.2.2 self extern_learn 
00:00:00:00:00:00 dst 10.0.1.2 self permanent
00:00:00:00:00:00 dst 10.0.2.2 self permanent

Both host MACs are now there, each pointing at the right VTEP with dst. The extern_learn flag means these entries were installed by something outside the kernel’s own learning, here FRR acting on the BGP routes. The VXLAN device is in nolearning mode, so the kernel does not add MAC addresses from tunnel frames. These entries exist because of the type-2 routes.

The BGP summary reflects it. leaf1 now has received two prefixes from leaf2, its type-3 route and h2’s type-2 route, and one from leaf3, whose host hasn’t sent anything:

sudo podman exec clab-evpn-leaf1 vtysh -c "show bgp l2vpn evpn summary"

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
10.0.2.2        4      65000         5         5        3    0    0 00:00:13            2        2 N/A
10.0.3.2        4      65000         4         6        3    0    0 00:00:13            1        2 N/A

The second ping

Now that every leaf knows where both MACs are, ping again. Capture the tunnel traffic going toward leaf3 on the spine, and everything arriving at h3, while h1 pings h2:

sudo podman exec clab-evpn-spine tcpdump -nn -i eth3 udp port 4789
sudo podman exec clab-evpn-h3 tcpdump -nn -i eth1
sudo podman exec clab-evpn-h1 ping -c 2 192.168.100.2

PING 192.168.100.2 (192.168.100.2): 56 data bytes
64 bytes from 192.168.100.2: seq=0 ttl=64 time=0.130 ms
64 bytes from 192.168.100.2: seq=1 ttl=64 time=0.088 ms

--- 192.168.100.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.088/0.109/0.130 ms

0 packets captured
0 packets received by filter
0 packets dropped by kernel

Both captures show zero packets. The pings were delivered, and leaf3 never saw them: leaf1 looked up h2’s MAC, found it behind 10.0.2.2, and sent the frames only there. h1’s ARP entry was still in its cache, so no broadcast went out either. If you wait long enough for that entry to expire, h1 will broadcast an ARP request again, and h3 will see it, as it did the first time.

Tear it down

sudo containerlab --runtime podman destroy -t topology.clab.yml --cleanup