A provider edge (PE) router is where a network attaches to a provider’s VPN service. It is normally a dedicated router or switch, and the machines that need the service plug into it. But when the machines are Linux hosts that run virtual machines, containers or cluster workloads, there is another option: let the host attach to the VPN itself. The host then needs to do what a PE does, which is keep its routing in a VRF, speak BGP, encapsulate traffic into the provider’s tunnels, and, if the workloads need it, stretch a layer 2 segment to other hosts.
Linux can do all of that. This post builds it: two Linux hosts that each behave as a PE router, joined by a plain IPv6 core. The hosts share a layer 2 segment, carried by EVPN and VXLAN, and the VXLAN tunnel itself travels through an SRv6 L3VPN. It combines three techniques, each of which is simple on its own, and the point is how they stack. At the end, one ARP request is captured at three places, and you can see it wrapped in one layer after another.
The pieces
Network namespaces. A namespace is an independent copy of the network stack, with its own interfaces, routes and firewall. Each PE splits its roles into namespaces: the host keeps the default one, and two more, pe-router and l2-vxlan, each run their own routing daemon. Veth pairs, which are virtual cables, connect them.
L3VPN over SRv6. In pe-router, a VRF keeps the VPN’s routes in their own routing table. BGP exchanges them between the PEs, together with an SRv6 SID per VRF. To send traffic to a remote PE, the local PE wraps the packet in an IPv6 packet addressed to that SID, and the remote PE removes the wrapper and looks the packet up in the VRF. The core router in the middle only forwards IPv6.
EVPN and VXLAN. In l2-vxlan, a Linux bridge joins the host’s port and a VXLAN device, which wraps Ethernet frames in UDP packets between VXLAN endpoints (VTEPs). EVPN, running in BGP, tells each VTEP where the other VTEPs are and which MAC addresses sit behind them.
How they stack. A VXLAN tunnel needs IP connectivity between its VTEPs. Here, the VTEP addresses live in the VRF, and the VPN provides that connectivity. So a frame from the host goes into VXLAN, the VXLAN packet goes into the VRF, and the VRF wraps it in SRv6.
host frame → VXLAN between VTEPs → SRv6 between PEs
(ARP, ICMP) UDP 4789, VNI 330 outer IPv6 + SRH to the remote PE's VRF SID
The lab
Two PE routers, each an AlmaLinux VM, connected through a plain IPv6 router (P):
PE1 (VM) ────────── P ────────── PE2 (VM)
host 10.55.99.1 host 10.55.99.2
| Node | Interface | Address | Connected to |
|---|---|---|---|
| pe1 | eth1 | 2001:db8:1::1/64 | p |
| p | eth1 | 2001:db8:1::2/64 | pe1 |
| p | eth2 | 2001:db8:2::1/64 | pe2 |
| pe2 | eth1 | 2001:db8:2::2/64 | p |
Inside each PE, with N standing for 1 or 2:
| Namespace | Interface | Address | Role |
|---|---|---|---|
| default | ve-host | 10.55.99.N/24 | The host, on the stretched segment |
| l2-vxlan | ve-l2-host | — | Bridge port toward the host |
| l2-vxlan | br0, vxlan330 | — | Bridge and VXLAN device (VNI 330) |
| l2-vxlan | ve-l2-pe | 10.100.N.2/30 | The VTEP address |
| pe-router | ve-pe-l2 | 10.100.N.1/30 | In VRF CMN, the VTEP’s path into the VPN |
| pe-router | eth1 | the fabric address above | Fabric interface |
| pe-router | lo | fc00:0:N::1/128 | Loopback, inside the SRv6 locator fc00:0:N::/48 |
Each PE has one VRF, CMN. The VTEP subnet 10.100.N.0/30 sits in it and is advertised to the other PE through the L3VPN. That is how the two VTEPs reach each other. The VM runs two FRR instances, one per namespace, as podman containers that share the namespace’s network: one runs the L3VPN, the other runs EVPN.
This lab runs on virtual machines and not on containers, because a PE router here is a whole Linux host with several namespaces, its own kernel, and containers of its own. The VMs are managed by containerlab. The P router is a regular container.
Requirements
You need a Linux host with KVM, containerlab and podman. The VMs are AlmaLinux 10, and the lab builds the image for containerlab itself. The vm/ directory holds a few small files: the launcher that boots the VM in QEMU and connects its network interfaces to containerlab, a Containerfile, a script that downloads AlmaLinux’s official cloud image, and a build script. The build fetches the shared part of the launcher, vrnetlab’s common code, from the upstream vrnetlab project, and builds with podman:
sh vm/build.sh
The result is the image localhost/vrnetlab/almalinux_almalinux:10. The script calls sudo podman build, so the image lands in the storage that containerlab uses.
By default the launcher gives the VM 512 MB of RAM and one vCPU. That is too little: installing podman had not finished after ten minutes at that size, and took 43 seconds with 2 GB. vrnetlab reads QEMU_MEMORY and QEMU_SMP from the node’s environment, so the topology file sets them for each PE:
pe1:
kind: generic_vm
image: localhost/vrnetlab/almalinux_almalinux:10
env:
QEMU_MEMORY: 2048
QEMU_SMP: 2
The P router uses a small image built from the official Alpine Linux image with iproute2 and tcpdump. The lab contains its two-line Containerfile:
sudo podman build -t localhost/pe-router-node:latest image/
The PEs install podman and pull the official FRR image themselves, so the VMs need internet access. Get the files from the lab repository and cd into linux-pe-router.
Deploy and provision
sudo containerlab --runtime podman deploy -t topology.clab.yml
The VMs take about two minutes to boot. Each PE is then configured by one script, pe.sh, which takes the PE’s number. The VM user is clab and the password is clab@123:
scp -r configs clab@clab-pe-router-pe1:~/
ssh clab@clab-pe-router-pe1 'cd configs && sudo sh pe.sh 1'
scp -r configs clab@clab-pe-router-pe2:~/
ssh clab@clab-pe-router-pe2 'cd configs && sudo sh pe.sh 2'
Each run takes under a minute. This is the script:
#!/bin/sh
set -e
N=$1
FRR=quay.io/frrouting/frr:10.7.1
DIR=$(cd "$(dirname "$0")" && pwd)
if [ "$N" = 1 ]; then UPLINK=2001:db8:1::1; else UPLINK=2001:db8:2::2; fi
# The strict VRF mode sysctl below only exists once the vrf module is loaded
modprobe vrf
dnf install -y podman tcpdump
podman pull "$FRR"
# Namespaces: pe-router for the SRv6 L3VPN, l2-vxlan for the bridge, VXLAN and EVPN
ip netns add pe-router
ip netns add l2-vxlan
ip -n l2-vxlan link set lo up
# pe-router: the fabric NIC, SRv6, and the CMN VRF
ip link set eth1 netns pe-router
ip netns exec pe-router sh -c "
sysctl -qw net.ipv6.conf.all.forwarding=1
sysctl -qw net.ipv4.ip_forward=1
sysctl -qw net.ipv6.conf.all.seg6_enabled=1
sysctl -qw net.ipv6.conf.eth1.seg6_enabled=1
sysctl -qw net.vrf.strict_mode=1
sysctl -qw net.ipv4.conf.default.rp_filter=0
ip link set lo up
ip link set eth1 up
ip addr add fc00:0:$N::1/128 dev lo
ip addr add $UPLINK/64 dev eth1
ip link add CMN type vrf table 4
ip link set CMN up
"
# veth between pe-router (inside the CMN VRF) and l2-vxlan: the VTEP's path into the VPN
ip link add ve-pe-l2 type veth peer name ve-l2-pe
ip link set ve-pe-l2 netns pe-router
ip link set ve-l2-pe netns l2-vxlan
ip -n pe-router link set ve-pe-l2 master CMN
ip -n pe-router addr add 10.100.$N.1/30 dev ve-pe-l2
ip -n pe-router link set ve-pe-l2 up
ip -n l2-vxlan addr add 10.100.$N.2/30 dev ve-l2-pe
ip -n l2-vxlan link set ve-l2-pe up
ip -n l2-vxlan route add default via 10.100.$N.1
# l2-vxlan: a bridge with the VXLAN device and a port toward the host
ip -n l2-vxlan link add br0 type bridge mcast_snooping 0
ip -n l2-vxlan link add vxlan330 type vxlan id 330 local 10.100.$N.2 dstport 4789 nolearning
ip -n l2-vxlan link set vxlan330 master br0
ip link add ve-host type veth peer name ve-l2-host
ip link set ve-l2-host netns l2-vxlan
ip -n l2-vxlan link set ve-l2-host master br0
# Only the SRv6 side needs IPv6. Turning it off on the bridge side and on the host keeps their
# background traffic out of the tunnel, so the frames we look at later are the ones we send.
ip netns exec l2-vxlan sysctl -qw net.ipv6.conf.all.disable_ipv6=1
ip netns exec l2-vxlan sysctl -qw net.ipv6.conf.default.disable_ipv6=1
sysctl -qw net.ipv6.conf.ve-host.disable_ipv6=1
ip -n l2-vxlan link set br0 up
ip -n l2-vxlan link set vxlan330 up
ip -n l2-vxlan link set ve-l2-host up
# The host: the default namespace gets an address on the stretched segment
ip addr add 10.55.99.$N/24 dev ve-host
ip link set ve-host up
# One FRR instance per namespace
mkdir -p /etc/frr/pe-router /etc/frr/l2-vxlan
cp "$DIR/daemons" "$DIR/vtysh.conf" /etc/frr/pe-router/
cp "$DIR/daemons" "$DIR/vtysh.conf" /etc/frr/l2-vxlan/
cp "$DIR/pe$N/router.conf" /etc/frr/pe-router/frr.conf
cp "$DIR/pe$N/l2.conf" /etc/frr/l2-vxlan/frr.conf
podman run -d --name frr-pe-router --privileged --security-opt label=disable \
--network ns:/var/run/netns/pe-router -v /etc/frr/pe-router:/etc/frr "$FRR"
podman run -d --name frr-l2-vxlan --privileged --security-opt label=disable \
--network ns:/var/run/netns/l2-vxlan -v /etc/frr/l2-vxlan:/etc/frr "$FRR"
In order, it does these things:
| Step | What it does |
|---|---|
modprobe vrf | Loads the VRF module first. The net.vrf.strict_mode sysctl does not exist until it is loaded, and SRv6 behaviors that look up a VRF table need it. |
| Namespaces | Creates pe-router and l2-vxlan. The VM’s default namespace stays as the host. |
ip link set eth1 netns pe-router | Moves the fabric NIC into pe-router. Inside, SRv6 is enabled, strict VRF mode is on, and reverse-path filtering is off by default so that the VRF created next doesn’t drop the traffic arriving from the tunnel. |
VRF CMN and the veth ve-pe-l2 / ve-l2-pe | The VRF holds the VPN’s routes. One end of the veth is placed in the VRF in pe-router, and the other end in l2-vxlan has the VTEP address 10.100.N.2, with a default route through the VRF. |
Bridge, vxlan330, ve-host | In l2-vxlan, a bridge joins the VXLAN device (VNI 330, nolearning, sourced from the VTEP address) and a port toward the host. The host end stays in the default namespace with 10.55.99.N/24. |
| IPv6 off | Turns IPv6 off on the bridge side and the host. Otherwise their background IPv6 traffic would enter the tunnel and be learned as MAC addresses before we send anything. |
| Two FRR containers | Starts one FRR instance per namespace as a podman container sharing that namespace’s network (--network ns:…). Each reads its own config directory. |
The two FRR configurations differ per PE only by numbers. This is PE1’s pe-router configuration, the L3VPN:
frr defaults traditional
hostname pe1-router
!
vrf CMN
exit-vrf
!
ipv6 route fc00:0:2::/48 2001:db8:1::2
!
router bgp 65000
bgp router-id 1.1.1.1
no bgp default ipv4-unicast
neighbor fc00:0:2::1 remote-as 65000
neighbor fc00:0:2::1 update-source lo
neighbor fc00:0:2::1 capability extended-nexthop
!
segment-routing srv6
locator MAIN
exit
!
address-family ipv4 vpn
neighbor fc00:0:2::1 activate
exit-address-family
exit
!
router bgp 65000 vrf CMN
sid vpn per-vrf export auto
!
address-family ipv4 unicast
redistribute connected
rd vpn export 65000:4
rt vpn both 65000:4
export vpn
import vpn
exit-address-family
exit
!
segment-routing
srv6
encapsulation
source-address fc00:0:1::1
locators
locator MAIN
prefix fc00:0:1::/48 block-len 32 node-len 16
behavior usid
exit
exit
exit
exit
This one has an iBGP session to the other PE’s loopback, a VPN address family, and a VRF with its route distinguisher and target (65000:4) that redistributes its connected subnet, the VTEP’s /30, as a VPN route. Each PE also gets an SRv6 locator and one SID for the VRF (sid vpn per-vrf export auto). The route to the other PE’s locator is a static route through the P router. And this is the l2-vxlan configuration, the EVPN side:
frr defaults traditional
hostname pe1-l2
!
router bgp 65001
bgp router-id 10.100.1.2
no bgp default ipv4-unicast
neighbor 10.100.2.2 remote-as 65001
!
address-family l2vpn evpn
neighbor 10.100.2.2 activate
advertise-all-vni
exit-address-family
exit
A single iBGP session to the other VTEP’s address, with the EVPN address family on and advertise-all-vni so FRR picks up the VXLAN device. That session’s TCP connection runs across the L3VPN.
The VPN layer
Run the commands in this section on pe1 (ssh clab@clab-pe-router-pe1). First, check that the PE has the structure the script built:
ip netns list
sudo ip -n pe-router -br addr
sudo ip -n l2-vxlan -br addr
ip -br addr show ve-host
sudo podman ps --format '{{.Names}} {{.Image}}'
pe-router (id: 0)
l2-vxlan (id: 1)
--- pe-router
lo UNKNOWN 127.0.0.1/8 fc00:0:1::1/128
CMN UP
eth1 UP 2001:db8:1::1/64 fe80::a8c1:abff:fe73:7341/64
ve-pe-l2@if4 UP 10.100.1.1/30 fe80::e8be:74ff:fe43:df60/64
--- l2-vxlan
lo UNKNOWN 127.0.0.1/8
br0 UP
vxlan330 UNKNOWN
ve-l2-pe@if5 UP 10.100.1.2/30
ve-l2-host@if7 UP
--- default
ve-host@if6 UP 10.55.99.1/24
--- frr containers
frr-pe-router quay.io/frrouting/frr:10.7.1
frr-l2-vxlan quay.io/frrouting/frr:10.7.1
(The --- lines mark which command each block of output came from.) Each namespace holds only its own interfaces: the fabric NIC, the loopback and the VRF in pe-router, the bridge, VXLAN device and VTEP address in l2-vxlan, and the host’s address in the default namespace. Now the L3VPN:
sudo podman exec frr-pe-router vtysh -c "show bgp ipv4 vpn summary"
sudo podman exec frr-pe-router vtysh -c "show ip route vrf CMN"
sudo ip -n pe-router route show vrf CMN
sudo ip -n pe-router -6 route show | grep seg6
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
fc00:0:2::1 4 65000 5 7 3 0 0 00:00:44 1 1 N/A
C>* 10.100.1.0/30 is directly connected, ve-pe-l2, weight 1, 00:00:47
B> 10.100.2.0/30 [200/0] via fc00:0:2::1 (vrf default) (recursive), label 16, seg6 fc00:0:2:1::, weight 1, 00:00:43
10.100.1.0/30 dev ve-pe-l2 proto kernel scope link src 10.100.1.1
10.100.2.0/30 nhid 19 encap seg6 mode encap segs 1 [ fc00:0:2:1:: ] via inet6 2001:db8:1::2 dev eth1 proto bgp metric 20
fc00:0:1:1:: nhid 11 encap seg6local action End.DT46 vrftable 4 dev CMN proto bgp metric 20 pref medium
(FRR also prints a legend of route codes before its routes, which I left out.) pe1 has one VPN neighbor, pe2’s loopback. In VRF CMN it has its own VTEP subnet, 10.100.1.0/30, and a BGP route to pe2’s, 10.100.2.0/30, whose SRv6 SID is fc00:0:2:1::. The kernel holds that as an encapsulating route out of eth1. The last line is pe1’s own SID, an End.DT46 that decapsulates packets and looks them up in the VRF, table 4. This is what lets the two VTEPs reach each other through the VPN.
The EVPN layer
sudo podman exec frr-l2-vxlan vtysh -c "show bgp l2vpn evpn summary"
sudo podman exec frr-l2-vxlan vtysh -c "show evpn vni 330"
sudo ip netns exec l2-vxlan bridge fdb show dev vxlan330
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.100.2.2 4 65001 4 4 1 0 0 00:00:17 1 1 N/A
Local VTEP IP: 10.100.1.2
Mcast group: 0.0.0.0
Remote VTEPs for this VNI:
10.100.2.2 flood: HER
Number of MACs (local and remote) known for this VNI: 0
Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 0
6e:49:19:81:d5:97 vlan 1 master br0 permanent
6e:49:19:81:d5:97 master br0 permanent
00:00:00:00:00:00 dst 10.100.2.2 self permanent
The EVPN session to 10.100.2.2 is up. Its TCP connection goes from one VTEP to the other through VRF CMN and across the SRv6 tunnel, because VRF CMN is the only way between them. The session carried one route, pe2’s type-3 route, and from it pe1 knows the remote VTEP 10.100.2.2 and added it to the flood list (flood: HER, head-end replication). No host has sent anything, so no MAC address is known yet.
One ARP request, three layers
Now ping from pe1’s host to pe2’s host. They are on the same subnet, 10.55.99.0/24, but they sit behind different PEs and a routed core. First start three captures: on pe1 at the host side of the bridge, and inside the VRF on the VTEP’s path; and on the P router, from the machine running containerlab:
sudo ip netns exec l2-vxlan tcpdump -nn -i ve-l2-host 'arp or icmp'
sudo ip netns exec pe-router tcpdump -nn -v -i ve-pe-l2 'udp port 4789'
sudo podman exec clab-pe-router-p tcpdump -nn -v -i eth1 'ip6 proto 43'
Then on pe1:
ping -c 2 10.55.99.2
PING 10.55.99.2 (10.55.99.2) 56(84) bytes of data.
64 bytes from 10.55.99.2: icmp_seq=1 ttl=64 time=1.47 ms
64 bytes from 10.55.99.2: icmp_seq=2 ttl=64 time=0.901 ms
--- 10.55.99.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 0.901/1.183/1.465/0.282 ms
The ping works, and the TTL is still 64, because for the hosts this is one layer 2 segment with no router in it. The captures show the first packet at each stage. I removed their timestamps and wrapped the long lines. At the host side of the bridge, the original frames:
ARP, Request who-has 10.55.99.2 tell 10.55.99.1, length 28
ARP, Reply 10.55.99.2 is-at ba:6a:2d:97:8b:5b, length 28
Inside the VRF, on the way to the core, the same ARP request, now inside a VXLAN packet between the VTEPs, with its reply coming back the same way:
IP (tos 0x0, ttl 64, id 8218, offset 0, flags [none], proto UDP (17), length 78)
10.100.1.2.57823 > 10.100.2.2.4789: VXLAN, flags [I] (0x08), vni 330
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.55.99.2 tell 10.55.99.1, length 28
IP (tos 0x0, ttl 63, id 55851, offset 0, flags [none], proto UDP (17), length 78)
10.100.2.2.39691 > 10.100.1.2.4789: VXLAN, flags [I] (0x08), vni 330
ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.55.99.2 is-at ba:6a:2d:97:8b:5b, length 28
On the P router, the same VXLAN packet, with the ARP request, wrapped in an outer IPv6 packet. Below it, the first ICMP echo request travels the same way:
IP6 (hlim 63, next-header Routing (43), payload length 102)
fc00:0:1::1 > fc00:0:2:1::: RT6 (len=2, type=4, segleft=0, last-entry=0, flags=0x0, tag=0,
[0]fc00:0:2:1::)
IP (tos 0x0, ttl 64, id 8218, offset 0, flags [none], proto UDP (17), length 78)
10.100.1.2.57823 > 10.100.2.2.4789: VXLAN, flags [I] (0x08), vni 330
ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.55.99.2 tell 10.55.99.1, length 28
IP6 (hlim 63, next-header Routing (43), payload length 158)
fc00:0:1::1 > fc00:0:2:1::: RT6 (len=2, type=4, segleft=0, last-entry=0, flags=0x0, tag=0,
[0]fc00:0:2:1::)
IP (tos 0x0, ttl 64, id 8219, offset 0, flags [none], proto UDP (17), length 134)
10.100.1.2.48332 > 10.100.2.2.4789: VXLAN, flags [I] (0x08), vni 330
IP (tos 0x0, ttl 64, id 60284, offset 0, flags [DF], proto ICMP (1), length 84)
10.55.99.1 > 10.55.99.2: ICMP echo request, id 1, seq 1, length 64
Reading from the outside in, the P router sees:
- An IPv6 packet from PE1’s
fc00:0:1::1tofc00:0:2:1::. That is pe2’s SID for VRF CMN, and with its segment routing header it is the SRv6 layer. The P router only needs the route tofc00:0:2::/48to forward it. - Inside it, a UDP packet from
10.100.1.2to10.100.2.2on port 4789 withvni 330. That is the VXLAN layer, between the two VTEPs. It travels through VRF CMN, which is what got it into the SRv6 tunnel. - Inside that, the host’s own frame: the ARP request, and then the ICMP echo.
The sizes show what each layer costs. The ICMP echo is an 84-byte IP packet. VXLAN makes it 134 bytes (length 134). SRv6 wraps that in an IPv6 header and an SRH with one segment, which add another 64 bytes, so 198 bytes cross the P router’s link for one 84-byte ping.
MAC advertisement
When the hosts sent their first frames, each PE learned its own host’s MAC on the bridge and advertised it. On pe2:
sudo podman exec frr-l2-vxlan vtysh -c "show bgp l2vpn evpn route type macip"
sudo podman exec frr-l2-vxlan vtysh -c "show evpn mac vni 330"
sudo ip netns exec l2-vxlan bridge fdb show dev vxlan330
Network Next Hop Metric LocPrf Weight Path
Extended Community
Route Distinguisher: 10.100.1.2:2
*>i [2]:[0]:[48]:[2a:11:42:9a:ce:1e] RD 10.100.1.2:2
10.100.1.2 100 0 i
RT:65001:330 ET:8
Route Distinguisher: 10.100.2.2:2
*> [2]:[0]:[48]:[ba:6a:2d:97:8b:5b] RD 10.100.2.2:2
10.100.2.2 32768 i
ET:8 RT:65001:330
Displayed 2 prefixes (2 paths) (of requested type)
MAC Type Flags Intf/Remote ES/VTEP VLAN Seq #'s
2a:11:42:9a:ce:1e remote 10.100.1.2 0/0
ba:6a:2d:97:8b:5b local ve-l2-host 0/0
2a:11:42:9a:ce:1e vlan 1 extern_learn master br0
2a:11:42:9a:ce:1e extern_learn master br0
16:14:08:72:79:2c vlan 1 master br0 permanent
16:14:08:72:79:2c master br0 permanent
00:00:00:00:00:00 dst 10.100.1.2 self permanent
2a:11:42:9a:ce:1e dst 10.100.1.2 self extern_learn
There are two type-2 routes. ba:6a:2d:97:8b:5b is pe2’s own host, local on ve-l2-host, and 2a:11:42:9a:ce:1e is pe1’s host, remote, behind 10.100.1.2. FRR installed the remote MAC in the forwarding database with extern_learn, pointing at the other VTEP.
That is the complete path. A frame from one Linux host crosses a bridge, a VXLAN tunnel and an SRv6 L3VPN, and arrives at another host on the same subnet, through three namespaces and a router that only forwards IPv6 and knows nothing about any of it.
Tear it down
sudo containerlab --runtime podman destroy -t topology.clab.yml --cleanup