L3VPN over SRv6: separate customer networks on one backbone

A service provider runs one backbone and sells private networks over it. Each customer expects their sites to be connected as if they had their own network, with their own addressing, invisible to everyone else. Two customers will happily pick the same subnet, and the backbone has to carry both without mixing them up. This is the layer 3 VPN (L3VPN) service, and it is one of the most widely deployed services in provider networks.

The established way to build it is with MPLS labels. With SRv6, the same service runs over a plain IPv6 core: the instruction that says “deliver this packet into customer red’s network” is an IPv6 address, carried by BGP together with the customer’s routes. The core routers in the middle only forward IPv6.

This post explains how an L3VPN works, with SRv6 as the transport, and builds a small provider network with FRR to show it: two customers with overlapping address space, a BGP session between two edge routers, and the tunnel that carries the traffic.

How an L3VPN over SRv6 works

The roles. Customer sites connect to the provider through a customer edge router (CE, here just a host). The provider edge router (PE) is the provider’s router that the CE connects to. A provider (P) router sits in the core and only forwards traffic between PEs.

VRFs. A PE keeps each customer’s routes in a separate VRF (virtual routing and forwarding instance): its own routing table, with the customer-facing interfaces attached to it. Two VRFs can contain the same prefix without conflict, which is how overlapping customer addresses are kept apart.

Route distinguishers and route targets. Two BGP values keep the customers separate when routes are exchanged between PEs. A route distinguisher (RD) is prepended to each prefix, so the same prefix from two customers becomes two different VPN routes: 65000:10:192.168.1.0/24 and 65000:20:192.168.1.0/24. A route target (RT) is an extended community that says which VRFs should import a route. A PE imports a VPN route into a VRF only if the route carries a target that VRF imports.

BGP carries the VPN routes. The PEs peer with each other using a BGP address family for VPN routes, ipv4 vpn here. Each PE advertises its customers’ prefixes with their RDs and RTs. The P routers don’t take part in BGP, so they hold no customer routes at all.

The service SID. For forwarding, the receiving PE needs to know where and how to hand the packet to the right customer. In an SRv6 L3VPN, each PE allocates a SID for each VRF, from its locator, and advertises it with the VPN routes. This is a service SID with a decapsulation behavior, End.DT46 in this lab: remove the outer IPv6 header and look the inner packet up in the VRF’s routing table. When a PE sends a customer packet toward a remote prefix, it wraps it in an outer IPv6 packet addressed to the remote PE’s SID for that VRF.

The path of one packet:

1. A host in site A sends a packet to a host in site B, through its CE to PE1.
2. PE1 looks up the destination in the customer’s VRF and finds a BGP route whose next hop is PE2 and whose SID is PE2’s SID for that VRF.
3. PE1 encapsulates the packet in an IPv6 packet whose destination is that SID, with its own address as the source.
4. The P router routes the outer packet by its destination, which is inside PE2’s locator. It knows nothing about VPNs.
5. PE2 receives a packet addressed to one of its own SIDs, removes the outer header, and looks up the inner destination in the VRF that the SID belongs to.
6. PE2 delivers the packet to the customer’s site B.

The lab

Two PEs run FRR and are connected through a plain IPv6 router. Two customers, red and blue, each have a site behind each PE, and they use the same two subnets.

 red-a  ┐                                  ┌  red-b
        ├── PE1 ────────── P ────────── PE2 ┤
 blue-a ┘                                  └  blue-b

Red and blue use identical subnets on purpose. The PEs run BGP in AS 65000 and peer with each other over IPv6, using their loopback addresses. Each PE owns an SRv6 locator, fc00:0:1::/48 for PE1 and fc00:0:2::/48 for PE2. The underlay is two static IPv6 routes, one on each PE, and the P router has a route to each locator. It has no SRv6 and no BGP.

The nodes are containers deployed with containerlab. You need containerlab and podman. The PEs run the official FRR image. The P router and the customer hosts run a small image built from the official Alpine Linux image with the iproute2 and tcpdump packages. The lab contains its two-line Containerfile, and you build the image once:

sudo podman build -t localhost/l3vpn-node:latest image/

Get the files from the lab repository and cd into srv6-l3vpn-basics.

The configuration

Each PE has a script for the Linux side and an FRR configuration for the routing. This is the script, with the PE’s addresses coming from variables in the topology file:

#!/bin/sh
set -e
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.seg6_enabled=1
sysctl -w net.ipv6.conf.eth1.seg6_enabled=1

ip addr add "$LOOPBACK/128" dev lo
ip addr add "$UPLINK/64" dev eth1

# The kernel requires strict VRF mode for SRv6 behaviors that look up a VRF table
sysctl -w net.vrf.strict_mode=1

# Reverse-path filtering on the VRF devices drops the packets that arrive through the
# SRv6 tunnel, so new interfaces, including the VRFs created below, start with it off
sysctl -w net.ipv4.conf.default.rp_filter=0

# One VRF per customer, with the customer-facing interface attached
ip link add red type vrf table 10
ip link add blue type vrf table 20
ip link set red up
ip link set blue up
ip link set eth2 master red
ip link set eth3 master blue
ip addr add "$RED/24" dev eth2
ip addr add "$BLUE/24" dev eth3

And PE1’s FRR configuration:

frr defaults traditional
hostname pe1
!
vrf red
exit-vrf
!
vrf blue
exit-vrf
!
ipv6 route fc00:0:2::/48 2001:db8:1::2
!
router bgp 65000
 bgp router-id 1.1.1.1
 no bgp default ipv4-unicast
 neighbor fc00:0:2::1 remote-as 65000
 neighbor fc00:0:2::1 update-source lo
 neighbor fc00:0:2::1 capability extended-nexthop
 !
 segment-routing srv6
  locator MAIN
 exit
 !
 address-family ipv4 vpn
  neighbor fc00:0:2::1 activate
 exit-address-family
exit
!
router bgp 65000 vrf red
 sid vpn per-vrf export auto
 !
 address-family ipv4 unicast
  redistribute connected
  rd vpn export 65000:10
  rt vpn both 65000:10
  export vpn
  import vpn
 exit-address-family
exit
!
router bgp 65000 vrf blue
 sid vpn per-vrf export auto
 !
 address-family ipv4 unicast
  redistribute connected
  rd vpn export 65000:20
  rt vpn both 65000:20
  export vpn
  import vpn
 exit-address-family
exit
!
segment-routing
 srv6
  encapsulation
   source-address fc00:0:1::1
  locators
   locator MAIN
    prefix fc00:0:1::/48 block-len 32 node-len 16
    behavior usid
   exit
  exit
 exit
exit

PE2’s FRR configuration is the same, with its own locator, router ID, source address and underlay route. Its interface addresses come from the variables in the topology file. The P router only has addresses, forwarding and two static routes.

Deploy and look at BGP

sudo containerlab --runtime podman deploy -t topology.clab.yml

The BGP session takes a few seconds to come up. Check it from PE1:

sudo podman exec clab-l3vpn-pe1 vtysh -c "show bgp ipv4 vpn summary"

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
fc00:0:2::1     4      65000         5         7        2    0    0 00:00:04            2        2 N/A

One iBGP neighbor, PE2’s loopback, over IPv6. PE1 received two prefixes and sent two, one per customer. Now the VPN routes themselves:

sudo podman exec clab-l3vpn-pe1 vtysh -c "show bgp ipv4 vpn"

     Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65000:10
 *>  192.168.1.0/24   0.0.0.0@3<               0         32768 ?
    UN=0.0.0.0 EC{65000:10} label=16 sid=fc00:0:1:: sid_structure=[32,16,16,0] type=bgp, subtype=5
 *>i 192.168.2.0/24   fc00:0:2::1              0    100      0 ?
    UN=fc00:0:2::1 EC{65000:10} label=16 sid=fc00:0:2:: sid_structure=[32,16,16,0] type=bgp, subtype=0
Route Distinguisher: 65000:20
 *>  192.168.1.0/24   0.0.0.0@4<               0         32768 ?
    UN=0.0.0.0 EC{65000:20} label=32 sid=fc00:0:1:: sid_structure=[32,16,16,0] type=bgp, subtype=5
 *>i 192.168.2.0/24   fc00:0:2::1              0    100      0 ?
    UN=fc00:0:2::1 EC{65000:20} label=32 sid=fc00:0:2:: sid_structure=[32,16,16,0] type=bgp, subtype=0

(The table is FRR’s output, trimmed of its legend.) The routes are grouped by route distinguisher: the two customers have the same prefixes, 192.168.1.0/24 and 192.168.2.0/24, under 65000:10 and 65000:20, so they are four separate VPN routes. The local ones, with next hop 0.0.0.0, come from PE1’s own VRFs. The *>i ones come from PE2 over iBGP, with PE2’s loopback as the next hop. The EC{65000:10} is the route target, and sid is PE2’s locator, the SRv6 information carried with the route. The label field differs by VRF, 16 for red and 32 for blue.

From the BGP route to a forwarding entry

The imported routes arrive in each customer’s VRF:

sudo podman exec clab-l3vpn-pe1 vtysh -c "show ip route vrf red"
sudo podman exec clab-l3vpn-pe1 vtysh -c "show ip route vrf blue"

C>* 192.168.1.0/24 is directly connected, eth2, weight 1, 00:00:08
B>  192.168.2.0/24 [200/0] via fc00:0:2::1 (vrf default) (recursive), label 16, seg6 fc00:0:2:1::, weight 1, 00:00:04

C>* 192.168.1.0/24 is directly connected, eth3, weight 1, 00:00:08
B>  192.168.2.0/24 [200/0] via fc00:0:2::1 (vrf default) (recursive), label 32, seg6 fc00:0:2:2::, weight 1, 00:00:04

(FRR also prints a legend of route codes, which I left out.) Each VRF has its own 192.168.1.0/24, connected on a different interface, and a BGP route to 192.168.2.0/24 through PE2. The SRv6 SID FRR uses for it is seg6 fc00:0:2:1:: in red and fc00:0:2:2:: in blue. They differ only in the fourth group, which is the function that selects the VRF. FRR programs these into the kernel as encapsulating routes:

sudo podman exec clab-l3vpn-pe1 ip route show vrf red
sudo podman exec clab-l3vpn-pe1 ip route show vrf blue

192.168.1.0/24 dev eth2 proto kernel scope link src 192.168.1.1 
192.168.2.0/24 nhid 35  encap seg6 mode encap segs 1 [ fc00:0:2:1:: ] via inet6 2001:db8:1::2 dev eth1 proto bgp metric 20 

192.168.1.0/24 dev eth3 proto kernel scope link src 192.168.1.1 
192.168.2.0/24 nhid 37  encap seg6 mode encap segs 1 [ fc00:0:2:2:: ] via inet6 2001:db8:1::2 dev eth1 proto bgp metric 20 

Traffic from the red VRF to 192.168.2.0/24 is encapsulated with the segment fc00:0:2:1:: and sent toward the P router. Blue traffic to the same prefix uses fc00:0:2:2::. On PE2, these SIDs are the other half of the mechanism:

sudo podman exec clab-l3vpn-pe2 ip -6 route show | grep seg6
sudo podman exec clab-l3vpn-pe2 vtysh -c "show segment-routing srv6 sid"

fc00:0:2:1:: nhid 19  encap seg6local action End.DT46 vrftable 10 dev red proto bgp metric 20 pref medium
fc00:0:2:2:: nhid 20  encap seg6local action End.DT46 vrftable 20 dev blue proto bgp metric 20 pref medium

 SID           Behavior    Context       Daemon/Instance    Locator    AllocationType  
 ------------------------  ------------  -----------------  ---------  ----------------
 fc00:0:2:1::  uDT46       VRF 'red'     bgp(0)             MAIN       dynamic         
 fc00:0:2:2::  uDT46       VRF 'blue'    bgp(0)             MAIN       dynamic         

Each SID on PE2 maps to one VRF. The kernel route says End.DT46 vrftable 10: decapsulate, then look up the inner packet in table 10, which is the red VRF. FRR calls the same behavior uDT46, the micro-SID form. These are the addresses PE1 uses as the destination of its encapsulated packets.

Customer traffic

Ping from each customer’s site A to its site B. Both customers use the same destination address:

sudo podman exec clab-l3vpn-red-a ping -c 2 192.168.2.2
sudo podman exec clab-l3vpn-blue-a ping -c 2 192.168.2.2

PING 192.168.2.2 (192.168.2.2): 56 data bytes
64 bytes from 192.168.2.2: seq=0 ttl=63 time=0.168 ms
64 bytes from 192.168.2.2: seq=1 ttl=63 time=0.138 ms

--- 192.168.2.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.138/0.153/0.168 ms
PING 192.168.2.2 (192.168.2.2): 56 data bytes
64 bytes from 192.168.2.2: seq=0 ttl=63 time=0.169 ms
64 bytes from 192.168.2.2: seq=1 ttl=63 time=0.123 ms

--- 192.168.2.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.123/0.146/0.169 ms

Both work, but do they reach the right hosts? There are two machines with the address 192.168.2.2, red-b and blue-b. Capture on both while red-a pings:

sudo podman exec clab-l3vpn-red-b tcpdump -nn -i eth1 icmp
sudo podman exec clab-l3vpn-blue-b tcpdump -nn -i eth1 icmp
sudo podman exec clab-l3vpn-red-a ping -c 2 192.168.2.2

red-b:
21:16:23.430822 IP 192.168.1.2 > 192.168.2.2: ICMP echo request, id 5, seq 0, length 64
21:16:23.430840 IP 192.168.2.2 > 192.168.1.2: ICMP echo reply, id 5, seq 0, length 64

blue-b:
0 packets captured
0 packets received by filter
0 packets dropped by kernel

red-b receives the requests and answers them. blue-b, which has the same address, sees nothing at all. The red VRF only knows the red route, and the red SID only decapsulates into the red VRF.

On the wire

The P router sits between the PEs. Capture on its link toward PE1 while red-a pings:

sudo podman exec clab-l3vpn-p tcpdump -nn -i eth1 -v ip6
sudo podman exec clab-l3vpn-red-a ping -c 1 192.168.2.2

tcpdump prints the packet on one long line. I’ve wrapped it here:

IP6 (hlim 63, next-header Routing (43), payload length 108)
    fc00:0:1::1 > fc00:0:2:1::: RT6 (len=2, type=4, segleft=0, last-entry=0, flags=0x0, tag=0,
    [0]fc00:0:2:1::)
    IP (tos 0x0, ttl 64, id 31646, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.2 > 192.168.2.2: ICMP echo request, id 6, seq 0, length 64

The customer’s original packet, 192.168.1.2 to 192.168.2.2, is inside an outer IPv6 packet. Its source is PE1’s fc00:0:1::1, the encapsulation source address. Its destination is fc00:0:2:1::, PE2’s service SID for the red VRF, and the segment routing header carries that single segment. Nothing in the packet says “red” in words: the destination SID is what selects the customer. The P router saw an IPv6 packet addressed inside fc00:0:2::/48, matched the static route toward PE2 and forwarded it. It holds no customer routes and no SIDs.

Tear it down

sudo containerlab --runtime podman destroy -t topology.clab.yml --cleanup