SRv6: segment routing over IPv6

An IP network normally forwards each packet by its destination address alone, and the path is whatever the routing protocols compute. That is usually what you want, until you need traffic to take a specific path, pass through a particular node, or arrive at a particular place with an instruction attached, such as “deliver this inside customer A’s VPN”. Networks have traditionally done that with extra machinery: MPLS label stacks, and the protocols that distribute the labels.

Segment routing takes another approach. The sender puts a list of instructions, called segments, into the packet itself, and the network follows them. SRv6 is segment routing carried in IPv6: a segment is an IPv6 address, the list travels in an IPv6 extension header, and the routers that aren’t named in the list just forward ordinary IPv6. There is no separate data plane to run next to IPv6.

This post explains how SRv6 works and shows it on a chain of Linux routers with real packets: a segment list steering traffic through a waypoint, a router that does nothing SRv6-specific, and the compressed form of the list.

How SRv6 works

Segments and SIDs. A segment is an instruction. In SRv6 its identifier, the SID, is a 128-bit IPv6 address. Where an ordinary destination address says “deliver to this host”, a SID says “go to this node, and do this”.

Locator and function. A SID has two parts. The locator is a prefix that routes to the node that owns the SID, and the function says what that node should do. In fc00:0:3::e, the locator is fc00:0:3::/48, and e selects the function. The rest of the network only needs a route to the locator, which is plain IPv6 routing, and never has to know about individual SIDs.

Behaviors. The function behind a SID is called a behavior. A few standard ones (RFC 8986):

This lab uses End and End.DX4.

The segment routing header. The list of segments travels in an IPv6 extension header called the segment routing header (SRH), routing header type 4. It holds the segment list and a counter, segments left. The list is stored in reverse order, so the last segment is first. The active segment is always the one in the IPv6 destination address, which is how the packet gets routed to the next node in the list. When a node runs the End behavior, it decrements segments left and copies the next segment into the destination address.

Encapsulation. The node where traffic enters the SRv6 network, the headend, wraps the original packet in a new IPv6 header and the SRH. This is the H.Encaps behavior. The last segment in the list is usually a decapsulation behavior such as End.DX4, which removes the outer header and delivers the original packet. In between, the packet is an ordinary IPv6 packet.

Transit nodes. Only the nodes named in the list act on it. A router that is merely on the way sees a packet addressed to a locator it already has a route for, and forwards it like any other IPv6 packet. It needs no SRv6 configuration at all.

Compressed segment lists (uSID). A full list of 128-bit segments adds up quickly: the SRH in this lab costs 40 bytes for two segments. In the compressed form, a list of micro-SIDs (uSIDs) is packed into a single IPv6 address. A common block of the address is shared, and each following 16-bit piece is one instruction. The node shifts the remaining pieces up by 16 bits after it has used its own, so the address itself acts as the segment list.

The lab

Four Linux routers in a chain, with a host at each end. The hosts use IPv4. The routers carry that IPv4 traffic across an IPv6 network, using SRv6.

 h1 ──── r1 ──── r2 ──── r3 ──── r4 ──── h2
        ingress  plain  waypoint egress

Each router owns the locator fc00:0:N::/48, where N is its number, and the loopback address is the first address in it. The underlay is static IPv6 routes, so every router has a route to every other router’s locator. The SIDs sit inside those locators:

For traffic from h1 to h2, r1 will encapsulate with the segment list fc00:0:3::e, then fc00:0:4::d: go to r3 and run End, then go to r4 and deliver to h2. r2 sits between them and is not in the list.

The nodes are containers deployed with containerlab. You need containerlab and podman, and a Linux kernel with SRv6 support. They run a small image built from the official Alpine Linux image with the iproute2 and tcpdump packages. The lab contains its two-line Containerfile, and you build the image once:

sudo podman build -t localhost/srv6-node:latest image/

Get the files from the lab repository and cd into srv6-basics.

The configuration

Each router runs a small script when the lab starts. This is r1, the ingress:

#!/bin/sh
set -e
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.seg6_enabled=1
sysctl -w net.ipv6.conf.eth2.seg6_enabled=1

ip addr add 10.1.1.1/24 dev eth1
ip addr add 2001:db8:12::1/64 dev eth2
ip addr add fc00:0:1::1/128 dev lo

# Underlay: ordinary IPv6 routes to the other routers' locators
ip -6 route add fc00:0:2::/48 via 2001:db8:12::2
ip -6 route add fc00:0:3::/48 via 2001:db8:12::2
ip -6 route add fc00:0:4::/48 via 2001:db8:12::2

# Source address of the outer IPv6 header
ip sr tunsrc set fc00:0:1::1

# Ingress: traffic for h2's network gets a segment list: r3 (End), then r4 (End.DX4)
ip route add 10.2.2.0/24 encap seg6 mode encap segs fc00:0:3::e,fc00:0:4::d dev eth2

# Egress SID for the return path: decapsulate and hand the packet to h1
ip -6 route add fc00:0:1::d encap seg6local action End.DX4 nh4 10.1.1.2 dev eth1

And r3, the waypoint:

#!/bin/sh
set -e
sysctl -w net.ipv6.conf.all.forwarding=1
sysctl -w net.ipv6.conf.all.seg6_enabled=1
sysctl -w net.ipv6.conf.eth1.seg6_enabled=1
sysctl -w net.ipv6.conf.eth2.seg6_enabled=1

ip addr add 2001:db8:23::2/64 dev eth1
ip addr add 2001:db8:34::1/64 dev eth2
ip addr add fc00:0:3::1/128 dev lo

ip -6 route add fc00:0:1::/48 via 2001:db8:23::1
ip -6 route add fc00:0:2::/48 via 2001:db8:23::1
ip -6 route add fc00:0:4::/48 via 2001:db8:34::2

# End: process the segment list and forward to the next segment
ip -6 route add fc00:0:3::e encap seg6local action End dev eth1

r4 mirrors r1: an End.DX4 SID toward h2, and an encapsulating route for the way back, with the segments r3 then r1. r2 only has addresses, forwarding and the underlay routes.

Deploy and look at the routes

sudo containerlab --runtime podman deploy -t topology.clab.yml

SRv6 in Linux is expressed as routes. On r1:

sudo podman exec clab-srv6-r1 ip route show 10.2.2.0/24
sudo podman exec clab-srv6-r1 ip -6 route show | grep seg6

10.2.2.0/24  encap seg6 mode encap segs 2 [ fc00:0:3::e fc00:0:4::d ] dev eth2 scope link 
fc00:0:1::d  encap seg6local action End.DX4 nh4 10.1.1.2 dev eth1 metric 1024 pref medium

The first route is the headend: anything for 10.2.2.0/24 is encapsulated with the two segments. The second is r1’s own End.DX4 SID. On r3 and r4:

sudo podman exec clab-srv6-r3 ip -6 route show | grep seg6
sudo podman exec clab-srv6-r4 ip -6 route show | grep seg6

fc00:0:3::e  encap seg6local action End dev eth1 metric 1024 pref medium
fc00:0:4::d  encap seg6local action End.DX4 nh4 10.2.2.2 dev eth2 metric 1024 pref medium

And r2, the router in the middle, has nothing SRv6-specific at all:

sudo podman exec clab-srv6-r2 sysctl -n net.ipv6.conf.eth1.seg6_enabled
sudo podman exec clab-srv6-r2 ip -6 route show | grep -c seg6
sudo podman exec clab-srv6-r2 ip -6 route show | grep ^fc00

0
0
fc00:0:1::/48 via 2001:db8:12::1 dev eth1 metric 1024 pref medium
fc00:0:2::1 dev lo proto kernel metric 256 pref medium
fc00:0:3::/48 via 2001:db8:23::2 dev eth2 metric 1024 pref medium
fc00:0:4::/48 via 2001:db8:23::2 dev eth2 metric 1024 pref medium

SRv6 is disabled on its interfaces, it has no SRv6 routes, and its routing table holds only the locators, as plain IPv6 prefixes. Now ping from h1 to h2:

sudo podman exec clab-srv6-h1 ping -c 2 10.2.2.2

PING 10.2.2.2 (10.2.2.2): 56 data bytes
64 bytes from 10.2.2.2: seq=0 ttl=63 time=0.297 ms
64 bytes from 10.2.2.2: seq=1 ttl=63 time=0.109 ms

--- 10.2.2.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.109/0.203/0.297 ms

The hosts have nothing but IPv4 addresses and a default route, and the ping works across the SRv6 network.

The packet at each stage

To watch what happens to a packet, capture on three links: r2’s eth1 (before the waypoint), r3’s eth2 (after it), and r4’s eth2 (the link to h2). Start the captures first, in separate terminals, then ping once:

sudo podman exec clab-srv6-r2 tcpdump -nn -i eth1 -v ip6
sudo podman exec clab-srv6-r3 tcpdump -nn -i eth2 -v ip6
sudo podman exec clab-srv6-r4 tcpdump -nn -i eth2 -v
sudo podman exec clab-srv6-h1 ping -c 1 10.2.2.2

tcpdump prints each packet on one long line. I’ve wrapped them here for reading. On r2’s eth1, the request just after it left r1:

IP6 (hlim 63, next-header Routing (43), payload length 124)
    fc00:0:1::1 > fc00:0:3::e: RT6 (len=4, type=4, segleft=1, last-entry=1, flags=0x0, tag=0,
    [0]fc00:0:4::d, [1]fc00:0:3::e)
    IP (tos 0x0, ttl 64, id 2968, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.1.2 > 10.2.2.2: ICMP echo request, id 5, seq 0, length 64

The original ICMP packet from h1 to h2, at the bottom, is wrapped in an outer IPv6 header. Its source is r1’s fc00:0:1::1 and its destination is fc00:0:3::e, r3’s End SID: the first segment, which is what routes the packet toward r3. The RT6 is the SRH: type=4 is the segment routing header type, and [0]fc00:0:4::d, [1]fc00:0:3::e is the segment list, in reverse order. segleft=1 says one segment is left after the current one. next-header Routing (43) in the first line says the IPv6 header is followed by a routing header.

On r3’s eth2, after r3 ran End:

IP6 (hlim 61, next-header Routing (43), payload length 124)
    fc00:0:1::1 > fc00:0:4::d: RT6 (len=4, type=4, segleft=0, last-entry=1, flags=0x0, tag=0,
    [0]fc00:0:4::d, [1]fc00:0:3::e)
    IP (tos 0x0, ttl 64, id 2968, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.1.2 > 10.2.2.2: ICMP echo request, id 5, seq 0, length 64

Two things changed. The destination is now fc00:0:4::d, r4’s SID, and segleft went from 1 to 0. That is what End does: decrement segments left and copy the next segment into the destination address. The segment list itself is unchanged. The hop limit dropped from 63 to 61, because r2 and r3 each forwarded the packet. On r4’s eth2, the link to h2:

IP (tos 0x0, ttl 63, id 2968, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.1.2 > 10.2.2.2: ICMP echo request, id 5, seq 0, length 64

It is the original IPv4 packet again, with the same id 2968, delivered by r4’s End.DX4 behavior, which removed the outer header. Note that r2 handled this packet without knowing anything about SRv6. It saw an IPv6 packet for fc00:0:3::e, matched the locator route fc00:0:3::/48, and forwarded it.

Compressing the segment list with uSID

The segment list above is carried in an SRH that costs 40 bytes: 8 bytes of header and 16 for each of the two segments. With uSID, the same instructions fit into one address, with no SRH. Here the block is fc00:0, the first 32 bits, which all the nodes share, followed by 16-bit pieces: 3 for r3, 4 for r4, and d as the final function, so the single destination is fc00:0:3:4:d::.

Three changes are needed. r3 gets an End behavior with the next-csid flavor on its whole locator. r4 and r1 get End.DX4 SIDs shaped like uSIDs. The headends switch to a single segment, using the encap.red mode, which leaves out the SRH when there is only one segment:

sudo podman exec clab-srv6-r3 ip -6 route add fc00:0:3::/48 \
  encap seg6local action End flavors next-csid lblen 32 nflen 16 dev eth1
sudo podman exec clab-srv6-r4 ip -6 route add fc00:0:4:d::/64 \
  encap seg6local action End.DX4 nh4 10.2.2.2 dev eth2
sudo podman exec clab-srv6-r4 ip route replace 10.1.1.0/24 \
  encap seg6 mode encap.red segs fc00:0:3:1:d:: dev eth1
sudo podman exec clab-srv6-r1 ip -6 route add fc00:0:1:d::/64 \
  encap seg6local action End.DX4 nh4 10.1.1.2 dev eth1
sudo podman exec clab-srv6-r1 ip route replace 10.2.2.0/24 \
  encap seg6 mode encap.red segs fc00:0:3:4:d:: dev eth2

lblen 32 is the length of the shared block and nflen 16 the length of each node piece. Now the routes show a single segment in each direction, and the ping still works:

sudo podman exec clab-srv6-r1 ip route show 10.2.2.0/24
sudo podman exec clab-srv6-r3 ip -6 route show | grep next-csid
sudo podman exec clab-srv6-h1 ping -c 2 10.2.2.2

10.2.2.0/24  encap seg6 mode encap.red segs 1 [ fc00:0:3:4:d:: ] dev eth2 scope link 
fc00:0:3::/48  encap seg6local action End flavors next-csid lblen 32 nflen 16 dev eth1 metric 1024 pref medium
PING 10.2.2.2 (10.2.2.2): 56 data bytes
64 bytes from 10.2.2.2: seq=0 ttl=63 time=0.107 ms
64 bytes from 10.2.2.2: seq=1 ttl=63 time=0.150 ms

--- 10.2.2.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.107/0.128/0.150 ms

Capture on r2’s eth1 and r3’s eth2 again while pinging, and compare with the earlier captures. Before the waypoint:

IP6 (hlim 63, next-header IPIP (4), payload length 84)
    fc00:0:1::1 > fc00:0:3:4:d:::
    IP (tos 0x0, ttl 64, id 3519, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.1.2 > 10.2.2.2: ICMP echo request, id 7, seq 0, length 64

And after r3:

IP6 (hlim 61, next-header IPIP (4), payload length 84)
    fc00:0:1::1 > fc00:0:4:d:::
    IP (tos 0x0, ttl 64, id 3519, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.1.2 > 10.2.2.2: ICMP echo request, id 7, seq 0, length 64

There is no routing header any more: the next header is IPIP (4), the inner IPv4 packet, and the payload length is 84 instead of 124. The 40 bytes of SRH are gone. The destination is fc00:0:3:4:d:: before r3, and fc00:0:4:d:: after: r3’s End with next-csid used its own piece, 3, and shifted the rest up by 16 bits. The reverse direction works the same way, with r4 sending to fc00:0:3:1:d::. On r2’s eth2, the reply arriving from r3 already has the destination fc00:0:1:d::, after r3’s shift:

IP6 (hlim 62, next-header IPIP (4), payload length 84)
    fc00:0:4::1 > fc00:0:1:d:::
    IP (tos 0x0, ttl 64, id 52354, offset 0, flags [none], proto ICMP (1), length 84)
    10.2.2.2 > 10.1.1.2: ICMP echo reply, id 7, seq 0, length 64

The routing is still plain IPv6 on the locators, and r2 is still not configured for anything. Only the way instructions are packed in the packet changed.

Tear it down

sudo containerlab --runtime podman destroy -t topology.clab.yml --cleanup