An interior routing protocol (an IGP) answers the question “how do routers inside one network reach each other?” by discovering the topology automatically. BGP answers a different question: “how do independently operated networks tell each other what they can reach?” It is the protocol that connects the networks of the internet, and it is also widely used inside data centers and provider networks.
BGP has no neighbor discovery of its own. You tell each router exactly who its neighbors are, and each router tells them which prefixes it can reach and how. That’s why so much of BGP is configuration.
This post builds a four-router lab spanning three autonomous systems and shows the fundamentals: sessions, address families, advertising prefixes, AS paths, the difference between eBGP and iBGP, and what happens when a session goes down.
How BGP works
Autonomous systems. An autonomous system (AS) is a network under one administration, identified by an AS number. The lab uses numbers from the private range, 64512 to 65534.
Sessions. BGP routers are called peers or neighbors. Each pair is configured explicitly, and the session runs over TCP port 179. The routers exchange an OPEN message to agree on parameters, then keepalives, and from then on UPDATE messages that advertise or withdraw prefixes. If the two ends are in different autonomous systems the session is eBGP. If they are in the same one it is iBGP.
Address families. A session carries routes for an address family, such as IPv4 unicast or IPv6 unicast, and you enable each family per neighbor. In this lab every pair of routers has two sessions: one over IPv4 that carries IPv4 prefixes, and one over IPv6 that carries IPv6 prefixes.
Path attributes. Every prefix is advertised along with attributes that describe it. Two matter here:
- AS_PATH lists the autonomous systems the route has passed through. A router adds its own AS number when it sends a route over eBGP. A router that sees its own AS number in a received path discards the route, which is how BGP avoids loops.
- NEXT_HOP is the address to forward traffic to. A router sets itself as the next hop when it sends a route over eBGP. Over iBGP it leaves the next hop unchanged.
That last point is the main difference between eBGP and iBGP in practice, and the lab shows it.
The lab
Four routers in a line. r2 and r3 are in the same AS and peer with iBGP. Every other session is eBGP.
AS 65001 AS 65002 AS 65002 AS 65003
r1 ──────────── r2 ──────────── r3 ──────────── r4
eBGP iBGP eBGP
| Node | AS | Router ID | IPv4 prefix | IPv6 prefix |
|---|---|---|---|---|
| r1 | 65001 | 1.1.1.1 | 10.1.0.0/24 | 2001:db8:1::/64 |
| r2 | 65002 | 2.2.2.2 | 10.2.0.0/24 | 2001:db8:2::/64 |
| r3 | 65002 | 3.3.3.3 | 10.3.0.0/24 | 2001:db8:3::/64 |
| r4 | 65003 | 4.4.4.4 | 10.4.0.0/24 | 2001:db8:4::/64 |
Each prefix sits on the router’s loopback, as 10.N.0.1/24 and 2001:db8:N::1/64.
| Node | Interface | IPv4 address | IPv6 address | Connected to |
|---|---|---|---|---|
| r1 | eth1 | 10.0.12.1/24 | 2001:db8:12::1/64 | r2 |
| r2 | eth1 | 10.0.12.2/24 | 2001:db8:12::2/64 | r1 |
| r2 | eth2 | 10.0.23.1/24 | 2001:db8:23::1/64 | r3 |
| r3 | eth1 | 10.0.23.2/24 | 2001:db8:23::2/64 | r2 |
| r3 | eth2 | 10.0.34.1/24 | 2001:db8:34::1/64 | r4 |
| r4 | eth1 | 10.0.34.2/24 | 2001:db8:34::2/64 | r3 |
The sessions run directly between the link addresses, so no other routing protocol is needed. The routers run FRR as containers, deployed with containerlab.
Get the files from the lab repository and cd into bgp-basics.
The configuration
This is r2’s frr.conf:
frr defaults traditional
hostname r2
!
interface lo
ip address 10.2.0.1/24
ipv6 address 2001:db8:2::1/64
exit
!
interface eth1
ip address 10.0.12.2/24
ipv6 address 2001:db8:12::2/64
exit
!
interface eth2
ip address 10.0.23.1/24
ipv6 address 2001:db8:23::1/64
exit
!
router bgp 65002
bgp router-id 2.2.2.2
no bgp default ipv4-unicast
no bgp ebgp-requires-policy
neighbor 10.0.12.1 remote-as 65001
neighbor 10.0.12.1 timers connect 5
neighbor 10.0.23.2 remote-as 65002
neighbor 10.0.23.2 timers connect 5
neighbor 2001:db8:12::1 remote-as 65001
neighbor 2001:db8:12::1 timers connect 5
neighbor 2001:db8:23::2 remote-as 65002
neighbor 2001:db8:23::2 timers connect 5
!
address-family ipv4 unicast
network 10.2.0.0/24
neighbor 10.0.12.1 activate
neighbor 10.0.23.2 activate
neighbor 10.0.23.2 next-hop-self
exit-address-family
!
address-family ipv6 unicast
network 2001:db8:2::/64
neighbor 2001:db8:12::1 activate
neighbor 2001:db8:23::2 activate
neighbor 2001:db8:23::2 next-hop-self
exit-address-family
exit
| Line | What it does |
|---|---|
router bgp 65002 | Starts the BGP process and sets the router’s AS. |
bgp router-id 2.2.2.2 | Gives the router a 32-bit identifier, written like an IPv4 address. |
neighbor … remote-as | Defines a peer. BGP compares the remote AS with the local one to decide whether the session is eBGP or iBGP: 65001 is different from 65002, so the first neighbor is eBGP, and the second, also 65002, is iBGP. Each peer appears twice, once by its IPv4 address and once by its IPv6 address. |
no bgp default ipv4-unicast | By default FRR activates IPv4 unicast on every new neighbor, including the IPv6 ones. Turning that off lets the lab activate each neighbor in exactly the family it should carry. |
address-family ipv4 unicast | Starts the IPv4 address family. The lines below it apply to IPv4 only. |
address-family ipv6 unicast | Starts the IPv6 address family. The lines below it apply to IPv6 only. |
network 10.2.0.0/24 | Originates a prefix into BGP. FRR only does this if the prefix exists in the routing table, here as the connected route on the loopback. |
neighbor … activate | Turns on the address family for a neighbor. |
neighbor … next-hop-self | Applies to the iBGP neighbor in each family. A later section explains why. |
no bgp ebgp-requires-policy | By default FRR won’t exchange routes with an eBGP neighbor until an inbound and an outbound policy are configured. Without this line the sessions come up but no prefixes move, and the summary shows (Policy) in the prefix columns. Writing policy is a topic of its own, so the lab turns the requirement off. |
timers connect 5 | Retries a failed connection after 5 seconds. The routers in the lab don’t all start at the same instant, and with the default timer I measured about 32 seconds before every session was up. With this line it takes about 7. |
FRR also needs the BGP daemon enabled, which the lab’s daemons file does with bgpd=yes.
Deploy
sudo containerlab --runtime podman deploy -t topology.clab.yml
After about ten seconds, check r2’s sessions:
sudo podman exec clab-bgp-r2 vtysh -c "show bgp summary"
IPv4 Unicast Summary:
BGP router identifier 2.2.2.2, local AS number 65002 VRF default vrf-id 0
BGP table version 4
RIB entries 4, using 640 bytes of memory
Peers 2, using 46 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.0.12.1 4 65001 6 8 4 0 0 00:00:36 1 4 N/A
10.0.23.2 4 65002 5 5 4 0 0 00:00:36 2 2 N/A
Total number of neighbors 2
IPv6 Unicast Summary:
BGP router identifier 2.2.2.2, local AS number 65002 VRF default vrf-id 0
BGP table version 4
RIB entries 4, using 640 bytes of memory
Peers 2, using 46 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
2001:db8:12::1 4 65001 7 7 4 0 0 00:00:35 1 4 N/A
2001:db8:23::2 4 65002 5 5 4 0 0 00:00:31 2 2 N/A
Total number of neighbors 2
There is one summary per address family, and each line is a session. When a number appears in the State/PfxRcd column, the session is Established and the number is how many prefixes r2 received: 1 from r1 (its own prefix), and 2 from r3 (r3’s prefix and r4’s). PfxSnt can look larger than you expect. FRR also sends a neighbor the routes that came from that neighbor, and the neighbor discards them because its own AS number is in the path. That’s why r2 reports 4 sent to r1, which counted only 3 as received.
The BGP table
On r4, at the far end, one table per family:
sudo podman exec clab-bgp-r4 vtysh -c "show bgp ipv4 unicast"
sudo podman exec clab-bgp-r4 vtysh -c "show bgp ipv6 unicast"
Network Next Hop Metric LocPrf Weight Path
*> 10.1.0.0/24 10.0.34.1 0 65002 65001 i
*> 10.2.0.0/24 10.0.34.1 0 65002 i
*> 10.3.0.0/24 10.0.34.1 0 0 65002 i
*> 10.4.0.0/24 0.0.0.0 0 32768 i
Displayed 4 routes and 4 total paths
Network Next Hop Metric LocPrf Weight Path
*> 2001:db8:1::/64 fe80::a8c1:abff:fefb:65f2
0 65002 65001 i
*> 2001:db8:2::/64 fe80::a8c1:abff:fefb:65f2
0 65002 i
*> 2001:db8:3::/64 fe80::a8c1:abff:fefb:65f2
0 0 65002 i
*> 2001:db8:4::/64 :: 0 32768 i
Displayed 4 routes and 4 total paths
(FRR prints a legend of status codes above each table, which I left out.) * means the route is valid and > that it is the best one. The Path column is the AS_PATH. r1’s prefix arrives with the path 65002 65001: it started in AS 65001, and AS 65002 added itself when passing it on. r4’s own prefix has no AS path yet and a next hop of 0.0.0.0 (or :: for IPv6), meaning it is local.
The IPv4 next hop is the neighbor’s address on the link. The IPv6 next hops are link-local addresses, because r3 is directly connected. The detail view for an IPv4 prefix shows everything BGP knows about it:
sudo podman exec clab-bgp-r4 vtysh -c "show bgp ipv4 unicast 10.1.0.0/24"
BGP routing table entry for 10.1.0.0/24, version 4
Paths: (1 available, best #1, table default)
Advertised to peers:
10.0.34.1
65002 65001
10.0.34.1 from 10.0.34.1 (3.3.3.3)
Origin IGP, valid, external, best (First path received)
The route is external, meaning it was learned over eBGP, from the neighbor with router ID 3.3.3.3. The routes BGP selects go into the router’s routing table:
sudo podman exec clab-bgp-r4 vtysh -c "show ip route bgp"
sudo podman exec clab-bgp-r4 vtysh -c "show ipv6 route bgp"
IPv4 unicast VRF default:
B>* 10.1.0.0/24 [20/0] via 10.0.34.1, eth1, weight 1, 00:00:35
B>* 10.2.0.0/24 [20/0] via 10.0.34.1, eth1, weight 1, 00:00:35
B>* 10.3.0.0/24 [20/0] via 10.0.34.1, eth1, weight 1, 00:00:35
IPv6 unicast VRF default:
B>* 2001:db8:1::/64 [20/0] via fe80::a8c1:abff:fefb:65f2, eth1, weight 1, 00:00:32
B>* 2001:db8:2::/64 [20/0] via fe80::a8c1:abff:fefb:65f2, eth1, weight 1, 00:00:32
B>* 2001:db8:3::/64 [20/0] via fe80::a8c1:abff:fefb:65f2, eth1, weight 1, 00:00:35
B marks BGP routes, and 20 is the administrative distance of eBGP.
Now test the data path from r1 to r4, over both families:
sudo podman exec clab-bgp-r1 ping -c 3 -I 10.1.0.1 10.4.0.1
sudo podman exec clab-bgp-r1 ping -6 -c 3 -I 2001:db8:1::1 2001:db8:4::1
64 bytes from 10.4.0.1: seq=2 ttl=62 time=0.105 ms
--- 10.4.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.068/0.084/0.105 ms
64 bytes from 2001:db8:4::1: seq=2 ttl=62 time=0.102 ms
--- 2001:db8:4::1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.095/0.109/0.130 ms
The -I option sets the source address to r1’s loopback, which is inside the prefix r1 advertises. The replies come back to that prefix, and r4 has a route for it. The lab doesn’t advertise the link subnets, so a ping from a link address would have no way home.
eBGP and iBGP
Compare how the same prefix, 10.1.0.0/24, looks on r3, which learned it over iBGP from r2, and on r4, which learned it over eBGP from r3 (shown above):
sudo podman exec clab-bgp-r3 vtysh -c "show bgp ipv4 unicast 10.1.0.0/24"
sudo podman exec clab-bgp-r3 vtysh -c "show ip route bgp"
sudo podman exec clab-bgp-r3 vtysh -c "show ipv6 route bgp"
BGP routing table entry for 10.1.0.0/24, version 4
Paths: (1 available, best #1, table default)
Advertised to peers:
10.0.34.2
65001
10.0.23.1 from 10.0.23.1 (2.2.2.2)
Origin IGP, metric 0, localpref 100, valid, internal, best (First path received)
IPv4 unicast VRF default:
B>* 10.1.0.0/24 [200/0] via 10.0.23.1, eth1, weight 1, 00:00:32
B>* 10.2.0.0/24 [200/0] via 10.0.23.1, eth1, weight 1, 00:00:32
B>* 10.4.0.0/24 [20/0] via 10.0.34.2, eth2, weight 1, 00:00:34
IPv6 unicast VRF default:
B>* 2001:db8:1::/64 [200/0] via 2001:db8:23::1, eth1, weight 1, 00:00:32
B>* 2001:db8:2::/64 [200/0] via fe80::a8c1:abff:fe0d:2429, eth1, weight 1, 00:00:32
B>* 2001:db8:4::/64 [20/0] via fe80::a8c1:abff:fe74:81b, eth2, weight 1, 00:00:34
The two views of the same prefix, side by side:
| r4 (eBGP) | r3 (iBGP) | |
|---|---|---|
| AS path | 65002 65001 | 65001 |
| Administrative distance | 20 | 200 |
| Route type in the BGP table | external | internal |
| Local preference | none | localpref 100 |
| IPv4 next hop | 10.0.34.1 (r3) | 10.0.23.1 (r2) |
AS 65002 adds its number when a route leaves it over eBGP, not between its own routers, so r3 has no reason to see it in the path. Routes learned over iBGP are trusted less than routes learned over eBGP, which is why their distance is 200 instead of 20: r1’s and r2’s prefixes on r3 are [200/0], while r4’s prefix, learned over eBGP, is [20/0]. The local preference is a value that only travels inside an AS, and 100 is the default.
The next hop is also worth a look. r3 sends traffic for r1’s prefix to 10.0.23.1, which is r2. That is because of next-hop-self on r2. iBGP leaves the next hop unchanged, so without it r2 would pass r1’s route to r3 with r1’s own address, 10.0.12.1, as the next hop. That address is on a link r3 isn’t attached to, and nothing in this lab’s BGP tells r3 how to reach it. With next-hop-self, r2 advertises itself, an address r3 can reach directly. The IPv6 entry for r1’s prefix shows the same thing: its next hop is 2001:db8:23::1, r2’s address.
When a session goes down
The two address families are independent, so a session can go down in one while the other keeps working. Administratively shut down r2’s IPv4 session with r1:
sudo podman exec clab-bgp-r2 vtysh \
-c "configure terminal" \
-c "router bgp 65002" \
-c "neighbor 10.0.12.1 shutdown"
sudo podman exec clab-bgp-r2 vtysh -c "show bgp summary"
sudo podman exec clab-bgp-r4 vtysh -c "show bgp ipv4 unicast"
sudo podman exec clab-bgp-r4 vtysh -c "show bgp ipv6 unicast"
The session lines from the summary:
IPv4 Unicast Summary:
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
10.0.12.1 4 65001 7 10 0 0 0 00:00:03 Idle (Admin) 0 N/A
10.0.23.2 4 65002 6 15 5 0 0 00:01:09 2 1 N/A
IPv6 Unicast Summary:
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
2001:db8:12::1 4 65001 8 8 4 0 0 00:01:08 1 4 N/A
2001:db8:23::2 4 65002 6 6 4 0 0 00:01:04 2 2 N/A
And r4’s two tables:
Network Next Hop Metric LocPrf Weight Path
*> 10.2.0.0/24 10.0.34.1 0 65002 i
*> 10.3.0.0/24 10.0.34.1 0 0 65002 i
*> 10.4.0.0/24 0.0.0.0 0 32768 i
Displayed 3 routes and 3 total paths
Network Next Hop Metric LocPrf Weight Path
*> 2001:db8:1::/64 fe80::a8c1:abff:fefb:65f2
0 65002 65001 i
*> 2001:db8:2::/64 fe80::a8c1:abff:fefb:65f2
0 65002 i
*> 2001:db8:3::/64 fe80::a8c1:abff:fefb:65f2
0 0 65002 i
*> 2001:db8:4::/64 :: 0 32768 i
Displayed 4 routes and 4 total paths
The IPv4 session is Idle (Admin). r2 withdrew r1’s IPv4 prefix and told r3, and r3 told r4, so 10.1.0.0/24 is gone from r4’s IPv4 table. In my runs it disappeared in about 380 milliseconds. The IPv6 session never went down, so 2001:db8:1::/64 is still in r4’s IPv6 table.
Remove the shutdown and the session re-establishes:
sudo podman exec clab-bgp-r2 vtysh \
-c "configure terminal" \
-c "router bgp 65002" \
-c "no neighbor 10.0.12.1 shutdown"
10.1.0.0/24 is back on r4 about 2.4 seconds later. A session can also fail without being shut down. If the link under an eBGP session goes down, FRR tears the session down immediately, but a peer that merely goes silent is only declared dead when the hold timer expires, which defaults to 180 seconds here (keepalives go every 60).
Tear it down
sudo containerlab --runtime podman destroy -t topology.clab.yml --cleanup