Data Center Networking

How servers inside a data center are connected: top-of-rack switches, spine-leaf fabrics built for east-west traffic, VXLAN and EVPN overlays, and the link speeds in use today.

Two directions of traffic

Network engineers describe data center traffic by direction, as if the network were drawn with the internet at the top and the servers at the bottom.

North-south traffic enters or leaves the data center: users reaching a web portal, servers calling an external API, replication to another site. It passes the edge routers, firewalls and load balancers. East-west traffic stays inside the data center, between servers: a web server querying a database, a hypervisor live-migrating a VM, a storage node copying data to two other nodes, a backup server reading every VM at night.

Twenty years ago north-south dominated. Virtualization, distributed applications built from many small services, software-defined storage and AI training changed that: one user request can cause dozens of internal requests, and a storage write is copied to several nodes before it is confirmed. Today east-west traffic is usually far larger than north-south, and the network design follows from that.

From three tiers to spine-leaf

The classic three-tier design

Campus-style networks have three layers: access switches for the servers, aggregation (distribution) switches above them, and a core on top. Redundant links between the layers create loops, so Spanning Tree Protocol blocks half of them. That design suits north-south traffic, but for east-west traffic it means paths of different lengths, blocked links that carry nothing, and a failure domain the size of a VLAN that can stretch across the whole data center.

Spine-leaf

A spine-leaf architecture has two layers of switches. Servers connect only to leaf switches; every leaf connects to every spine switch; spines connect only to leaves, never to each other. The design is a form of Clos network, named after Charles Clos, who described it for telephone exchanges in the 1950s.

  • Predictable distance. Any server is exactly leaf-spine-leaf away from any server in another rack, so latency is the same everywhere.
  • All links active. Leaf-to-spine links are routed (Layer 3), so there are no loops to block and traffic is spread over all spines.
  • Graceful failure. Losing one of four spines removes a quarter of the bandwidth, not connectivity.
  • Simple growth. More racks means more leaves; more bandwidth between racks means more spines. Very large sites add a third layer ("super-spines") that connects several spine-leaf pods.

Two kinds of leaf have special roles: the border leaf connects the fabric to the WAN, internet edge and firewalls (the north-south exit), and a services leaf may host load balancers or other shared appliances.

Four spine switches at the top and six leaf switches below. Every leaf has a link to every spine. Servers hang below leaves one to five; the sixth leaf is a border leaf connected to the WAN and internet. A highlighted path runs from a server under leaf 1 up to leaf 1, to spine 2, down to leaf 4 and to a server below it: east-west traffic. The border leaf carries north-south traffic. spine1 spine2 spine3 spine4 leaf1 leaf2 leaf3 leaf4 leaf5 border WAN / internet east-west: server → leaf1 → spine2 → leaf4 → server north-south: via the border leaf every leaf links to every spine; spines never link to each other
A small spine-leaf fabric. Any server in one rack reaches any server in another rack in three switch hops; the four spines give four equal paths.

Top-of-rack and end-of-row

A top-of-rack switch (ToR) is a switch mounted in each server rack, usually in the top rack units, that connects the servers in that rack with short copper or fiber cables and has a few fast uplinks to the spines. In spine-leaf fabrics the ToR switches are the leaves. The alternative, end-of-row (EoR), places larger switches at the end of each row and runs cables from every server along the row; it means fewer switches but much more cabling (see Racks, Rack Units and Cabling).

Racks normally get two ToR switches. Each server connects one port to each switch and bonds them with LACP (Link Aggregation Control Protocol). The two switches present themselves to the server as one, using multi-chassis link aggregation (MLAG; Cisco calls its version vPC) or EVPN multihoming. Either switch can then fail or be upgraded without disconnecting a server.

Using every path: ECMP

Because leaf-to-spine links are routed, a leaf learns the route to another leaf's subnets via every spine, all at equal cost. The fabric's routing protocol is usually eBGP (one autonomous system number per leaf, as described in RFC 7938) or OSPF; see BGP and Dynamic Routing and OSPF.

ECMP (equal-cost multipath) is the routing feature that installs several best paths with the same cost for one destination and spreads traffic over all of them. To keep the packets of one connection in order, a switch does not spread packets but flows: it hashes the 5-tuple (source and destination IP, protocol, source and destination port) and uses the result to pick a path. Every packet of the same flow gives the same hash and takes the same spine; different flows land on different spines.

That works well for thousands of small flows and badly for a few large ones. The example spreads web sessions and then four large backup streams over four spines. 

import hashlib

SPINES = ["spine1", "spine2", "spine3", "spine4"]


def pick_spine(src, dst, proto, sport, dport):
    # a switch hashes the 5-tuple; every packet of one flow takes the same path
    key = f"{src}|{dst}|{proto}|{sport}|{dport}".encode()
    return SPINES[int.from_bytes(hashlib.sha256(key).digest()[:4], "big") % len(SPINES)]


load = {spine: 0.0 for spine in SPINES}
# 400 small web flows of 0.05 Gbit/s each from different client ports
for port in range(40000, 40400):
    load[pick_spine("192.0.2.10", "198.51.100.20", "tcp", port, 443)] += 0.05
print("400 small flows:  ", "  ".join(f"{s} {g:4.1f}" for s, g in load.items()), "Gbit/s")

# add four backup streams of 20 Gbit/s each
for host in range(31, 35):
    spine = pick_spine(f"192.0.2.{host}", "198.51.100.50", "tcp", 50123, 10000)
    load[spine] += 20
    print(f"backup from 192.0.2.{host} -> {spine}")
print("with 4 big flows: ", "  ".join(f"{s} {g:4.1f}" for s, g in load.items()), "Gbit/s")
400 small flows:   spine1  5.0  spine2  5.1  spine3  5.7  spine4  4.1 Gbit/s
backup from 192.0.2.31 -> spine1
backup from 192.0.2.32 -> spine1
backup from 192.0.2.33 -> spine4
backup from 192.0.2.34 -> spine3
with 4 big flows:  spine1 45.0  spine2  5.1  spine3 25.7  spine4 24.1 Gbit/s

The small flows spread roughly evenly. Two of the four backup streams hash to spine1, which now carries 45 Gbit/s while spine2 carries almost nothing. Such elephant flows are a classic cause of "the fabric is half empty, yet one link is congested". Remedies include more parallel streams per job (more flows to hash), faster spine links, and newer load-balancing modes in switches that react to link load; on the operations side, look at per-link utilization, not only the fabric average.

Oversubscription and link speeds

Oversubscription

A leaf's oversubscription ratio is its total server-facing bandwidth divided by its total uplink bandwidth. At 3:1, the servers in a rack could together send three times what the uplinks carry, which is fine as long as they rarely do so at the same time. General-purpose racks often run 2:1 or 3:1; storage, virtualization and especially AI training fabrics aim for 1:1 (non-blocking), because their traffic really does peak all at once. 

# (design, server ports, Gbit/s each, uplinks to spines, Gbit/s each)
leaves = [
    ("general-purpose rack", 48, 25, 4, 100),
    ("general-purpose rack", 48, 25, 6, 100),
    ("virtualization/storage", 32, 100, 8, 400),
    ("AI/GPU back-end", 32, 400, 16, 800),
]

for design, down_n, down_g, up_n, up_g in leaves:
    down = down_n * down_g
    up = up_n * up_g
    ratio = down / up
    note = "non-blocking" if ratio <= 1 else f"{up / down_n:.1f} Gbit/s per server if all send at once"
    print(f"{design:<23} {down_n}x{down_g}G down = {down:>5} G, {up_n:>2}x{up_g}G up = {up:>5} G"
          f" -> {ratio:.1f}:1 ({note})")
general-purpose rack    48x25G down =  1200 G,  4x100G up =   400 G -> 3.0:1 (8.3 Gbit/s per server if all send at once)
general-purpose rack    48x25G down =  1200 G,  6x100G up =   600 G -> 2.0:1 (12.5 Gbit/s per server if all send at once)
virtualization/storage  32x100G down =  3200 G,  8x400G up =  3200 G -> 1.0:1 (non-blocking)
AI/GPU back-end         32x400G down = 12800 G, 16x800G up = 12800 G -> 1.0:1 (non-blocking)

The number of uplinks also sets the number of spines: a leaf with four uplinks can connect to at most four spines, one link each. Oversubscription at the spine level (spine-to-super-spine) adds to the leaf ratio for traffic between pods.

Where Typical speed (2026)
General-purpose servers 2 × 25 GbE
Virtualization, HCI and storage hosts 2 × 100 GbE increasingly standard (25 GbE in smaller sites)
AI/GPU servers 200 or 400 Gbit/s per GPU network card, often several per server
Leaf-to-spine 100G in older and smaller fabrics; 400G and 800G in new builds
Out-of-band management 1 GbE, on a separate network

400G and 800G Ethernet are standardized in IEEE 802.3df-2024. The next step, IEEE P802.3dj (200 Gbit/s per lane, up to 1.6 Tbit/s Ethernet), is targeted for completion in late 2026 and was not yet approved as of October 2026. Fast links use pluggable optics or direct-attach copper cables (DACs) for short runs within a rack; see transceivers and fiber optics.

Underlay and overlay

A routed fabric solves the bandwidth problem but creates a new one. Each leaf has its own IP subnets, and a VLAN ends at the leaf. Yet VMs must keep their IP address when they live-migrate to a host in another rack, and some applications still need two servers in the same Layer 2 segment. Also, the 12-bit VLAN ID allows only 4,094 VLANs, too few for a multi-tenant cloud.

The answer is to separate two networks that share the same cables:

  • The underlay network is the physical, routed IP fabric of leaves and spines. Its only job is to deliver IP packets between switches quickly and reliably, using ECMP. It knows nothing about tenants or VMs.
  • An overlay network is a virtual network built on top of the underlay by tunneling: the original frame is wrapped in an extra header and carried across the underlay to the switch or host where its destination lives. Overlays can be created, changed and deleted in software without touching the physical network.

The same idea underlies cloud virtual networks and software-defined networking. In data center fabrics, the standard overlay is VXLAN with EVPN.

VXLAN

VXLAN (Virtual Extensible LAN) is an overlay protocol that carries Ethernet frames inside UDP packets across a routed IP network, defined in RFC 7348 (August 2014, an Informational RFC). Each virtual network is identified by a 24-bit VXLAN network identifier (VNI), allowing about 16 million segments instead of 4,094 VLANs. The registered UDP destination port is 4789.

The device that wraps and unwraps frames is a VTEP (VXLAN tunnel endpoint). In a hardware fabric the VTEPs are the leaf switches; in some virtualization platforms the hypervisor is the VTEP. A leaf maps a local VLAN to a VNI, so a server sees an ordinary VLAN while the frame travels across the fabric as VXLAN.

Top: VM A connects to leaf 1, which is a VTEP. Leaf 1 sends the frame across the routed underlay to leaf 4, also a VTEP, which delivers it to VM B. Both leaves map VLAN 110 to VNI 10110. Bottom: the packet on the underlay consists of an outer Ethernet header, an outer IP header, a UDP header with port 4789 and a VXLAN header with the VNI, together 50 bytes added by the VTEP, followed by the original Ethernet frame from VM A, unchanged. VM A leaf1 VTEP underlay leaf4 VTEP VM B VLAN 110 = VNI 10110 VLAN 110 = VNI 10110 added by the VTEP: 50 bytes (IPv4) unchanged outer Ethernet outer IP UDP 4789 VXLAN VNI 10110 original Ethernet frame from VM A outer IP: 192.0.2.111 → 192.0.2.114 (VTEP to VTEP) the underlay routes on the outer header only
VXLAN encapsulation. The underlay sees an ordinary UDP packet between two leaf switches; only the VTEPs look inside.

VXLAN and MTU

The outer headers add 50 bytes (14 Ethernet, 20 IPv4, 8 UDP, 8 VXLAN; more with IPv6 or an outer VLAN tag). A VM that sends a full 1,500-byte packet therefore needs an underlay MTU of at least 1,550 bytes. Fabrics normally run jumbo frames (9,000 bytes or more) on every underlay link to leave room. If one link keeps the default 1,500, small packets work and large ones vanish: pings succeed while file copies and web pages hang.

VTEPs also pick the outer UDP source port from a hash of the inner headers. Different inner flows thus get different outer 5-tuples, and ECMP in the underlay can still spread them over the spines.

EVPN: the control plane

Plain VXLAN learns like a classic switch: a VTEP floods unknown traffic to all other VTEPs in the VNI and learns MAC addresses from what comes back ("flood and learn"). That works, but floods grow with the fabric and every VTEP must be told who its peers are.

EVPN (Ethernet VPN) replaces flooding with BGP. Each leaf advertises, as BGP routes, the MAC and IP addresses of the hosts connected to it and the VNIs it serves; all other leaves learn from those routes which VTEP to send each destination to. EVPN was first defined for MPLS networks in RFC 7432 (2015); RFC 8365 (2018) defines its use as the control plane for overlays such as VXLAN, the combination usually called EVPN-VXLAN. Benefits:

  • Less flooding. Leaves know remote MAC and IP addresses in advance and can answer ARP requests locally (ARP suppression).
  • Fast, clean moves. When a VM live-migrates, its new leaf advertises the MAC address with a higher sequence number and the old route is withdrawn.
  • Distributed anycast gateway. Every leaf acts as the default gateway for a subnet, with the same IP and MAC address, so a VM's first hop is always its own leaf, wherever it runs.
  • Multihoming. EVPN can attach a server to two leaves without vendor-specific MLAG.

The configuration below shows the essential parts on one leaf of a four-spine fabric: a loopback as VTEP source, the VLAN-to-VNI mapping, eBGP to the spines for the underlay, and an EVPN session over loopbacks for the overlay. It is simplified; real fabrics are generated from templates rather than typed by hand. 

interface Loopback0
   description router ID and EVPN peering
   ip address 192.0.2.11/32
!
interface Loopback1
   description VTEP source
   ip address 192.0.2.111/32
!
interface Vxlan1
   vxlan source-interface Loopback1
   vxlan udp-port 4789
   vxlan vlan 110 vni 10110
!
router bgp 65101
   router-id 192.0.2.11
   maximum-paths 4
   neighbor UNDERLAY peer group
   neighbor UNDERLAY remote-as 65100
   neighbor 198.51.100.0 peer group UNDERLAY
   neighbor 198.51.100.2 peer group UNDERLAY
   neighbor 198.51.100.4 peer group UNDERLAY
   neighbor 198.51.100.6 peer group UNDERLAY
   neighbor EVPN peer group
   neighbor EVPN remote-as 65100
   neighbor EVPN update-source Loopback0
   neighbor EVPN ebgp-multihop 3
   neighbor EVPN send-community extended
   neighbor 192.0.2.1 peer group EVPN
   neighbor 192.0.2.2 peer group EVPN
   neighbor 192.0.2.3 peer group EVPN
   neighbor 192.0.2.4 peer group EVPN
   network 192.0.2.11/32
   network 192.0.2.111/32
   !
   vlan 110
      rd 192.0.2.11:10110
      route-target both 10110:10110
      redistribute learned
   !
   address-family evpn
      neighbor EVPN activate

Read it as two layers. The UNDERLAY sessions run over the four point-to-point links to the spines and advertise the loopbacks, with maximum-paths 4 enabling ECMP. The EVPN sessions run between loopbacks and carry the MAC and IP routes for VLAN 110, tagged with a route target that every leaf serving VNI 10110 imports.

Storage, management and AI traffic

  • Storage on the fabric. iSCSI, NFS, NVMe over TCP and HCI replication run over the same Ethernet fabric, usually in their own VLANs or VNIs. RDMA-based storage (RoCEv2) is sensitive to packet loss and typically needs switch features such as priority flow control and explicit congestion notification, configured consistently end to end. Fibre Channel, by contrast, is a separate network; see SAN Protocols.
  • Management stays out of band. Server management controllers, switch management ports and console servers sit on a separate, simple network, so you can still reach the fabric when the fabric is broken. See Out-of-Band Management.
  • AI clusters have two networks. GPU servers usually connect to a normal front-end fabric for storage and users, and to a separate, non-blocking back-end fabric (InfiniBand or Ethernet with RoCE) for GPU-to-GPU traffic during training, where one slow link delays the whole job.
  • Automation. A fabric of dozens of switches with identical patterns is configured from templates and a source of truth rather than by hand, and changes go through change enablement. See Network Automation.

Kestrel Energy's own data center shows a small version: two spines, a pair of leaf switches in each server rack (25 GbE to general servers, 100 GbE to its HCI nodes) and a border leaf pair toward the firewalls and its colocation site, and a separate 1 GbE management network.