VXLAN Basics — VNI, VTEP, and Why Modern DCs Use It

The one-sentence mental model

VXLAN takes an Ethernet frame, wraps it in a UDP-in-IP packet with a 24-bit “tenant ID” (VNI), and ships it across a routed underlay to a peer switch, which unwraps it back to Ethernet. From the host’s point of view, two servers on the same VNI are on the same LAN. From the network’s point of view, they’re just IP packets riding on a routed fabric.

Why we needed it

Classic Layer-2 data centers had three problems:

1. STP blocks half your links. In a Clos / spine-leaf topology with multiple uplinks, STP puts most of them in blocking state. Wasted capacity.
2. VLANs cap at 4094. With hundreds of tenants, containers, and micro-services, 12 bits of VLAN ID runs out.
3. Layer 2 doesn’t scale past a couple of racks without loops or huge broadcast domains.

VXLAN solves all three by running Layer 2 as a tunnel over a routed IP network. STP still exists locally at each leaf, but between leaves it’s all Layer 3 — ECMP takes over, every link is active, and the “tenant space” jumps from 4K VLANs to 16M VNIs.

The encapsulation

Original Ethernet frame → gets wrapped in:

Outer Ethernet | Outer IP | Outer UDP (dst 4789) | VXLAN header (VNI) | Original inner Ethernet frame | Outer CRC

Key fields:

• Outer UDP destination port = 4789 (IANA-assigned).
• VXLAN header = 8 bytes, including the 24-bit VNI.
• Outer source/dest IP = the VTEP addresses.

The added overhead is ~50 bytes — plan MTU accordingly (jumbo frames highly recommended: MTU 9000 in the underlay, 1500 in the overlay).

The two roles

VTEP — VXLAN Tunnel Endpoint

The switch (or hypervisor vSwitch) that does the encap/decap. Every leaf in a spine-leaf fabric is a VTEP. A VTEP has a loopback IP used as the tunnel source/destination.

Spine

Doesn’t do VXLAN encap. Just routes IP packets between leaves. Spines don’t even need to know VXLAN exists — they just carry underlay traffic.

Two learning modes

Flood-and-learn (classic, multicast-based)

• BUMs (Broadcast / Unknown-unicast / Multicast) flood to a multicast group per VNI in the underlay.
• Every VTEP joins the multicast group for VNIs it has interest in.
• MAC addresses learned via observing traffic in the overlay.
• Requires multicast in the underlay. Doesn’t scale beautifully.

BGP EVPN (modern control plane)

• MP-BGP with the EVPN address family exchanges MAC → VTEP mappings explicitly.
• No multicast in the underlay required (use ingress replication or MP-BGP-signaled multicast).
• Fast, deterministic, and pairs naturally with VRFs for multi-tenant.
• The Cisco default in modern DC platforms (Nexus 9K running NX-OS, Catalyst 9500 in fabric mode).

CCNP ENCOR expects you to know both exist and why EVPN is preferred in new builds.

Sample verification (Nexus / Cat 9K in EVPN mode)

show nve peers                        ← which VTEPs am I talking to
show nve vni                          ← which VNIs are up locally
show l2route evpn mac vni 10001       ← MAC table for a specific VNI
show bgp l2vpn evpn summary           ← BGP EVPN neighbor state

Common exam / real-world mistakes

1. Confusing VLAN with VNI. VLANs are still used locally on each leaf to bind physical ports into a bridge domain. That local VLAN maps to a global VNI. VNI is not “the new VLAN” — it’s the fabric-wide identifier.
2. Underlay MTU too small. VXLAN adds ~50 bytes. If the underlay MTU is 1500 and you send a 1500-byte inner frame, the outer packet is 1550 → fragments or drops.
3. Ignoring the VTEP loopback design. VTEP source IP is a loopback advertised in the underlay routing protocol. If OSPF/IS-IS doesn’t advertise it correctly, EVPN peerings never come up.
4. Assuming multicast is required. Only for flood-and-learn. Ingress replication (head-end replication) is a common alternative when the underlay doesn’t have multicast.
5. Forgetting anycast gateway. In EVPN, every leaf shares the same gateway IP and MAC for a given VNI — hosts always talk to their local leaf. If you don’t configure anycast gateway, all inter-VNI traffic hairpins through one leaf.

Cheat strip

Underlay    routed IP fabric (OSPF or IS-IS). ECMP everywhere.
Overlay     VXLAN tunnels between VTEPs (leaves).
VNI         24-bit fabric-wide tenant ID. 16M values.
VTEP        the leaf that encaps/decaps. Source = loopback.
UDP dst     4789 (VXLAN).
Overhead    ~50 bytes. Jumbo underlay MTU (9000).

Learn       flood-and-learn (multicast)  |  BGP EVPN (modern, preferred)
Anycast GW  same VIP/MAC on every leaf → hosts talk to local leaf
Verify (NX) show nve peers | show nve vni | show bgp l2vpn evpn summary