Distributed VXLAN Implementation Solution Based on EVPN: Enhancing Data Center Network Scalability and Efficiency

Currently, there are two main ways to implement VXLAN: centralized and distributed. So should we choose a centralized VXLAN solution or a distributed VXLAN solution in the VXLAN deployment solution? In fact, this question is easy to answer. You can choose according to the scale of the data center. As shown in the following table, the best practice for large data centers with more than 5,000 server clusters is the distributed VXLAN solution.

	Centralized VXLAN	Distributed VXLAN
Layer 3 gateway location	Layer 3 gateway is deployed at Spine	Layer 3 gateway is deployed at Leaf
Layer 2 gateway location	Layer 2 gateway is deployed at Leaf	Layer 2 gateway is deployed at Leaf
Traffic forwarding rules	Layer 3 traffic under the same -Leaf needs to detour the Spine	Layer 3 traffic under the same -Leaf does not need to detour the Spine
Table item storage rule	The Spine needs to store the MAC and ARP of the entire network	The Spine does not need to store the MAC and ARP of the entire network
Control Plane	Static or EVPN	EVPN
Operation and Management (O&M)	Simple, because the gateway is at Spine	Complex, because the gateway is at Leaf
Applicable Data center scale	Suitable for small scale networking	Suitable for medium and large scale networking

▲ Table 1 Differences between centralized VXLAN and distributed VXLAN

The centralized approach will have the common problems of traditional large Layer 2 networks, such as broadcast storms. To avoid this problem, the broadcast domain should not be too large, and thus the cluster size should not be too large as well. However, the distributed approach can reduce the broadcast domain to the edge because of Layer 3 along with the control function of EVPN, thereby forming a larger network scale.

Why EVPN is introduce as the control plane?

By comparison, distributed gateways are more suitable for networking with public cloud vendors or medium-to-large-scale private cloud data centers. VXLAN is defined by RFC7348, which only defines the behavior of the forwarding plane and does not specify the VXLAN control plane. In the early days of VXLAN technology, forwarding information was obtained through the forwarding plane, which was relatively simple to implement. However, as the scale of the network grows, relying entirely on the forwarding plane for control will cause broadcast and multicast storms in the network, and bring great problems to operation and maintenance (O&M). Therefore, VXLAN also needs a control plane. Before understanding EVPN, let's take a look at the comparison of several VXLAN control planes, as shown in the following table:

	Static mode	IS-IS mode	EVPN mode	Controller mode
Tunnel Construction method	Adopts a static configuration approach	ENDP(Enhanced Neighbor Discovery Protocol) must be configured on VTEPs to discover VTEPs and establish tunnels	The extended BGP attribute enables neighbor discovery, automatically discovers VTEPs in VXLAN networks, and automatically creates VXLAN tunnels between VTEPs	Controller System-controller
Tunnel establishment method between VTEPs	Adopts a static configuration approach	IS-IS protocol extension mode	VXLAN tunnels are automatically established through BGP extended attribute associations	Controller System-controller
Address learning	Learns MAC address flooding	IS-IS protocol is implemented to learn remote addresses	Implementation of extended MP-BGP protocol	First packet reported, The forwarding table item is issued by the Controller
Standardization	No control plane	Private protocol, extend IS-IS packets	The control protocol of EVPN has been standardized, see RFC 7348RFC7209, RFC7432	VTEP and Controller are based on the standard protocol framework Netconf/0penFlow, but each manufacturer has its own private definition, so they are not compatible

▲ Table 2 VXLAN control plane technology comparison

From the above table, we can see the benefits of using EVPN as the control plane:

1. Routing Protocols

For medium and large data centers, most of the routing protocols for underlay networks have adopted BGP, and EVPN is implemented through the extended protocol of BGP. Therefore, only one set of routing protocols needs to be maintained, which has more O&M advantages.
2. Standardization

Compared with other control plane technologies, only EVPN has a standardized definition, which can achieve network heterogeneity and facilitate network compatibility and expansion.

Implementation Solution Based on EVPN+VXLAN

Next, let's take a look at the specific implementation solution based on EVPN+VXLAN. The network architecture is shown in the following figure:

▲ Figure 1 Physical network architecture

Type 3 routing - tunnel establishment
Based on EVPN-Type3 routing - Inclusive Multicast routing1

Automatically discover VTEPs (VXLAN Tunnel End Points, which are the endpoints of VXLAN tunnels) in the VXLAN network, create VXLAN tunnels between VTEPs, and automatically associate VNIs with VXLAN tunnels.

1This type of routing is mainly used for automatic discovery of VTEPs and dynamic establishment of VXLAN tunnels in the VXLAN control plane. VTEPs that are BGP EVPN peers transmit Layer 2 VNI and VTEP IP address information to each other through Inclusive Multicast routing. In particular, the Originating Router's IP Address field is the local VTEP IP address. If the peer VTEP IP address is reachable by Layer 3 routing, a VXLAN tunnel to the peer is established.

EVPN-Type2 Routing ——MAC and Routing Synchronization
Based on EVPN-Type2 routing - MAC/IP routing2, MAC and routing synchronization in the network can be automatically completed.

2The main functions of this type of routing are: host MAC address notification, host ARP notification, and host IP route notification.

EVPN-Type5 Routing —— Border Routing Access

In actual network deployment, we can choose either a Spine as the Border or a group of Leaf nodes as the Border, depending on the traffic in our actual network. If the traffic is mainly north-south, generally a group of Leaf nodes can be used as the Border. However, in some scenarios, such as hybrid cloud, east-west traffic still needs to be forwarded by the Border, so we recommend using Spine as the Border.

In this scenario where Underlay and Overlay are interconnected, a relatively important routing type of EVPN will be used - segment routing, which is transmitted through EVPN-Type5 routing - IP prefix routing. When the external route enters the Border, the Border will be synchronized to the VXLAN network through Type5 routing, thereby enabling hosts in the VXLAN network to access the external network.

Optimization of Distributed VXLAN Solution Deployment

1. Optimization point 1: ARP flood suppression
When the scale of virtual machines in the network expands, ARP broadcast will consume network bandwidth and there will be a risk of broadcast storm. In order to reduce the impact of ARP broadcast, we can suppress ARP flooding through ARP proxy.

ARP proxy implementation: After the gateway turns on the ARP proxy, the host learns that the remote ARP is the MAC address of the gateway. It has the following advantages:

● Reduce the broadcast domain and suppress broadcast storms;

● Increased cluster size: Since the ARP proxy is enabled, the Layer 2 broadcast becomes a Layer 3 route, so each VTEP only needs to maintain its own host MAC and can turn off the remote MAC synchronization function, thereby saving MAC table space and providing support for a larger cluster size.

However, it may not be applicable in some scenarios. For example, the DR mode of LVS requires LVS to learn the real MAC address of the server, so it cannot be replaced by the gateway MAC. In this scenario, we can disable the ARP proxy function of the VNI where the LVS is located to ensure normal communication in the DR mode.

2. Optimization point 2: ARP-TO-HOST

▲ Figure 2 Service model

As shown in the figure above, assuming that VM1-4 belongs to the same subnet, this situation may occur due to business deployment or virtual machine migration. If VETP1 and VTEP4 publish network segment routes, due to the existence of ECMP, some traffic to VM1 will bypass VTEP1->VTEP4.

In order to solve this problem, detailed routing or host routing is needed. If you want to configure it manually, the workload is large and it is easy to make mistakes. Currently, a common method is to automatically generate host routing through ARP and publish it to the network.

Implementation method: After the gateway turns on ARP-TO-HOST, the host route is automatically generated according to the ARP table and automatically published to the network.

Summary

This article explains the implementation solution based on EVPN+VXLAN from the aspects of why distributed VXLAN is used, why EVPN is used as the control plane of VXLAN, the EVPN+VXLAN implementation solution and solution deployment optimization, which is also the recommended solution for Ruijie Network's virtualized data center network architecture. Welcome to leave a message for further communication.

The Rise of Multi-Gigabit Switches: A Deep Dive

What Is an Ethernet Switch?

Education

5-Star Hotel Solution

Technology Feast | Distributed VXLAN Implementation Solution Based on EVPN

Why adopt a distributed VXLAN solution?

Why EVPN is introduce as the control plane?

Implementation Solution Based on EVPN+VXLAN

Optimization of Distributed VXLAN Solution Deployment

Summary

Featured blogs

2024 Ruijie Networks Brand Awareness Survey