Several modes in Linux multi-Nic bond

1. purpose of writing this document is mainly to introduce and sort out several modes in bond, and to facilitate future work viewing. II. document content with the increasing innovations of science and technology, data security has gradually reflected its importance. It can be imagined that when a person belongs to all people... information & n

I. purpose of the document
This document is mainly used to introduce and sort out several modes in bond, and to facilitate future work and viewing.

II. document content
With the increasing innovation of science and technology, data security has gradually reflected its importance. It can be imagined that it is terrible when all personal data of a person does not exist. With the in-depth use of network technology, network transmission of data has become an important and even a major transmission method. Therefore, the normal provision of network services by data servers is a concern for all vendors.

In this context, the application of a single network card has been stretched, and the popularization of equipment redundancy technology has become increasingly prevalent. This article will introduce the multi-Nic bonding technology in Linux to describe this fault tolerance concept.

Server load balancer is another feature of network card bonding. it can work with multiple network cards at the same time to improve the throughput of system network processing.

1. Nic load balancing mode (mode = BOND_MODE_ROUNDROBIN)

 

1) create a bond virtual device

 

Create a device for a ifcfg-bond0 and configure the following information.

 

DEVICE = bond0

 

BOOTPROTO = static

 

IPADDR = 172.16.64.208

 

NETMASK = 255.255.255.224.0

 

ONBOOT = yes

 

TYPE = Ethernet

 

2) configuration interface file

 

Because a virtual ip address is used, ip information is not configured for other interface devices.

 

3) configure bonding

 

Open the/etc/modprobe. conf file and configure the bonding working mode as follows.

 

Alias bond0bonding

 

Optionsbond0 mode = 0 arp_interval = 500arp_ip_target = 172.16.64.86

 

4) start bonding

 

You need to add a route to customize the sending rules. you can add a route as needed. Restart the device after configuration.

 

Ifenslave bond0eth0 eth1

 

Route add-net0/0 gw 172.16.64.254

 

Bond0 Link encap: Ethernet HWaddr 00: 14: 10: 70: 00: 25

 

Inet addr: 172.16.64.208 Bcast: 172.16.95.255 Mask: 255.255.224.0

 

Inet6 addr: fe80: 200: ff: fe00: 0/64 Scope: Link

 

Up broadcast running mastermulticast mtu: 1500 Metric: 1

 

RX packets: 39335 errors: 0 dropped: 0 overruns: 0 frame: 0

 

TX packets: 3178 errors: 0 dropped: 0 overruns: 0 carrier: 0

 

Collisions: 0 txqueuelen: 0

 

RX bytes: 3020656 (2.8 MiB) TX bytes: 269722 (263.4 KiB)

 

Eth0 Link encap: Ethernet HWaddr 00: 14: 10: 70: 00: 25

 

Up broadcast running slavemulticast mtu: 1500 Metric: 1

 

RX packets: 18208 errors: 0 dropped: 0 overruns: 0 frame: 0

 

TX packets: 5 errors: 0 dropped: 0 overruns: 0 carrier: 0

 

Collisions: 0 fig: 1000

 

RX bytes: 1371589 (1.3 MiB) TX bytes: 378 (378.0 B)

 

Eth1 Link encap: Ethernet HWaddr 00: 14: 10: 70: 00: 25

 

Up broadcast running slave multicast mtu: 1500 Metric: 1

 

RX packets: 21128 errors: 0 dropped: 0 overruns: 0 frame: 0

 

TX packets: 3174 errors: 0 dropped: 0 overruns: 0 carrier: 0

 

Collisions: 0 fig: 100

 

RX bytes: 1649127 (1.5 MiB) TX bytes: 269498 (263.1 KiB)

 

As shown in, in this mode, the bonding module sets the MAC address of the virtual interface and all slave interfaces to the same. Through the timer, each slave interface continuously sends ARP packets to constantly change the relationship between the switch port and MAC.

 

In this way, every Nic is working. The ARP sending rule is:

 

Each arp_interval (MS) interval sends arp requests to arp_ip_target. You can also send arp requests to multiple arp_ip_targets.

 

Observe the learned MAC address on the vSwitch port and find that the MAC will be switched over and over again on the two ports.

 

In BOND_MODE_ROUNDROBIN mode, the processing logic of bonding for sending and receiving data is inconsistent, and bonding basically does not process the data, the switch port and MAC are used to receive data alternately. If it is sent, the switch learns the relationship between the port and MAC based on the source MAC of the data, so what bonding does is to select a different Nic for sending.

 

For data transmission,

 

Static inline voidbond_set_mode_ops (struct net_device * bond_dev, int mode)

 

{

 

Switch (mode ){

 

Case BOND_MODE_ROUNDROBIN:

 

Bond_dev-> hard_start_xmit = bond_xmit_roundrobin;

 

Break;

 

...

 

The sending function of bond is registered as bond_xmit_roundrobin. Through the implementation of bond_xmit_roundrobin, we can find that

 

Static int bond_xmit_roundrobin (structsk_buff * skb, struct net_device * bond_dev)

 

{

 

Read_lock (& bond-> curr_slave_lock );

 

Slave = start_at = bond-> curr_active_slave;

 

Read_unlock (& bond-> curr_slave_lock );

 

Bond_for_each_slave_from (bond, slave, I, start_at ){

 

If (IS_UP (slave-> dev )&&

 

(Slave-> link = BOND_LINK_UP )&&

 

(Slave-> state = BOND_STATE_ACTIVE )){

 

Res = bond_dev_queue_xmit (bond, skb, slave-> dev );

 

Write_lock (& bond-> curr_slave_lock );

 

Bond-> curr_active_slave = slave-> next;

 

Write_unlock (& bond-> curr_slave_lock );

 

Break;

 

}

 

Bond_xmit_roundrobin will be sent through the device pointed to by the curr_active_slave pointer. of course, curr_active_slave will point to the next slave device after calling bond_dev_queue_xmit to complete the actual sending. Bond_dev_queue_xmit is actually implemented by calling the universal sending function dev_queue_xmit. it passes to dev_queue_xmit as a skb. before passing, skb-> dev is specified as the current slave device, in this way, the kernel will find the corresponding real Nic device for sending, and finally round robin switching of the curr_active_slave pointer, realizing the bonding load balancing working mode.

 

In this mode, we can see that bonding implements a module similar to the NIC driver. the corresponding bond0 device is a pure virtual device. although data transmission passes through it, it passes through a series of calls, after turning around, the system will return to the real Nic device for sending, which will undoubtedly consume a certain amount of system performance.

 

The BOND_MODE_ROUNDROBIN mode is tested with a UDP packet of Mbps.

 

During the test, it is found that the receiving end has many unordered packets. observe the switch port status and the switching frequency between ports is irregular. this is closely related to the configuration or performance of the switch, further research is required if necessary. Whether the correctness and timing of data can be ensured requires further careful tests.

 

2. fault tolerance mode of the network adapter (mode = BOND_MODE_ACTIVEBACKUP)

 

The configuration method of the fault tolerance mode is similar to that of the load balancing mode. you just need to modify/etc/modprobe. conf.

 

Alias bond0 bonding

 

Options bond0 mode = 1 miimon = 100

 

The mii link detection method is used here, which is different from the previous arp detection method. Of course these two link detection methods can be used in various modes, but note that they cannot be used at the same time.

 

This section describes how to implement bonding mii detection. First, like arp-monitor, mii is also triggered by a timer.

 

If (bond-> params. miimon) {/* link checkinterval, in milliseconds .*/

 

Init_timer (mii_timer );

 

Mii_timer-> expires = jiffies + 1;

 

Mii_timer-> data = (unsigned long) bond_dev;

 

Mii_timer-> function = (void *) & bond_mii_monitor;

 

Add_timer (mii_timer );

 

}

 

The essence of bond_mii_monitor function is to detect the link status of the NIC. bonding defines that the NIC has four link statuses:

 

BOND_LINK_UP: normal (the NIC in this status is a candidate for potential packet sending)

 

BOND_LINK_FAIL: Nic fault. switch to BOND_LINK_DOWN

 

BOND_LINK_DOWN: Invalid

 

BOND_LINK_BACK: Nic recovery, switching to BOND_LINK_UP

 

From top to bottom, it indicates that the Nic link is in the normal, failed, and restored state. The bond_mii_monitor function checks whether the NIC's link status is in these statuses, and then marks the do_failover variable to indicate whether to switch the slave Nic. The code is too long, but the logic is still very clear, so it is not listed here.

 

In BOND_MODE_ACTIVEBACKUP mode, one of the two NICs does not work and is set to the IFF_NOARP status. At the same time, the MAC addresses of bond virtual devices and slave devices are the same, so this Nic will not be noticed by the outside world. The switch does not want to send packets to this port. When bond's mii detection finds that the current active device is invalid, it will switch to this backup device.

 

In the bond_change_active_slave function

 

If (bond-> params. mode = BOND_MODE_ACTIVEBACKUP ){

 

If (old_active ){

 

Bond_set_slave_inactive_flags (old_active );

 

}

 

If (new_active ){

 

Bond_set_slave_active_flags (new_active );

 

}

 

}

 

This is the switch logic in BOND_MODE_ACTIVEBACKUP mode. it is very simple to note that in bond_set_slave_inactive_flags (old_active), you need to set the interface status to IFF_NOARP, otherwise, the switch may send data packets to an incorrect port.

 

The data transmission in BOND_MODE_ACTIVEBACKUP mode is very simple. you can imagine that the curr_active_slave pointer always points to the currently available device, so you can directly send it without switching the slave device in BOND_MODE_ROUNDROBIN mode.

 

3. Nic virtualization mode (mode = BOND_MODE_ALB)

 

After investigation, many disk array devices use Nic virtualization for multi-Nic applications. It is actually the Linux bonding technology. It adopts mode = BOND_MODE_ALB.

 

The implementation method of BOND_MODE_ALB is complicated. if you do not understand it, you will not perform in-depth analysis. The core idea can be explained as follows:

 

 

From this figure, we can see that the transmission between client A and server is MACA, while that between client B and server is mac B, but for the IP layer, both client A and client B are associated with server X. x. x. the IP address of X is not aware of the changes in the underlying transmission interface. In this way, the server's overall working mode for the client is in the load balancing status. Of course, this process is controlled by the bonding module.

 

The test shows that the arp tables displayed by the two clients are consistent, as shown below (the server address is 172.16.64.208, the MAC address of the server's preferred interface is 00-14-10-70-00-25)

 

CLIENT

 

172.16.64.208 00-14-10-70-00-25 dynamic

 

CLIENT B

 

172.16.64.208 00-14-10-70-00-25 dynamic

 

Packet capture shows that the source MAC addresses of the two ARP responses are different

 

00-14-10-70-00-25 and

 

00-14-10-70-00-26.

 

Therefore, the reason for this solution is that bonding modifies the source address of the ARP response, so that the outside world does not perceive the status of multiple NICs on the server, the unique ing between IP and MAC is determined on the network, ensuring the logical consistency of upper-layer service transmission. At the same time, internal processing is handed over to different NICs to achieve load balancing.

 

In addition, it also gives us some fault tolerance. When an interface fails, bonding will quickly set another Nic as the preferred slave device and modify its MAC address. As the MAC address of the server learned from the outside world remains unchanged, the link status will not be greatly affected.

 

 

 

 

 

 

 

There are 7 Bonding modes:

# DefineBOND_MODE_ROUNDROBIN 0 (balance-rr mode) Nic load balancing mode

# DefineBOND_MODE_ACTIVEBACKUP 1 (active-backup mode) fault tolerance mode of the NIC

# DefineBOND_MODE_XOR 2 (balance-xor mode) requires switch support

# DefineBOND_MODE_BROADCAST 3 (broadcast mode)

# DefineBOND_MODE_8023AD 4 (IEEE 802.3ad dynamic link aggregation mode) requires switch support

# DefineBOND_MODE_TLB 5 adaptive transmission load balancing mode

# DefineBOND_MODE_ALB 6 Nic virtualization mode

The bonding module can work in two modes: multi-master mode and master-slave mode, balance-rr and broadcast belong to multi-master mode while active-backup belongs to master-slave mode. (Balance-xor, balance-tlb and balance-alb) are also in multi-master mode, IEEE 802.3ad dynamic link aggregation mode (802.3ad) is a master-slave mode.