KEEALIVED+LVS Maintenance skills and tuning

"Troubleshooting Keealived+lvs Load Balancing Deployment Problem"

1, iptables firewall problem

[Directort and Real Server]

Iptalbes firewall will cause the Clinet user, Director and RS Real server 3 between the two any connection between the problem, during the deployment process, we'd better configure the firewall to allow all the IP in the LAN to each other access. That is, if the intranet is interoperability, if it is to study the test environment, the proposed iptables firewall off the test, directors between the failover switch will also be affected by the firewall, which led to the LVS director on both sides of the launch of the VIP split brain

2. Linux Load balancer forwarding problem [Director]

for NAT mode, the Linux kernel forwarding and iptables forwarding functions need to be developed, for Dr Mode load balancing, the forwarding mode can not be opened

3, the RS side suppress ARP problem [Real Server]

This is often forgotten, and it's important to note that every RS side is going to do it, and we're used to doing it through a script, for example: LVS

4, RS terminal lo binding business VIP, multi-service VIP, each RS to bind [Real Server]

This is often forgotten, and it's important to Note that each RS side is engaged, and we're used to doing it through a script, such as: ipvs_client, especially when an RS is forwarded by multiple load balancers, with multiple VIPs

5, RS terminal lo bound business VIP, sometimes lo interface VIP will be lost, such as after the execution of/etc/init.d/network restart

Workaround:

1. LVs Full script Start command put into/etc/local

2, the RS terminal lo bound business VIP made network card configuration files to provide services such as/etc/sysconfig/network-scripts/ifcfg-lo:181

3. Modify the/etc/init.d/network script to start the LVs script when restarting the network

6, ipvsadm-l-n Check LVS Real server situation and VIP connection and configuration

7, check keepalived configuration file (master-slave different)

Single-instance master-slave mode difference: master-slave configuration file difference is (route_id,state,priorities)

8. View the System log/var/log/messages

For LVS load balancing, System log/var/log/messages is important to keep the habit of viewing logs at all times

9. Tracking packet flow through tcpdump

Tcpdump is very powerful and can be used in all kinds of situations, and here is no exception. Example: monitoring data flow information from a director at RS

Tcpdump-nnn-i eth1-s 10000-a Host 192.168.1.1 and Port 80

Tcpdump-i Eth0-nn ' Port 52114 and src host 192.168.1.1 '

10, pay attention to the LVS configuration syntax, especially the braces problem, to appear in pairs, you can write a script to determine whether curly braces are paired

"Keepalived+lvs production environment load balancing maintenance idea"

For a load balancer with dozens of instances keepalived.conf maintenance is really not an easy thing to do.

1. Peak business hours try not to modify the load balancer configuration to prevent failures affecting the user experience

2. When modifying keepalived.conf, perform temporary backup operation/BIN/CP keepalived.conf keepalived.conf.20160610

3, modify the keepalived.conf, the two negative equalizer renamed the name, and then download down, retain the original configuration, and then copy the new two configuration, changed a good copy, and then compare the tool to compare and modify another copy, online with Vimdiff can also

4, after replacing the official configuration file, the main load balancer can perform/etc/keepalived stop, cut the business to the backup load balancer, or directly perform the/etc/init.d/keepalived restart restart directly let the main take effect, if you find an exception, Stop immediately, or restore with a prepared command, for example:

Restart immediately check whether the business is normal, browser access or Linux under the wget business address, if you find the problem immediately with a prepared recovery command recovery, or stop keepalived, cut the business to the unmodified standby, and then view the primary negative server configuration

/etc/init.d/keepalived restart

wget http://www.wsyht.com (check not on load balancer)

Pre-prepared restore command

/BIN/CP KEEPALIVED.CONF/BIN/CP keepalived.conf.new

/BIN/CP keepalived.conf.2016.06.19 keepalived.conf

/etc/init.d/keepalived restart

After the primary negative equalizer is adjusted, the backup load balancer directly executes the/etc/init.d/keepalived restart restart directly, if necessary deliberately over the switch separately under test, so as not to prevent the configuration is not caused by the automatic switch after the exception

"LVs Performance Worry"

1, switch off the hardware firewall when iptables

Iptables is a performance bottleneck when it comes to high traffic, turn off or switch hardware firewalls

2. Kernel optimization

3, Network card optimization

Increase the network throughput capacity of LVS host, and improve the processing speed and capability of LVS.

1, using a faster network card, such as the use of gigabit, million gigabit network card

2, can further bind two or more network card (multi-block network card binding to be verified)

4. TCP/IP optimization

Net.core.netdev_max_backlog = 65000

5. Hardware optimization

Ipvs run, the server resources used are mainly CPU, memory I/O, network I/o;ipvs fully run in memory, and run on the kernel

When the Ipvs application in Dr Mode, that is, no CPU, no consumption of I/O, running very fast, so the system load is very low, so the configuration requirements of the LVS application server is very low, the LVS is very important, so the configuration of a fairly high-end server, is a waste, as long as the server stability, such as:

Production Environment load Balancing cluster system architecture equipment purchase case

Options are based on: Price, performance, redundancy

6, increase the hash table, tune to Ip_vs_conn_tab_bits to 20

By rebooting, passing parameters to the kernel to adjust, on the boot program's kernel-related configuration line, add: ip_vs_conn_tab_bits=20, then, restart

This article is from the "Wsyht blog" blog, make sure to keep this source http://wsyht2015.blog.51cto.com/9014030/1790731

KEEALIVED+LVS Maintenance skills and tuning