Open-source slb lvs error diagnosis case Column Analysis

Open-source slb lvs error diagnosis case Column Analysis
In response to national calls and advocating security, autonomy and controllability of IT information system hardware and software facilities, our company is also striving to promote localization and open source of IT software and hardware. Therefore, the open source Server Load balancer software LVS is introduced and multiple LVS systems are deployed in production. A strange problem was found during use. After some LVS systems are configured to start randomly, when the operating system is restarted, the service can be started normally with the server, but one of the systems cannot start the service with the operating system. To solve this problem, we have the following content.
I. symptom
The specific LVS faults are as follows:
LVS has been configured as a system service and set to 2345 boot, but after the server is restarted, the service is not started as expected. By viewing the system's diagnostic log tail-f/var/log/messages, it is found that ipvs frequently fl logs to the messages file.
The error message is as follows: Keepalived_healthcheckers: S: Can't initialize IPVS: Protocol not available. After you manually execute ipvsadm-Ln, the service returns to normal.
Ii. analysis process:
1. The diagnosis log prompts us to confirm that the root cause of this problem is that the ipvs module is not properly loaded during system restart.
2. Run the following command lsmod | grep ip_vs to verify that the module is not loaded into the kernel.
So why didn't the ipvs module be loaded into the linux kernel?
Iii. Specific debugging ideas:
1. Test whether to restart keepalived. Only restart ipvsadm to check whether it can be started. The problem persists after restart, indicating that the problem is not in keepalived;
2. Replace ipvsadm of a later version with 1.25. the problem persists after the test;
3. I started to suspect that the installation sequence was the cause. Therefore, both ipvsadm versions test the different installation sequence from keepalived. the problem persists when combined with the startup sequence;
4. Use the/etc/init. d/ipvsadm start command to start the ipvs service and restart the operating system again. The ipvsadm is successfully pulled up by the system and the snapshot is saved immediately.
5. Check the startup script of ipvsadm. The script is complex and involves many files. A configuration file/etc/sysconfig/ipvsadm is found. It suddenly came to the idea that there was a problem with the configuration file. After searching for the ipvsadm file, a problem is found, and a configuration file is missing in/etc/sysconfig;
6. manually create the ipvsadm configuration file touch ipvsadm and restart the system. The ipvsadm service is successfully started.
Iv. Best Solution
After the LVS is deployed and configured, run the service ipvsadm save command.
5. cause of the problem
The deployment personnel did not follow the deployment manual to perform missing steps. As a result, the random Automatic startup of ipvs is dependent on the relevant configuration files.