Frequent switch interruptions (1)

As a very important transport hub in a LAN, a switch directly determines the stability of the LAN. Generally, when the quality of the switch is relatively good, the switch equipment is rarely prone to failures. Of course, this does not mean that the switch equipment can always work stably, with the passage of its "service" time, the switch's working stability will also continue to decline, in addition to the aging of its internal component performance, it may also be caused by a low version of the switch system. This is not the case. The frequent switch interruption and fault that I have encountered is caused by the low version of the switch system! Considering that the probability of network faults caused by such factors is not very high, it may take some detours to solve the problem. Therefore, this article will now restore the specific troubleshooting process of the fault, for your reference!

Fault symptom

In the recent period, switches at a certain aggregation layer in the LAN frequently experienced network interruptions. When such a fault occurs, the author needs to arrive at the fault site, manually disconnect the power supply to restart the switch device, or remotely log on to the core device that is connected to the faulty switch, and restart the downstream port that is connected to the switch once, in order to restore the network to normal. The target Faulty Switch is usually connected to two virtual working subnets. The daily network transmission traffic of these two working subnets is not very large, even if it is not during peak hours, the Faulty Switch may also be interrupted inexplicably. Therefore, I have ruled out the factors that cause abnormal operation of the faulty switch due to excessive network traffic, and also ruled out the possibility of Network viruses.

Troubleshooting

Considering that the faulty switch is connected to the uplink core device through a broadband optical fiber, I am worried about the stability of the broadband optical fiber line, therefore, the local telecom technical staff were specially invited to use professional tools to test the broadband optical fiber line. After multiple tests, the working status of the broadband optical fiber line was proved to be normal. With no clue, I accidentally found a thick layer of dust covering the case of the faulty switch, at this time, I thought that the faulty switch has been in service for nearly four years, and the vswitch's background management system version is relatively low, the traditional command line is still used, and a network fault that has occurred in the LAN is related to the BUG of the switch system, is this frequent network interruption failure caused by a low version of the switch system? In order to verify whether my analysis is correct, I immediately use the telnet command to remotely log on to the background management interface of the faulty switch system and run the "dis cpu" string command at the command line prompt on the interface, it is found that the system CPU usage of the vswitch is always above 95%, which is obviously abnormal, because under normal working conditions, the CPU resource consumption rate of the switch device should be below 50%, the response capability of the switch exceeds this value will be significantly reduced. Later, the author executed the string command "dis ver ", from the results returned later, I found that the VRP platform software version used by the faulty switch is relatively low. Is the fault mentioned in this article really caused by the low version of the switch system software?

Troubleshooting

Considering that there is no problem with the detailed check of the physical line connecting the switch, when the switch experiences a network interruption fault, the author simply restarts the faulty switch device or corresponding connection port, and the working status of the faulty switch can be restored to normal in a short time, this indicates that the fault is indeed related to the switch itself. In order to eliminate the possibility of a low vswitch system software version, the author intends to update the VRP platform software of the faulty vswitch to the latest version.

When upgrading the vswitch system online, I first checked the specific model of the faulty vswitch, and then went to the official website of the corresponding brand product, download the latest version of the VRP upgrade file and the Bootrom upgrade file. For ease of operation, I chose the FTP method for upgrade, that is to say, the local normal workstation that saves the VRP upgrade file and Bootrom upgrade file will be used as the FTP server, and The Faulty Switch will be used as the FTP Client System. The advantage of this operation is that the steps are simple, you do not need to perform any complex configuration operations on the switch device;

The following code uses the FTP command in the vswitch system to connect to the FTP server that saves the VRP upgrade file and the Bootrom upgrade file to download the upgrade package file. Of course, before downloading the upgrade package file, the author first configures the FTP server so that it is in the same working subnet as The Faulty Switch to ensure smooth access between the switch and the FTP server. At the same time, the author also directly saves the VRP upgrade file and Bootrom upgrade file to the Home Directory of the FTP server. This way, after the switch system successfully establishes a connection with the FTP server through the FTP command, you can directly view the required update package file. In addition, for convenience of memory, I will save the downloaded VRP Update file as aaa. bin, replace the Bootrom upgrade file with bbb. btm.

After downloading and saving the upgrade files aaa. bin and bbb. btm to the Flash cache of the switch, I can officially start the online upgrade of the switch system. Of course, for the sake of security, the author backs up the old configuration file of the target switch to prevent exceptions during the upgrade process, but cannot restore the switch's working status; after the old configuration file is backed up, the author immediately runs the string command "boot aaa. bin ", after the command is successfully executed, I restarted the switch system again, during which the switch will automatically call aaa. binfile, so that the VRP platform software of the switch can be successfully upgraded to the latest version. Of course, this operation can also be completed through remote login;

Next, we need to connect to the vswitch through the Console to complete the upgrade of the Bootrom file locally, because after the VRP platform is updated, some configuration commands on the new platform are somewhat different from those on the old platform, at this time, the switch is often unable to be managed through the network. Follow the same operation method, we then execute the string command "boot bbb. btm ", and then restart the switch system, so that the switch upgrade operation is successful. At this time, I tried to use the "dis ver" string command to observe the status of the system version of the switch, and found that the system has been upgraded to the latest version.

After confirming that the switch is successfully upgraded, the author re-updates the Internet parameters of the switch based on the old configuration, and restores the switch to normal working status; after a period of practical testing, I found that this switch has never experienced any network interruptions since then, and I have continuously checked it, it is found that the CPU consumption rate of the upgraded vswitch is always around 25%, which indicates that the running performance of the upgraded vswitch is still very stable.