Two special cases of packet loss troubleshooting

Wang jiping

Packet Loss Caused by remote commercial password theft

Zhongtian Design Institute is directly affiliated to the Gansu Provincial Department of Construction and has a small network scale. The 152 hosts are divided into five subnets based on the functional departments of the Organization, and the Hub connects to the switch. Due to the frequent collaboration in the company, in addition to an online video system, a file server is deployed to provide data sharing and exchange for a single subnet. The external Internet requirements of the Organization are not very high. The network topology is shown in Figure 1.
Fault symptom
One day, the network of the unit suddenly experienced a serious blockage, and the data between hosts was frequently interrupted, leading to the inability to work collaboratively and frequent disconnection of online video systems. In addition, both uploading and downloading files from the file server are exceptionally slow and sometimes interrupted due to timeout. The host can connect to the Internet, but the network speed is slow.
Preliminary judgment
First, use the ping command on a host to test the connectivity of the Gateway. Enter the command "ping 192.168.2.1-n 1000" to send 1000 Ping packets to test the gateway. The test result shows that the gateway can be pinged, but packet loss is very serious: 1000 of the 720 packets are lost, the packet loss rate is 72%, and the packet loss duration is long. Run the arp-a command to find that the gateway IP address and the gateway MAC address point correctly. Through the above test, the network settings errors and ARP spoofing are basically ruled out.
Monitoring Analysis
Therefore, the core switch is mirrored and the entire intranet (five subnets) is monitored using Sniffer. First, go to the "dashboard" and find that the network utilization rate has reached 97%, which is abnormal. The author judges that the network size and daily business volume in this unit should be 20% ~ Between 30%, there is a large network redundancy. In this way, we can conclude that the root cause of packet loss is that abnormal traffic occupies a large amount of network bandwidth. Where are these abnormal traffic sources?
Switching to the matrix panel, we found that hosts with MAC 57.87%-06-e6-98-84-b7 account for of the total network traffic. Therefore, the target is initially locked on the host, and then switched to "hosttable" (host list) for further analysis. No large number of broadcast packages were found from this panel, so the impact of broadcast storms was completely ruled out. Find 00-0A-E6-98-84-B7. According to the host analysis, it is found that the network activity of the host is very suspicious. Only 700 packets are sent to the host, the outgoing packets have hundreds of thousands of packets in over 10 minutes.
Troubleshooting
In order to confirm the network activity of the above host, I analyze the packet capture on the switch. After data packets are decoded, the host copies data to a host with an IP address of 60.164.82.185 over the UDP protocol. Why is this IP address so familiar? Isn't it a local IP address? In addition, it is also found that the host is frequently connected to the file server. Based on the network segment and MAC address, the author isolates the host on the switch, disconnects the network, and the entire network will return to normal immediately, troubleshooting packet loss.
At this point, we found the cause of the packet loss through layer-by-layer troubleshooting-the host was planted with a Trojan and then remotely controlled to copy files to the remote through port 8888. In addition, the host is downloading a large number of files from the file server. It is estimated that attackers are using this host to steal information from the folder server.
The host was originally installed with anti-virus software, but the attacker did not report the virus. Manually clear the trojan and connect the host to the network. packet loss never occurs again. After the event, the host may recall a trojan in the mobile hard drive because he copied the project plan to the customer's mobile hard drive on that day. The commercial password theft caused by packet loss troubleshooting is not expected by everyone.

 

Network Topology of Zhongtian Design Institute

 

Packet Loss Caused by automatic cyclic scanning attacks

The LAN of a middle school in my location has about 1000 computers. Generally, there are about 600 computers online at the same time, and the network has been very stable. The Network encountered an exception before the end of the holiday. The specific symptom is: the network communication suddenly interrupted throughout the campus network, and internal users cannot access the Internet normally. During the ping packet test in the data center, it is found that the ping packet from the Data Center client to the management address of the center switch has a long response time and random packet loss occurs, the communication packet loss caused by the client on the second-level switch is more serious.
In-depth analysis
I have preliminarily determined that this phenomenon may be caused by ARP table update, broadcast or route loop faults, and virus attacks. Therefore, it is necessary to obtain ARP information, switch load, original packets transmitted in the network, and other information.
Configure packet capture. Configure the port image on the central switch, and connect the analysis laptop to this port. Then, start the network analysis tool to capture and analyze the network data communication, after about 10 minutes, stop capturing and analyze the captured data packets.
View the connection and locate the attack source. After the capture is stopped, the author found in the node browser on the left of the main interface of the network analysis system that the number of online IP hosts on the internal network reached 6515, which indicates that many counterfeit IP hosts exist in the network, counterfeit IP addresses or automatic scanning attacks may exist in the network. In the connection view, we found that 12108 connections were initiated in the network within 10 minutes, and most of them were in the client request synchronization status. Based on this, we determined that an automatic scan attack exists on the campus network.
Check the connection information in detail and find that most of these connections are initiated by the host 192.168.5.119, that is, the source address of the connection is 192.168.5.119. Select any connection whose source address is 192.168.5.119, right-click the connection, and select "locate browser node> endpoint 1 IP" from the shortcut menu. Then, the node browser automatically locates on the host 192.168.5.119.
Determine the attack method through the Protocol. Select the data packet view to view the original decoding information of the data transmitted at 192.168.5.119. We found that the machine 192.168.5.119 is actively scanning the TCP port 445 of the host on the network, probably because the host at 192.168.5.119 is infected with a virus program, or use scanning software for attacks. By analyzing the Chart view, we can further confirm that the host 192.168.5.119 has an automatic scanning attack.
After finding the root cause of the problem, we isolated the host 192.168.5.119. After some tests, the network packet loss was mitigated, but the problem was not fundamentally solved. Are there any fish that have missed the Internet still making waves? As a result, the network analysis system was started again to capture and analyze the data communication of the network, and three hosts with similar conditions as 192.168.5.119 were found in the network. In this case, we can be sure that 192.168.5.119 and the three newly discovered hosts are infected with a virus, and the virus will actively scan other hosts on the network to enable TCP port 445, if a host opens the port, it will attack and infect the host. This cycle causes the above network faults.
Release fault
Immediately isolate the three newly infected hosts and resume network communication. In addition, during the analysis, the author also found that the host 192.168.101.57 occupies a large amount of traffic, and both the source and destination of its communication data packets use UDP port 6020, the address 227.1.2.7 that communicates with 192.168.101.57 is a multicast IP address. In view of this, we speculate that 192.168.101.57 may be using applications such as online video-on-demand, which consumes network resources. The host is located as a server in the school data center and is configured as an online video server to provide video services for clients, the host is using P2P software to download the video-No wonder there is such a large amount of traffic.
In fact, there are many causes of network packet loss. In addition to the above network attacks and virus infection, connection lines, network adapters, switches, routers, and other hardware faults may also cause network delay and packet loss. Therefore, it is very important for network administrators to master packet loss troubleshooting methods.
We hope that the above ideas on Packet Loss troubleshooting will be helpful to everyone.