How to solve the problem of full bandwidth caused by linux virus recording

How to solve the problem of full bandwidth caused by linux virus recording

Case Description

In the morning, I received a call from the IDC, saying that an IP address of our network segment is constantly sending packets to the outside. It should be attacked. The specific IP address is unknown. Let's check it.

Rational Analysis and Solutions

First, we need to determine which machine's Nic is sending packets to the outside. Fortunately, we have zabbix monitoring on our side. I checked one server and found that one server was full of traffic, the problem should occur on this machine.

I logged on to the machine and checked the network card Traffic. My God, I ran this multi-traffic.

This machine runs a tomcat WEB service and an oracle database. The problem should not occur on the WEB service or database. I checked the WEB log and did not find any exceptions, it is also normal to view the database, and there are no error logs. You can view the system logs and see no exceptions. However, the system logon logs are cleared, I quickly checked the current running processes to see if there were any abnormal processes. I found several abnormal processes, but I couldn't really see them without looking at them carefully, these processes are not normal.

What process is this? Every time I change my ps-ef, process Number 1 is changing. I want to see what files the process has opened, so I can't start it for a moment, when I think of this, I suddenly realized that this should be all some sub-processes managed by a main process, so it is useless to look at these sub-processes. Even if I kill them, there will be a new generation, the thief first captured the king. Let's look for the main process. I used top d1 to check the resource usage of the Process in real time to see if any abnormal process occupied cpu memory and other resources and found a strange process, I have never seen it before. This should be the trojan main process we are looking.

I tried to killall-9 ueksinzina to kill this process. But after killing it, ps-ef checked that there were still child processes. Didn't I kill it? View top d1 again and find another main process. It seems that the kill cannot be killed. If it is so easy to kill, it is not a Trojan.

Let's see what he is. "which obgqtvdunq" found that this command is under/usr/bin. After killing it multiple times, it is generated again under the/usr/bin directory, think of the program to listen to the status of the process and the scheduled task, and find that the process is dead and re-executed, I checked the/etc/crontab scheduled task and/etc/init according to the current idea. d. Run the startup script to check whether any problem exists.

We can see that there is a scheduled task gcc4.sh, which is not set by us. It is even more strange to check the content. This should be started after the listening program dies, here, we delete all related configurations and delete/lib/libudev4.so.

This file is also found in the/etc/init. d/directory.

The content is the boot information, which is also deleted.

The above two are a trojan started at startup, and the other is started after the trojan program dies. However, when we kill the trojan, it does not die, instead, we immediately switch the name to another program file, so it is useless to directly kill it. Our purpose is to prevent new program files from being generated, first, we cancel the execution permission of the program and lock the/usr/bin directory of the program file.

chmod000/usr/bin/obgqtvdunqchattr+i/usr/bin

Then we will kill the process "killall-9 obgqtvdunq", and then we will view the/etc/init. d/directory, and the new process is generated, and the Directory changes to the/bin directory. Like above, cancel the execution permission and lock the/bin directory, he is not allowed to generate, kill, and view the new file. This time, he is not in the environment variable directory. In/tmp, the/tmp directory is also locked, then terminate the process.

So far, no new Trojan process has been generated. In principle, the trojan program has been terminated. The subsequent work is to identify the files generated by these directories, first clear/etc/init. d. The Trojan STARTUP script generated under the directory, and then the/etc/rc # is clear #. d/connection file under the directory.

Later, I checked the modification time of the files under the/etc directory and found a new file under the ssh directory. I don't know if there is any problem.

After cleaning up, we need to clear the several files generated just now, one by one directory, for example, "chattr-I/tmp", and then delete the trojan file, similarly, you can delete Trojans under the/bin and/usr/bin directories.

 

Trojan Cleaning Process

Assume that the trojan name is nshbsjdy. If the top cannot be seen, you can view it under the/etc/init. d directory.

1. First, three directories are locked, so new trojan files cannot be generated.

chmod000/usr/bin/nshbsjdychattr+i/usr/binchattr+i/binchattr+i/tmp

2. Delete scheduled tasks and files and boot files

Delete the scheduled task and file rm-f/etc/init. d/nshbsjdyrm-f/etc/rc #. d/Trojan connection file

3. Kill the Trojan process

killall-9nshbsjdy

4. Clear the Trojan process

chattr-i/usr/binrm-f/usr/bin/nshbsjdy

After processing, check the preceding directories again, especially the latest files under the/etc directory.

5. If it is a rootkit Trojan, you can use the following software for inspection.

Software chkrootkit: Software RKHunter:

 

The installation is very simple. I used RKHunter to perform a simple check and did not find any major problems. However, this does not mean that there are no problems, because our detection commands are also dependent on some system commands, if the system command is infected, it cannot be detected. It is best to back up a copy of the command to check the system. If the system command fails, reload the backup data.