Processing cases after the Linux host was compromised

Analysis of a post-Linux intrusion

The following is a case study of a server after a rootkit invasion of processing ideas and processing process, rootkit attack is the Linux system under the most common attack methods and attacks.

1, the attack phenomenon

This is a customer's portal server, hosted in the telecommunications room, the customer received the notice of telecommunications: Because this server continues to send data packets, resulting in 100M bandwidth exhaustion, so telecommunications cut off the network of this server. This server is a Centos5.5 version and is open to 80, 22 ports.
From the customer learned that the site is not a large number of visits, so the bandwidth consumption is not too high, and the exhaustion of 100M bandwidth is absolutely impossible, then most likely the server suffered a traffic attack, so the login server to do detailed detection.

2. Preliminary analysis

In cooperation with the telecommunications personnel through the switch to the server's network traffic detection, found that the host does have external 80 port scanning traffic, so the login system through the "Netstat–an" command to the system open port check, it is strange that there is no connection with the 80 port related to the network. Then the use of "Ps–ef", "top" and other commands also found no suspicious process. The system was then suspected to have been implanted with a rootkit.
In order to prove whether the system has been implanted in a rootkit, we have the Web server under the PS, top and other commands with a version of the trusted operating system to do a md5sum check, the results found that the two commands under the Web server has indeed been modified, thus determined that This server has been compromised and has a rootkit-level backdoor installed.

3, broken network analysis system

Since the server is sending out packets, the first thing to do is to disconnect the server from the network and then analyze the system log for the source of the attack. However, the system command has been replaced, if you continue to perform operations on the system will become untrusted, there are two ways to avoid this situation, the first method is to take this server's hard disk to mount to another secure host for analysis, Another way is to copy all commands from one version of the trusted operating system to a path under the intrusion server, and then specify the full path of the command when executing the command, using the second method.
We first look at the system log to see if there is suspicious login information, execute the following command:

More/var/log/secure |grep Accepted

With a view of the command output, there is a log that raises our suspicion:

OCT 3 03:10:25 webserver sshd[20701]: Accepted password for mail from 62.17.163.186 Port 53349 ssh2

This log is displayed at 3:10 A.M. on October 3, There is a mail account from 62.17.163.186 this IP successfully logged into the system, mail is the system's built-in account, by default, is unable to perform login operations, and 62.17.163.186 this IP, verified, is an address from Ireland. The time it takes to log in from the mail account is earlier than the time the site server was attacked.

Then look at the system password file/etc/shadow, and found suspicious information:

Mail:$1$kced3yd6$w1evay5bmpqiqftwtvjix1:15400:0:99999:7:::

Obviously, the mail account has been set up password, and has been modified to telnet, the reason for using the mail account, speculation may be because the intruder would like to leave a hidden account, so that the future to log on to the system again.
Then continue to view other system logs, such as/var/log/messages,/var/log/wtmp are empty files, can be seen, the intruder has cleaned up the system log files, as for why did not empty the/var/log/secure file, it is unclear.

4. Find the source of the attack

So far, all we know is that a mail account has been logged in to the system, but why does it cause this Web server to continue sending packets to the outside? A corresponding attack source must be found, and a new suspect is discovered by replacing the PS command on this server to see the current process running on the system:

Nobody 22765 1 6 Sep29? 4-00:11:58. t

What is this. T program, continue with the top command, the result is as follows:

PID USER PR NI VIRT RES SHR s%cpu%MEM time+ COMMAND22765 Nobody 0 1740m 1362m 1228 s 98.3 91.5 2892:19. t

From the output, this T program has been running for about 4 days, the program is running the nobody user, and this T program consumes a lot of memory and CPU, which is also before the customer reflects the site server unusually slow reason, from this output, we get the T program process PID 22765, Next, according to the PID find the path where the program executes:
Go to the in-memory directory and view the information for the EXE file in the corresponding PID directory:

[[email protected] ~]#/mnt/bin/ls-al/proc/22765/exe lrwxrwxrwx 1 root root 0 Sep 22:09/proc/22765/exe-/var/tm p/.../apa/t

This will find the process corresponding to the full program execution path, this path is very subtle, because the/var/tmp directory by default, any user readability, and the intruder is exploiting this vulnerability in the/VAR/TMP directory created a "..." directory, and in this directory hidden the source of the attack, Enter the/var/tmp/.../directory and discover some of the rootkit files placed by the intruder, listed below:

[[email protected]  ...] #/mnt/bin/ls -aldrwxr-xr-x 2 nobody nobody 4096 sep 29 22:09  apa-rw-r--r-- 1 nobody nobody     0 sep 29 22:09  Apa.tgzdrwxr-xr-x 2 nobody nobody 4096 sep 29 22:09 cacadrwxr-xr-x  2 nobody nobody 4096   sep 29 22:09 haha-rw-r--r-- 1  nobody nobody      0 Sep 29 22:10  kk.tar.gz-rwxr-xr-x 1 nobody nobody      0 sep 29  22:10 login-rw-r--r-- 1 nobody nobody      0 sep  29 22:10 login.tgz-rwxr-xr-x 1 nobody nobody      0  sep 29 22:10 z 

Through the analysis of these files, the basic judgment is that we are looking for the program attack source, where:

1) The z program is used to clear the system log and other related information, such as execution:

./Z 62.17.163.186

After this command is executed, all logs related to 62.17.163.186 in the system will be cleared out.
2) in the APA directory has a backdoor program T, this is the previous in the system to see, after running this program, this program will automatically read the APA directory IP this file, and IP this file records a variety of IP address information, guess this T program should be to scan IP files recorded in all IP information, In order to obtain the permissions of the remote host, it can be seen that the site server is an intruder of a broiler.
3) The Haha directory is a program used to replace system-related commands, that is, the program in this directory so that we can not see the operating system anomalies.
4) Login program is used to replace the system login program Trojan, this program can also record the login account and password.

5. Find the cause of the attack

Up to this point, the attack on the server has been basically clear, but how did the intruder hack into the server? This problem is important, we must find the source of the invasion, in order to fundamentally plug the loopholes.
by looking at the Apache configuration discovery, Apache only deal with static resource requests, and Web pages are mostly static pages, so the intrusion system via web page is unlikely, since the vulnerability may come from Apache, then try to view the Apache log, may be able to find some suspicious access traces, By looking at the Access.log file, the following information was found:

62.17.163.186 - - [29/sep/2013:22:17:06 +0800]   "get http://www.xxx.com/cgi-bin/awstats.pl?configdir=|echo;echo;ps+-aux%00 http/1.0"  200  12333  "-"   "mozilla/5.0  (windows; u; windows nt 5.1; pt-br;  rv:1.8.1)  gecko/20121010 firefox/2.0 "62.17.163.186 - - [29/sep/213:22:17:35 + 0800]  "get http://www.xxx.com/cgi-bin/awstats.pl?configdir=|echo;echo;cd+/var/tmp/.../haha;ls+-a%00  http/1.0 " 200 1626 "-" " mozilla/5.0  (windows; u; windows nt  5.1; pt-br; rv:1.8.1)  gecko/20121010 firefox/2.0 "

At this point, the source of the vulnerability is discovered, Originally a vulnerability in the awstats.pl script Configdir, by understanding the application of this server, the customer is indeed through a awstats open source plug-in to do Web page access statistics, through this vulnerability, the attacker can directly in the browser operation of the server, such as viewing processes, creating directories and so on. As can be seen from the second log above, the attacker's normal browser performs an operation that switches to the/var/tmp/.../haha directory.
This script vulnerability is very scary, but in Awstats official website has already given the method of patching, for this flaw, repair method is very simple, open the awstats.pl file, find the following information:

if ($QueryString =~/configdir= ([^&]+)/i) {$DirConfig =&decodeencodedstring ("$");} Modify to the following: if ($QueryString =~/configdir= ([^&]+)/i) {$DirConfig =&decodeencodedstring ("$"); $DirConfig =~tr/ A-Z0-9_-/./A-Z0-9_-/./CD;}

6. Uncover The Mysteries

Through the above stepwise analysis and introduction, the cause and process of the invasion of this service is very clear, the general process is as follows:

• (1) The attacker enters the system through the vulnerability of the Awstats script awstats.pl file, creates a hidden directory under the/var/tmp directory, and then passes the rootkit backdoor file to the path.

• (2) by implanting the backdoor program, the attacker obtains the system Superuser privileges and then controls the server, which is contracted out through this server.

• (3) The attacker's IP address, 62.17.163.186, may be through proxy or other broiler servers controlled by the attacker.

• (4) In order to permanently control the machine, the attacker modifies the system default account mail information, changes the mail account to be able to log in, and sets the password of the mail account.

• (5) After completing the attack, the attacker automatically cleans up the system access log through the backdoor and destroys the evidence.
  Through the analysis of this intrusion process, it is very simple and common to find the intruder's means, although the intruder deleted some of the system's logs, but still left a lot of traces can be traced, in fact, you can also view the user's. bash_history file, which is the history of user action commands.

7, how to restore the site

Since the system has been changed and replaced, this system has become completely untrusted, so it is recommended to back up the site data, reinstall the system, the basic steps are as follows:

• (1) Install a stable version of the operating system, remove the system default and unwanted users.

• (2) The system is changed to public key authentication mode, avoiding the defect of password authentication.

• (3) Install a later version of Apache and the latest stable version of the Awstats program.

• (4) Use Tcp_wrappers Firewall under Linux to restrict the source address of SSH login.

This article is from the "it--Brother" blog, please make sure to keep this source http://402753795.blog.51cto.com/10788998/1770394

Processing cases after the Linux host was compromised