Analysis and Summary of the CPU million murder case triggered by the autumn Garden

I wrote an article a few days ago: Analysis on the Causes of CPU usage in the autumn Garden

As we have introduced some causes, consequences, intermediate processes, and basic content above, we will not re-write what happens and what tests are done in the middle. Below we will write about the subsequent development of some things.

After solving the naming case caused by search engines, some netizens put a stress test on the autumn Garden. The number of connections went up to 1-2, leading to CPU failure.

I. Possible attack cases in URL cache:

 

At that time, I was remotely using the SQL event probe and performance counters. I found that there were thousands of concurrent IIS connections, constantly generating SQL statements, resulting in the database being fully occupied by CPU, the website cannot be opened.

From here, I found some logical discord in the system:

For example, http://www.cyqdata.com/tech/cate-19. the slow storage of the autumn garden website is basically based on urlto slow storage.

The stress test initiated by the peer party generates hundreds of concurrent Network Connections: http://www.cyqdata.com/tech/cate-19-nxxxxxxx.

Due to the changes in the URL, new page queries are constantly generated, resulting in constant database queries. The CPU of the database for continuous concurrent queries is directly full and goes down.

To solve this problem, I performed a comprehensive URL parameter processing. Before I read the database, I determined the validity of the parameter and made logical adjustments. I put the jump detection function in front, other reads are placed behind them. Invalid addresses do not cause unnecessary database statement queries or logic. (There are several websites in the same way ).

According to this rule, everyone should pay attention to the fact that, based on the URL address cache logic or dynamic generation of static page logic, we should check carefully.

 

Ii. Small DZ Forum check

 

Now there are a lot of forums and many personal sites are running on VPS or virtual machines. I ran to see them and tried to find some pages for dynamic processing to perform stress testing, it is found that, except for a few dynamic posts such as login, the website is almost static.

Generally, searching is dynamic and time-consuming. I found that DZ directly references Soso's search service, transfer the pressure.

 

 

Iii. Test Tool for concurrent website Stress

 

A lot of questions are related to the usage of the Tool. Web users use the AB .exe tool provided by apache. as long as Apache is installed, this tool is available in the directory. A simple command line can be used to perform a simple stress test on a website.

However, there is a limit, generally up to 64 concurrency, which is much better than the 15 free concurrency provided by the online concurrency test website by default. in Linux, there are several articles about modifying the number of concurrency, I can't find a way to modify the maximum number of concurrent jobs in windows.

 

 

4. An unknown thread deadlock:

 

This problem occurs several times during local testing. when it occurs, I also actively Dump it. Through processxp, I can see that two threads each account for 20% and the duration is long, no more information.

Dump is a sad thing. I am really not good at it. Just a few commands come and go back and forth, and there is no information. I dumped it once a few years ago. A few years later, Dump is still at that level, the miserable time cannot find the problem.

After a long time, I gave up. Later, I had been vigorously restructuring the code. Many code logic were deleted and rewritten.

After that, the problem also disappeared and never appeared again for a long time. I don't know if it was accidentally found in the code I deleted.

 

 

 

V. Murder caused by insufficient system disk space:

 

A Weibo user gave me a message saying that an error was reported when the autumn garden was opened, prompting that the hard disk space was insufficient. I had to log on remotely and check that there was only 128 kb left for drive C ,-_-.....

First, I cleared something and got more than one hundred mb. Then I recovered the system operation and found out who had taken the hard disk space:

I don't know. I was shocked. It turned out that IIS logs had nearly seven GB of logs (10 Gb for a c disk on VPS)

I opened the log and read all the logs .. At first glance, they are all requests generated during stress testing, with a log nearly 500 mb.

According to this situation, I am thinking that for small sites, requests are constantly sent using multiple threads, resulting in rapid log growth and the space of the other side is congested. Haha, it's so evil !!!

There are two tragedies in life. One person insists on seeking death, and the other one cannot survive. You are all inclusive.