Issues and workarounds for IIS handling concurrent requests

When an ASP. NET project is deployed to production environment, when the user concurrency reaches about 200, IIS has a noticeable request queuing phenomenon, the request sent to wait, unable to respond in time, the system is basically in an unusable state. Due to the lack of experience, spent a lot of time and energy to solve this problem, this article records my search for problems and the final solution for your reference.

Hardware and Software Environment:

IBM blade servers, Intel Xeon processors, 4 physical cores, 16 logical cores, memory 32G

Windows Server2008 Enterprise R2, ASP. 4.0 Webform IIS7.5 Integrated mode

When there is a noticeable delay in the discovery of requests, and there is no immediate processing, the first thing to look at is the Performance log performance Monitor that comes with Windows.

Because I noticed that only requests for. aspx or. ashx are delayed, and. htm or. jpg files are immediately responsive, so obviously the problem is with ASP. I chose the 2 main counters in ASP. NET 4.0 in Performance Monitor: Requests Current (number of currently requested), requests Queued (number of requests queued) for observation. By observing that the current number of requests reached about 200, the number of requests queued has risen from 0 to about 50, and if the number of requests continues to rise, the number of queued numbers increases. When the number of requests queued is >0, it means that at this time to access any. aspx page, the page will be in a long wait, without any response, until IIS finishes processing the other requests before it starts processing the requests in the queue. That is, the system is basically in an unusable state when the queue number is >0 for a long time.

Because the page layout of this system is more complicated, it uses a lot of ajax+.ashx way to display the content on the page in batches, so the total number of requests to the server will be more than the traditional ASPX mode, it may take 5-10 seconds for a page to complete loading. But I think this should not be the main cause of the problem, even if the system performance is poor, IIS should be enough to withstand such a small amount of concurrency.

To explore the question of whether the system was written, or the problem with IIS itself, I threw away our system and wrote a simple page, on an aspx file, Page_Load sleep for 10 seconds. Assuming this is a poor performance site, each page takes 10 seconds to show, I'm deploying it on IIS to test its performance, I'm using the Microsoft Web application Stress Tool, simulating the initiation of 80 threads, 4 sockets per connection, A total equivalent of 320 concurrent requests.

After the test starts, it can be seen that the current number of requests climbed to about 300 (the red line in the graph), and then the number of requests in the queue rose to about 300 (green Line in the figure), that is, in 300 concurrent requests, almost all of the requests are queued, the system is basically not available, through a simple test, The problem has been reproduced.

Over time, the Green line is gradually reduced, from 300 to more than 100, that is, the system usability gradually improved, some users can use, but most of them are still in line.

After 6, 7 minutes, the number of requests in the queue dropped to about 0, and there were some minor fluctuations. Most requests can be handled normally at this time. If you follow this behavior, it should be that IIS finds a large number of requests in the queue and tries to increase the number of processing threads to meet the requirements, but at a slower pace.

Is that the system after 6, 7 minutes of the adaptation period, the future can always be in this concurrency stable operation? That is not the case. I stopped the stress test for a few seconds, when the server's request dropped to 0, and then reopened the 320 requested tests, how does IIS behave? As you can see, as long as the number of requests rises significantly, the wait queue starts to reach the highest value, then slowly declines, repeating the process above. The summary is that, when large concurrency occurs, the ability of IIS to process requests is completely out of date and takes a long time to open up enough threads.

Then I did a test to see how many requests IIS can tolerate by default without queuing up? Seems to be in 100 concurrency around, performance is fair, did not appear in line.

When it's 200 or so.

Then I change the test program from SLEEP10 seconds to 3 seconds, for an application system, the average page 3 seconds processing time performance is relatively normal. But unfortunately, the queuing phenomenon and processing time do not have much to do with the queue is still very serious.

In view of the above problems, the relevant data, whether the queue is the application pool of available threads, through 2 methods to see the number of system bus and the number of currently available threads.

Threadpool.getavailablethreads (out Availableworker, out Availableio);

Threadpool.getmaxthreads (out Maxworker, out Maxio);

When the number of queue requests reaches about 120, the maxworker=1600 is obtained by this method, and availableworker=1472

Because the server is 16-core, the asp.net4.0 default per-core can use 100 threads, so Maxworker is 1600,1600-120=1480, roughly equal.

This means that there are currently 120 threads being used to process requests, and more than 1400 are idle. The key question is why are these idle threads not being enabled in time?

Asp. NET provides the thread configuration parameters, there is a parameter is very important, but may be ignored by everyone, is minworkerthreads.

Refers to the minimum worker thread, according to our test results, IIS managed threads start very slowly, Microsoft recognizes this problem, so provide this parameter to set the minimum number of worker threads under normal circumstances. For example, our system can be set to a minimum of 300 for daytime concurrency of 200-300, so the system response speed can be greatly improved.

Accordingly, I made the following modifications to the configuration file (machine.config). Note that for a single CPU, the system is automatically multiplied by the number of logical CPUs.

<system. web>
<processmodel autoconfig= "false" maxworkerthreads= "$" minworkerthreads= "/>"

The equivalent minimum worker thread is set to 50*16=800.

After restarting IIS for testing, we get the following results:

As you can see, because the reasonable minimum number of worker threads is set so that IIS does not have to constantly create new threads to handle requests, the responsiveness of the system can already meet the concurrency requirements.

In addition, a new feature, called Web Garden, was introduced after IIS6, which was designed to improve server performance in situations where the CPU is low, but the number of concurrent requests is much higher. This is in line with my current situation, so I enabled the Web garden, adjusted the number of worker processes from 1 to 5, and in Task Manager you can see that the w3wp process has increased from 1 to 5 and then re-tested.

With the same 320 requests, you can see that in addition to the first few seconds there are some queues, which basically perform well and do not request to enter the queue.

Through the above two methods, can effectively solve the problem raised at the beginning of this article. But the Web garden is working in a multi-process mode, if the application uses the dependent process of the session and cache objects must be another way, can not be saved in the server memory, and the Web garden of multiple processes to switch the context of replication, Its resource consumption is larger than the single process, which is a factor to consider.

Issues and workarounds for IIS handling concurrent requests