Blood cases caused by a local variable

Today, I received a temporary task to troubleshoot a website's strange problem. This is the case. The website has a large access volume. I went to a module and sent an HTTP request to the Page Server, after reading the data provided by another Java website and going online, it is found that once there is concurrency or a large number of accesses, the HTTP request will fail, and even no page can be opened on the server, however, the server can be pinged or other websites (I did not see the real situation, but I heard that there is such a situation ).

 

I always feel terrible when the server of a High-concurrency website initiates an HTTP request. I have never done this in my previous projects (I would rather make client Ajax requests ), from the very beginning, I suspected it was a thread pool problem. As we know, Asp.net requests are processed by the worker threads in the thread pool. If an HTTP request is initiated synchronously in this thread, it is like pulling a memory-level processing speed to the network-level processing speed (because synchronization is required to wait for the Network ), I wonder if the website's access traffic is huge (for example, the thread pool allows a maximum of 800 worker threads, and the current concurrency needs to occupy 200 ), then, after the speed (for example, from 100 milliseconds to 1 second), the thread pool is full (200 × 10 = 1000> 800 ). Therefore, I suggest developing asynchronous pages and asynchronous httpwebrequest. The simplest way is to use the event-based asynchronous mode downloadstringaync () of WebClient (). After the developer makes changes according to my suggestionsCodeAfter going online, the problem persists. I was confused at the moment and couldn't figure out why (I tested it online, using this method, even if a large number of requests always occupy a worker thread, iocp occupies a lot of resources ).

 

After obtaining the chance of online server debugging, the modified page code is embedded with a timer to regularly output the available working threads and available iocp of the thread pool, after the result request is sent, only 30 working threads are used (about 30 concurrent threads on our website), and iocp is not used (because the code is rolled back later, no asynchronous HTTP request is used ). Later, I wanted to check the TCP connection. After Entering netstat-s, I was dumb. I saw more than 4000 TCP connections. At that time, the first reflection was that the traffic was too large. Write beforeProgramRemember to modify the Registry to enable more than 60 thousand TCP ports for Windows Server (the default port range seems to be more than 1000 to 5000, which is exactly the same as the preceding 4000 connections ), remember that it takes four minutes to continue using the TCP port after it is disabled. Check the two parameters maxuserport and tcptimedwaitdelay, modify the two values to the maximum value of 65534 and the minimum value of 30 (seconds) According to msdn ). If every request to access the website is a new connection and an HTTP request is required for each request, the request occupies up to two ports, it can build up to 60 thousand ports and release all ports in 30 seconds, that is, it can process 30 thousand requests in 30 seconds, so our processing capability is 1000 requests/second, however, our website generally has no more than 50 concurrent connections per server, which is far enough. After I changed the registry and restarted the machine, I found that the modification took effect. However, after the pressure on the TCP connection, it quickly reached more than 60 thousand, and the server was unable to access the Internet, at this point, it is determined that the root cause is that there are too many TCP connections and no more connections can be established (of course, no HTTP connections can be established). However, the opened pages can indeed be opened, and new pages cannot be opened, I suddenly thought of the HTTP keep alive.

 

With this direction, modify the code and set the keep alive for httpwebrequest (to confirm whether the connection: Keep Alive HTTP header is added, a lot of twists and turns are required ), keep alive is also enabled for the target Web server, and a bunch of servicepointmanager attributes are set in a mess. After several hours of attempts, it still won't work, it is always found that the server can easily reach 60 thousand connections (less than one minute), and the CPU is soaring. Then, just like the network disconnection, it has been entangled in why there is no keep alive and why TCP connections are not reused. Later, I made a separate website and found that it didn't occupy so many ports (do other services or modules have opened many TCP ports ?).

 

Suddenly, why don't you see what connections have been established? Enter netstat-an-p tcp> C: \ a.txt, and then open a.txt. 99% of the connections are to the memcached server. First, memcached is widely used in the code? After checking the code, it is found that although a request may indeed perform memcache access for about 10 times (not to mention less, but it will not be highly concurrent ), however, the client we use has a connection pool, and it is impossible to establish so many connections at once when the concurrency is large. In addition, the maximum TCP connection in our connection pool is 500. So I continued to check the code. After seeing the initialization code of memcachedclient, I instantly declared a local variable every time and initialized the memcachedclient class instead of using static variables to save the instance of memcachedclient.

 

When investigating the bug in the memcache client connection pool, we learned that each time a new memcachedclient object is instantiated, a new connection pool is created. The so-called pool has a minimum connection, that is, these minimum connections will be established when the pool is initialized to achieve better initial performance, this value is configured as 10 (10 new TCP connections are required for each request ??). Write a loop test to run 100 cycles, and the TCP connection is set to 1000 (and each initialization of the Connection Pool takes more than 300 milliseconds, and the CPU is always 100%, it can be seen that creating so many connections consumes a lot of performance), which is exactly the same as the conjecture. I also thought that a dozen connections were added when I refreshed the page. This is the reason. After the local variable is changed to the Global static variable, the website still runs well with 100 concurrency, far greater than the original expectation (50 concurrency ).

 

From the thread pool suspicion to the insufficient port, then to the keep alive problem, and finally to locate the Final Cause (it does not matter with the newly added httpwebrequest module), it took a lot of twists and turns. Then the solution is simple. Check other memcache usage methods in the project and modify the memcache client (we use enyim. modify the memcachedclient constructor to private and provide the singleton entry.

 

My experience on performance optimization is that most of the performance problems often come from one or two root causes. If you can find and solve them, it may be very effective. I hope this article will help you.