Web request asynchronous processing lowers application-dependent risk

Problem Highlights

The day has come, the merchant busy promotion, the website is busy promotion, the Ali software service integration platform also faced with the first multi-party large-scale pressure test. Based on the 5.3 release of the pressure test results of the platform, we estimate the existing promotion will bring the pressure, basically determined that the service integration platform does not require expansion at the end of the year. SA (System administrator, sysadmin) for the sake of insurance or through the request method to do timed heartbeat detection, to ensure the reliability of the service integration platform. Results Ali Wangwang Promotion began on the first day, the SA alarm message in a few busy hours non-stop alarm, but look at the production environment of the server status and application status can not see what the problem, and began to doubt whether the alarm mechanism is not very reasonable. After a few days of visiting records, the statistics report has been read, found a number of problems, first of all, because the promotion is in the IM login period centralized promotion, so the peak is relatively concentrated, the pressure is also very large, and the timing of the alarm is also those times; also found that the extension of the use of the API processing time is relatively long, but also some problems, These days in addition to the service integration platform alarm, those API server is also in the alarm; therefore, it can be seen that the problem should be due to the API provider slow response to the service integration platform, the processing capacity of the monitoring mechanism in the peak situation did not get timely response, it is considered that the server has been in

In fact, this kind of problem is often present in our application architecture, because there are few pure "closed" applications, dependence on database, dependence on storage, reliance on third-party systems, and so on. It also reminds me that in a previous security meeting, the head of the security technology team at Tencent said the biggest problem with security now lies in the potential security implications of a third party's security without being controlled. Web applications are not, from the most basic transaction processing to a small granularity, do not include Third-party dependencies in the transaction, to heartbeat detection, fault-tolerant design, etc., have made us aware of this problem. But often this kind of problem is not the local design can see, if not a general architect for the overall grasp, coordination and prevention, then the problem arises and the impact will be very large.

Previously, the stress tests for the service integration platform were primarily done under the "basically normal" ISP service, but the exposure of the problem required us to do something or improve the design in a timely manner when a third party relied on a boundary problem.

Problem Analysis and Solutions

Problem Reason:

Blocking method for HTTP request processing.

The back-end service processing time is too long, the service quality is not stable.

Web container Accept request thread resources are limited.

Solution:

Blocking is a non-blocking way to process requests.

Set back-end timeout, actively disconnect, and reclaim resources.

Modify the container configuration, increase the thread pool size, and wait for queue length.

Solution one is the hardest to do, and the rest of the space will describe the exploration of the technology.

Solution Two is relatively easy, allowing each ISP to set the maximum timeout allowed by its API.

Solution three Tomcat and JBoss have two parameter configurations (MaxThreads and Acceptcount) in connector that can be adjusted.