Architecture of the application layer of Tencent WAF (Web application protection system)

Preface

As a vulnerability protection system for Tencent webserver, Tencent's door god system (hereinafter referred to as Door God) already covers nearly webserver servers and processes tens of billions of HTTP data packets per day.

There are many WAF implementations. For details, see mainstream WAF architecture analysis and exploration. Based on the company's business characteristics, we adopted the "server module + detection cloud model" mentioned in this article ".

This article mainly describes the overall back-end architecture and related technical solutions for implementing such WAF, the various difficulties encountered in the specific implementation process, and the advantages and disadvantages of such WAF.

Overall Framework

Figure 1 overall framework of the door gods

The overall framework is divided into online and offline parts.

The online part is linked in the whole link of the user's access to Tencent's website. The Door God Module (Blue part) this includes forwarding http data to the door god agent at the back end of the Door God, and determining whether http requests are malicious;

The offline part mainly generates malicious/non-malicious rules for the server, as well as data statistics and abnormal data alarms.

Next, we will introduce the three most important modules in detail:

User request data forwarding module-Door God agent

We added a door god agent module in the business webserver program. When the user requests the page, the business webserver first calls the Door God agent registered entrance api after parsing the http request package, the agent obtains and processes the srv ip and port according to a certain load balancing algorithm, and then forwards the http request header, request body, user ip address, and other data to the door god judgment server through udp and tcp.

The difficulty of this module is that the company's business has a wide variety of webservers. Do we need to make an appropriate agent module for each webserver?

Our solution is to provide a uniform agent module for mainstream webservers, such as apache and nginx, and provide protocol unpacking APIs for self-developed websrv to complete socket communication by services; add an nginx proxy for forwarding at the front end of a non-mainstream webserver, and add the Door God agent module at the proxy layer.

Compared with apache's multi-process synchronization mechanism, nginx's asynchronous mechanism makes the agent module much more complex and requires that the module be asynchronous, this includes asynchronous retrieval of body data and asynchronous processing of srv.

There is no ready-made sample in the Nginx Open-source module. After our research on nginx source code and multiple versions (including the use of open-source mtask module in the early stage and self-developed at the end), we can solve the full asynchronous problem. The specific implementation method can be seen in the subsequent article "implementation of the door god-nginx module and difficulties encountered"

Identification Module of malicious user data-determining the server

After determining that the server has received user data, parse the http request data and divide it into uri, args, host, clientip, and other fields for preprocessing, then, use a single field or a combination of fields to match malicious rules to determine whether the content is malicious.

The [user request data forwarding module] and [user malicious data identification module] are online systems. Once a fault occurs, the service will be directly affected. Therefore, in addition to functional requirements, it is also necessary to meet the general requirements of massive services in the background architecture:

Stability

The program has no core and no endless loops.

Disaster Tolerance

Once some backend judgment servers fail, the door god agent can automatically switch to the available judgment server.

Performance

Simplify the online processing process as much as possible, and determine that the server cluster uses machines as the minimum processing unit, and that multiple processes listen to the same port to compete for data processing, instead of using the traditional "data forwarding + backend multi-process server" mode. 99% of the HTTP data experts determine that the server can return results within 1 ms.

Scalable

New functions are supported: for example, adding the CC module to online automatic real-time interception.

High maintainability

The online logic is simple enough and efficient: the online module and the offline module are separated in low coupling mode (the Door God log forwarding server is added );

Efficient maintenance: One-click compilation, one-click operation, one-click function/performance testing, and one-click release

High Availability

Easy and quick to add rules: add rules in xml format on the web front-end, which can be verified in real time and released to the test environment and grayscale formal environment with one click;

System Monitoring & alarms:

The door determines server availability monitoring and exception alarms, including: malicious blocking, non-malicious Miss, cpu/memory resource monitoring, business websrv timeout monitoring, and exception alarms.

Generation of rules for malicious data identification-rule management system

We generate an xml rule file on the front-end of the rule system and send a command to notify all the door gods to determine that the server dynamically loads the rule. The brief process is as follows:

Figure 2 rule issuing process

There are two ways to generate rules:

1. Collect industry web vulnerabilities, including 0 days, and convert them into defensive rules;

2. The underreporting analysis system extracts possible underreporting based on loose rules (with an accuracy of about 50%), and manual analysis converts real underreporting into defense rules;

Advantages and disadvantages of this WAF model

L advantages

1. Low business maintenance costs

After a deployment, all rules and program changes caused by malicious determination of logical changes are performed on the backend, and no updates are required on the business side.

The logic of the door-to-door agent on the business side is simple enough, which basically does not affect the service performance and functions.

2. Comprehensive types of protection vulnerabilities

This WAF model summarizes all HTTP data and identifies malicious requests based on user behavior characteristics. For example, our malicious pull recognition module can prevent CC attacks and other malicious behaviors.

L disadvantage

Compared with the industry's waf solution, our solution also has its disadvantages, such:

1. webserver type problems

We need to adapt to the latest mainstream webserver. Once the version of webserver is significantly updated or a new popular webserver is available, our door god agent may need to be re-developed. Due to business complexity, it may take several iterations to stabilize the system. For example, if the nginx Door God Module iterates over 16 versions, the system will be stable;

2. Deployment problems

Services need to be deployed, which may involve re-compilation of webserver and other workloads at a certain cost. In addition, when thousands of domain names are involved, the problem becomes more complex.

To address these disadvantages, we also try to use other modes to balance its defects. For example, we have added the WAF function in the shield system to block malicious user requests at the network layer.

Finally, due to my limited ability, the solutions used for various problems mentioned in this article may be limited, or other solutions can be used to completely circumvent these problems, we sincerely welcome suggestions and criticisms from the industry.