Seven-layer flow access system

This is a creation in Article, where the information may have evolved or changed.

Talk to everyone. Seven-layer traffic access middleware.

1. Introduction and architecture of access system

1.1 Go Reverse Proxy

A custom-made reverse proxy is implemented with the go language.

Go language

In recent years, more popular in China, with the fame of Docker more popular. At present domestic use go development of the team and system more and more, like Baidu's BFE, 360 long connection push, seven Qiniu storage, drip login authentication, etc., the list is very long.

Go is more suitable for the development of middleware (reverse proxies, message queues, etc.) and bypass systems (storage, long-connected push, etc.), and many teams are starting to use go to write web

API (using the Beego framework). The Go language provides native co-processes (go

Routine) support, inherently high concurrency, but also with synchronous logic to write asynchronous programs (the language underlying encapsulated asynchronous I/O), development efficiency is very high.

Re-build the wheel

The industry has a lot of mature reverse proxy (Nginx, tengine), why not based on these open-source projects two times development? The main considerations are three points: 1) Development efficiency, 2) custom-made, 3) maintenance costs.

Nginx and Tengine are based on C language development, C language development efficiency is relatively low, the maintenance cost is greater, in the language selection more inclined to go, considering the custom-made and easy to understand (small code, less logic), tend to re-implement a reverse proxy, maintain high scalability and eliminate unnecessary logic. In fact, the dominant thought is: bogey chatty.

1.2 One-core multi-channel

As a flow inlet, its stability is the primary consideration in the design, so the idea of a nuclear multi-channel is proposed. A bypass system that covers basic functions and runs at the core + multiple extended functions. A bit like the idea of microservices.

Core Server

The server must guarantee high concurrency and provide a scalable incision.

Nginx achieves high concurrency by Master-worker multi-process + asynchronous I/O. With the advantage of the go language, it is easier to achieve high concurrency. With a master Goroutine + multiple worker goroutine. To be exact, a new goroutine is created for each HTTP request.

In each worker process, logic such as decryption, shunt, anti-crawl, and forwarding is provided. Using Nginx's request processing process (phase), the server divides the request arrival into the request return into multiple phases, selecting several of the typical stages for the callback registration point. The so-called callback registration point, which allows the handler function to be registered at this point, server callback this function when the request executes to this process point. Each callback point corresponds to a handlers array, and handlers is executed sequentially when the callback occurs. It is obvious that the callback point mechanism provides a very convenient extensibility.

Core server

In each worker process, logic such as decryption, shunt, anti-crawl, and forwarding is provided. Using Nginx's request processing process (phase), the server divides the request arrival into the request return into multiple phases, selecting several of the typical stages for the callback registration point. The so-called callback registration point, which allows the handler function to be registered at this point, server callback this function when the request executes to this process point. Each callback point corresponds to a handlers array, and handlers is executed sequentially when the callback occurs. It is obvious that the callback point mechanism provides a very convenient extensibility.

Module Model

The handler on the callback point represents a function, and the module represents a class of functional entities. That is, a module can register handler on multiple callback points separately. For example, the data reporting client and the module of the access log registered handler on two callback points respectively.

The advent of module model symbolizes the practice and the landing of a multi-channel design concept. When monitoring the system running state load is too high, the pressure is too large, you can take off some module to ensure the stability of the core, to achieve service degradation.

2. Configure hot and graceful restart

Access services, changes to the triage rules, changes to the business backend machines, new access services are unavoidable, and the upgrade and iteration of the service itself is a continuous process. How to ensure the continuous operation of system services and the efficiency of change when making changes is the next topic to talk about.

2.1 Configuring Heat More

In the case of non-stop service configuration changes are called configuration hot more. The only difficulty with heat is the consistency of the data before and after the change

Sex, that is, when the t time occurs to configure the heat more, for the T time before arrival positive request should use the pre-change configuration; For a request that arrives after T, the changed configuration should be used.

The industry generally has two solutions: fork process and pointer switch.

Fork Process

The master process fork out a child process to load the configuration, and when load is complete, the master process gracefully exits the child process before the configuration change. The benefit of such a scheme is that there is no need to consider the coupling between configurations, and the disadvantage is that it is slightly more complex to implement. Nginx uses this kind of scheme.

Pointer Toggle

The change is implemented by toggling the address of the configuration data memory. The pseudo code is as follows:

Pointer Toggle Pseudo-code

This scenario is simple and relies on a premise: the language itself provides a GC mechanism. The old configuration memory will not be freed until the request for the T moment has not been served, and the GC reclaims the memory when all the requests for the T moment have been served. Based on the go implementation of the Access system, natural selection pointer switching mode to achieve the configuration of heat more.

There is a potentially huge benefit to configuring heat: platform-based. Open the platform to each business classmate, freeing the access system maintenance personnel's hands (no longer need to receive a large number of configuration change tasks every day).

2.2 Graceful Restart

In the system iterative upgrade, we have to face the issue of launch. How can the system be upgraded with no-stop service? There are three solutions for the general industry: reuse Port, fork + exec, and Healthcheck + Supervisord.

Reuse Port

Linux kernel 3.9 provides so_reuseport properties that allow different processes to bind ()/listen () to the same tcp/udp port. means that when we iterate through the code, we can run both the old and the new two code on line, and when the new Code service is stable, let the old process exit to complete the code smoothing upgrade. There are two points to using this scheme: 1) kernel requirements; 2) old service gracefully exits. Graceful exit is not difficult, send a signal to the old process, the old process closed off the listen port, and so on the system residual request service after the exit.

Fork + exec

The master Process fork the child process, calling itself with exec, when the system service runs, sends a signal to the master process, the master process closes listen and then exits, which is the most graceful way to restart. The pseudo code is as follows:

Graceful restart

Healthcheck + Supervisord

Before releasing the Healthcheck file on the machine, and so on, the system flow clean, began to replace the system code file (binary or other), and then kill the service process, Supervisord pull up, so as to achieve the upgrade, after publishing, add healthcheck files.

This kind of scheme is most used in the industry, very trik, but it is more effective.

3. Best practices

3.1 GC Optimization

There are several methods of GC optimization for GO: Small object merging, on-stack allocation, Sync.pool, CGO, memory pool and upgrade go version.

The CGO, the memory pool, and the upgrade go version have a noticeable effect. CGO is the way go provides the call C code, the efficiency is higher than go, go1.7 release, and 1.4.2 comparison test, GC pause time decreased by 30%. Memory pool (shown), which is self-managing memory, appears to be an object in the GC, so it can also be a good mitigation for GC.

Memory Pool Model

3.2 TCP + protobuf

When designing a private protocol based on the TCP protocol, it is generally necessary to define a header and MSG, as shown in. For the content of MSG, it is expected that the receiving end can map directly into the data structure, that is, serialization and deserialization are required. There are many common serialization protocols, such as JSON, thrift, Hessian, Protobuf, and so on. I personally recommend Protobuf, the main protobuf support a lot of languages, protocol field compatibility, serialization deserialization faster and Big factory (Google) support.

Private protocol

3.3 UDP

UDP as a non-connected protocol, the use of "best effort" transmission of the theme, that is, regardless of network conditions and the recipient's ability to receive, as long as the hair can be sent, without concern. This feature of UDP means that it does not guarantee accurate data delivery and is not guaranteed to be orderly, for congested networks may make the network worse, which is the disadvantage of UDP compared to TCP. However, it is also because of this, UDP transmission efficiency is high.

For inter-intranet communication, network conditions can be controlled, to assess the characteristics of the business (such as high real-time requirements, but can tolerate a certain degree of packet loss), you can try to use UDP as a transport protocol. It is the UDP packet loss rate of the intranet inter-machine room that I measured.

UDP packet loss rate test data

3.4 Unix Domain Socket

For interprocess communication, the Unix Domain socket has a greater advantage over socket communication, which does not require the processing of the network protocol stack, and is very fast through a simple copy between the memory. It's a good choice for communication between services that are deployed on the same machine.

As for the connection-oriented byte stream in the UNIX Domain socket and the non-connected datagram, both can guarantee the accurate arrival and order of the data, the only difference lies in the semantics. For example, a read operation, a byte stream service, can call a read operation multiple times to receive a message data sent by the sending side, but the packet's service allows only one read operation for a packet.

4. Service downgrade and plan

Access system as a traffic entry, stability is the first, when the attack and back-end business failure, we must have a solution. This chapter discusses this issue with you.

4.1 Service downgrade

Inlet flow Control

When an attack causes traffic to burst, it can cause system resources to run out of time and be killed by the operating system. The rough way to deal with this is to set up a global counter that records the number of requests that reside in the current system, and once the value exceeds a threshold, the new incoming request is rejected.

If you want to gracefully solve the above problem, you need a bypass DDoS anti-attack system, which is a multi-dimensional count of requests, when a threshold is reached, issued an instruction to the access system, the access system to the new incoming match marked characteristics of the request to implement the rejection.

Business Isolation

When the back-end service has a serious time-out fault, the forwarding of the request to this business time-out retry, over time, the resources consumed in the system increased linearly over time, resulting in excessive GC pressure, which affects the response time of other services. To cope with this situation, a business quota mechanism was designed to isolate the business failures, as shown in.

Business Isolation Flowchart

There are two mechanisms for setting quotas for each business: static and dynamic. Static quotas are strictly dependent on experiments and experience, not optimal configuration, but simple to implement, dynamic quotas, to maximize the use of system resources, but the implementation of a little more difficult.

4.2 Plan

Slightly