Combat Record | Listen to "old driver." Distributed cluster high concurrency problem _ distributed cluster high concurrency

"Actual combat record" introductory

Cloud Guardian "combat record" section regularly will share some of the experience and skills in the development of operation and maintenance, hope to pay attention to our friends benefit. This issue is shared by Ding Ming, a security operation engineer for the cloud, to hear him talk about the high concurrency problems of distributed clusters.

As a veteran driver with years of work experience, oh no, it is an old employee, in the project actual combat more headaches in one of the several scenes is in the process of concurrency, some ordinary seemingly normal business may be in the concurrent volume is too large problems.

In working to solve such problems, we often use the following methods (practical simple ~\ (≧▽≦)/~).

Practical and simple

1. Use Java Synchronization keyword synchronized lock method or object, but not suitable for distributed environment, just beginning to learn Java more easily think of method.

2. Using database characteristics, database row-level lock, read lock is also a way.

3. Can think of the use of transactions, the size of the transaction, as well as the speed of implementation are required, if the transaction is small enough to execute fast enough, you can.

4. First update in the fetch number, this method is generally used in the inventory scene.

So when a web system access to a large amount of time, to solve such problems are the methods of the value of our work for reference, you can refer to the following scenario.

Load Balancing

Load Balancing (Balance) is an application of cluster technology (Cluster). Load balancing can spread work tasks across multiple processing units, increasing concurrent processing power. The most common load balancing applications at the moment are web load balancing. According to the principles of implementation, common Web load balancing techniques include: DNS polling, IP load balancing, and CDN. IP load Balancing can be implemented using hardware devices or software methods.

Basic principle

Any load-balancing technique would have to find a way to establish a one-to-many mapping mechanism: A requested portal maps to multiple processing requests, thereby implementing divide and conquer (Divide and Conquer).

This mapping mechanism makes multiple physical presence a virtual whole, shielding the internal structure of the requester of the service.

First, DNS

DNS polling is the simplest way to load balance. With domain names as Access portals, multiple DNS A records are configured to allow requests to be assigned to different servers.

DNS polling does not have a fast health check mechanism, and only WRR scheduling policies that support the workload are difficult to "equalize" and are often used for scenarios that are not demanding. And the DNS polling way directly exposes the real address of the server to the user, which is not conducive to server security.

Second, CDN

CDN (Content Delivery network, contents distribution network). The publishing mechanism synchronizes content to a large number of cache nodes and expands on the DNS server to find the nearest cached node of the user as a service provider node.

Because it is difficult to build a large number of cache nodes, it is common to use CDN operator services. At present, the domestic service providers are very few, and according to the flowmeter fee, the price is also more expensive.

Third, IP load Balancing

IP load Balancing is a load balance based on a specific TCP/IP technology. such as NAT, DR, turning and so on. Is the most frequently used way. IP load Balancing can use hardware devices, or you can use software implementations. The main products of hardware equipment is F5-BIG-IP-GTM (referred to as F5), software products are mainly LVs, Haproxy, NginX. One of the LVS, Haproxy can work on the 4-7 floor, nginx work on the 7 floor. Hardware load balancing devices can make the core part of the chip, performance and stability is better, and commercial products manageability, documentation and services are relatively good.

Four, F5

F5 's full name is F5-big-ip-gtm, is the most popular hardware load balancing equipment, its concurrent ability to achieve millions. The main features of F5 include:

Main Features

1. Multi-link load balancing and redundancy: You can access multiple ISP links to achieve load balancing and high availability between links.

2. Firewall load balance: F5 has heterogeneous firewall load balancing and automatic fault elimination ability;

3. Server load balancing: This is the most important function of F5, F5 can be configured for all the external service server configuration Virtual server to achieve load balancing, health Check, reply to maintain, etc.

4. High availability: The redundancy design of F5 equipment can guarantee 99.999% uptime, and the failover time of F5 is millisecond. Using F5, you can configure the entire cluster's link redundancy and server redundancy to improve reliable health-checking mechanisms to ensure high availability.

5. Security: Similar to firewalls, F5 adopts the default rejection policy, which can add additional security to any site, defending against common network attacks, including DDoS, IP spoofing, SYN attacks, teartop and land attacks, ICMP attacks, etc.

6. Easy to manage: F5 provides HTTPS, SSH, Telnet, SNMP and other management methods, including detailed real-time reports and historical reports. Two development kits (I-control) are also available.

7.F5 also provides accessibility features such as SSL acceleration, software upgrades, IP address filtering, bandwidth control, and more.

Caching mechanism

One, use MySQL database internal cache

MySQL's caching mechanism, from from within MySQL, the following content will be the most common InnoDB storage engine.

1. Establishing an appropriate index

The simplest is to build the index, the index in the table data is relatively large, play the role of fast retrieval data, but the cost is also some. First, it takes up a certain amount of disk space, where the combination index is the most prominent and uses caution, and it produces an index that is even larger than the source data. Second, the data insert/update/delete after indexing, and so on, because of the need to update the original index, time will increase. Of course, in fact our system is generally based on select query operations, so the use of indexes still has a significant effect on system performance.

2. Database connection thread pool cache

If every database operation request needs to create and destroy a connection, it is a huge overhead for the database. To reduce this type of overhead, you can configure thread_cache_size in MySQL to indicate how many threads are retained for reuse. When the thread is not enough, then create, idle too much time, then destroy.

3. InnoDB Cache Settings (innodb_buffer_pool_size)

Innodb_buffer_pool_size This is a memory buffer for storing indexes and data, and if the machine is a MySQL-exclusive machine, it is generally recommended as 80% of the machine's physical memory. In the scenario of fetching table data, it can reduce disk IO. In general, the larger the value setting, the higher the cache hit rate.

4. Sub-Library/Sub-table/partition.

MySQL database tables generally bear the amount of data in the millions, and then up, the performance will be a significant decline, so when we anticipate the amount of data will exceed this level, we recommend that the Sub-library/table/partition operations. The best way is to build the service at the beginning of the design as a storage mode of the library, fundamentally eliminate the risk of the middle and late.

Second, build MySQL database multiple services

1. Set up MySQL master, from the library as backup

When the main library fails, switch to from the library. However, this practice is actually a waste of resources, since the library is actually idle.

2. mysql read and write separation, main library write, reading from the library.

Two databases do read-write separation, the main library is responsible for writing the operation of the class, from the library responsible for read operations. And, if the main library fails, still does not affect the read operation, but also can be all read and write temporarily switch to from Couchen (need to pay attention to traffic, may be due to the traffic is too large, from the library also drag down).

3. Mutual preparation of the main owner.

Two MySQL from each other from the library, but also the main library. This scheme not only achieves the pressure diversion of traffic, but also solves the problem of "single point Failure". There is another set of services available for each failure.

However, this scheme can only be used in two machine scenarios. If business development is still fast, you can choose to separate the business and build multiple primary owners.

Iii. data synchronization between MySQL database machines

Whenever we solve a problem, the new problem must be born on the old solution. When we have multiple MySQL, at the peak of the business, it is likely that the data between the two libraries has a latency scenario. Also, network and machine load can affect the latency of data synchronization. We have encountered, in the daily traffic near 100 million in a special scenario, appear, from the library data need many days to catch up with the main library data. In this scenario, the basic loss of utility from the library.

So, to solve the synchronization problem, is the next point we need to focus on.

1. mysql Self-threading synchronization

MySQL5.6 starts to support the main library and synchronizes from the library data, taking multiple threads. However, the restrictions are also more obvious, only in the library as a unit. MySQL data synchronization is through the Binlog log, the main library to write to the Binlog log operations, is in order, especially when the SQL operation contains changes to the table structure, and other operations, for subsequent SQL statement operations are affected. Therefore, synchronizing data from a library must take a single process.

2. Implementation of the resolution Binlog, multi-threaded writing.

In the database table as the unit, parse Binlog multiple tables and do data synchronization. In doing so, the efficiency of data synchronization can be accelerated, but there is also a problem of write order if there is a structural relationship or data dependency between tables and tables. This way, it can be used for some relatively stable and relatively independent data tables.

China's first-line internet companies, most of which are in this way, to speed up data synchronization efficiency. There is also a more radical approach, which is to parse binlog directly, ignoring the direct writing of the table as a unit. However, this approach is complex and is more limited in scope, and can only be used in some special scenarios of the database (no table structure changes, no data dependencies between tables and tables).

Create a cache between the Web server and the database

In fact, to solve the problem of large traffic, you can't just focus on the database level. According to the "28 law", 80% of requests are focused on 20% of hot data. Therefore, we should establish a caching mechanism between the Web server and the database. This mechanism can be used as a disk cache, or in memory caching mode. Through them, most of the hotspot data are queried and blocked before the database.

1. Page static

When a user visits a page on a Web site, most of the content on the page may not change for a long time. A news report, for example, will almost never change the content once it is published. In this case, the static HTML page generated via CGI is cached locally to the Web server's disk. Except for the first time, the local disk file is returned directly to the user after a dynamic CGI query to retrieve the database.

This approach seems perfect when the web system is smaller in size. However, once the web system becomes larger, for example, when I have 100 Web servers. So these disk files, there will be 100, this is a waste of resources, but also difficult to maintain. This time someone will think, can concentrate on a server to save up, hehe, let's look at one of the following caching methods, it is done.

2. Single Memory cache

Through the example of static page, we can know that the "cache" built on the Web machine is not good maintenance, will bring more problems (in fact, through the APC expansion of PHP, through the key/value operation of the Web server's native memory). Therefore, we choose to build the memory caching service, also must be an independent service.

The choice of memory cache, mainly has redis/memcache. In terms of performance, the difference between the two is small, from the functional richness of the extent, Redis more than a chip.

3. Memory Cache Cluster

When we build a single memory cache, we face a single point of failure, so we have to turn it into a cluster. The simple approach is to give him a slave as a backup machine. However, if the request is really many, we found that the cache hit rate is not high, need more machine memory. Therefore, we recommend that it be configured as a cluster. For example, similar to Redis cluster.

Redis cluster cluster Redis each other as a group of master and subordinate, while each node can accept requests, in the expansion of the cluster is more convenient. The client can send a request to any node, and if it is its "responsible" content, it returns the content directly. Otherwise, find the actual responsible Redis node and then notify the client of the address, and the client requests it again.

For clients that use caching services, this is all transparent.

The memory cache service has a certain risk when switching. In the process of switching from cluster A to cluster B, must ensure that B cluster in advance to "Preheat" (b cluster in the memory of the hot data, should be as far as possible with a cluster same, otherwise, switch the moment a large number of requests, in the B cluster memory cache to find, the flow of direct impact on the back end of the database service, is likely to cause database downtime).

4. Reduce database "Write"

The above mechanism, all realize reduces the database "reads" The operation, but, writes the operation is also one big pressure. Write operations, though not reduced, can be used to reduce stress by merging requests. At this point, we need to create a change synchronization mechanism between the memory cache cluster and the database cluster.

Change the request into the cache, let the outside query appear normal, and then put these SQL modifications into a queue storage, queue full or at intervals, merge into a request to the database to update the database.

In addition to the above, by changing the architecture of the system to improve the performance of the write, MySQL itself can configure the parameter innodb_flush_log_at_trx_commit to adjust the policy to write to the disk. If machine costs allow, troubleshooting from the hardware level, you can choose older raid (redundant Arrays of independent disks, disk array) or newer SSD (Solid state drives, solid-state drive).

5. NoSQL Storage

Regardless of the database read or write, when the flow of further increase, will eventually reach the "manpower when the poor" scene. The cost of continuing to add machines is high, and it is not always possible to really solve the problem. This time, some of the core data, you can consider using NoSQL database. NoSQL storage, most of them are in the way of Key-value, here it is recommended to use the above described Redis,redis itself is a memory cache, but also as a storage to use, let it directly to the landing data to disk.

In this way, we will be some of the database is frequently read and written data, separated, placed in our newly built Redis storage cluster, and further reduce the original MySQL database pressure, and because the Redis itself is a memory level cache, read and write performance will be greatly improved.

Domestic first-line internet companies, the architecture adopted a lot of solutions are similar to the above scheme, however, the use of the cache service is not necessarily redis, they will have a richer alternative, even according to their own business characteristics to develop their own nosql services.

6. Empty node Query problem

When we have finished building all the services mentioned above, we think that the web system is already strong. We still say that, the new question will come. Empty node queries are data requests that do not exist at all in those databases. For example, I request a query without the presence of personnel information, the system will be searched from all levels of the cache, and finally to the database itself, and then come to a conclusion that is not found, returned to the front end. Because the cache at all levels is not valid for it, this request is very consuming system resources, and if a large number of empty node queries, can impact the system services.

In my past work experience, has been deeply hurt. /(ㄒoㄒ)/~~ Therefore, in order to maintain the stability of the web system, it is necessary to design an appropriate filtering mechanism for the empty nodes.

The way we used to do that was to design a simple record-mapping table. Store existing records and put them into a memory cache so that if there are empty node queries, the cache layer is blocked.

Geographically distributed Deployment

One, core concentration and node dispersion

When a system and service are large enough, you must begin to consider the problem of offsite deployment. Make your service as close to the user as possible. This time, offsite deployment began. Offsite deployments generally follow: core focus, nodes dispersed.

Offsite deployment

1. Core focus: In the actual deployment process, there is always a part of the data and services can not be deployed multiple sets, or the deployment of multiple sets of costs are huge. For these services and data, there is still a set of locations where the deployment location chooses a geographically comparable center and communicates with each node through the network's internal line.

2. Node decentralization: The deployment of a number of services to multiple sets, distributed in various urban nodes, so that users request to choose as close as possible node access services.

Second, node disaster tolerance and overload protection

Node disaster tolerance means that if a node fails, we need to establish a mechanism to ensure that the service is still available. There is no doubt that the most common disaster-tolerant approach here is to switch to nearby city nodes.

Overload protection, refers to a node has reached the maximum capacity, can not continue to receive more requests, the system must have a protection mechanism. A service is already full load, also continue to accept new requests, the result is likely to be downtime, affect the entire node of the service, in order to protect at least the majority of users of normal use, overload protection is necessary.

Solution overload protection, general 2 directions:

Overload protection

1. Denial of service, after the full load is detected, the new connection request is no longer accepted. For example, online login in the queue.

2. Shunt to other nodes. In this case, the system is more complex and involves the problem of load balancing.

Avoid concurrency

Some of the scenarios mentioned above can solve our concurrency problems in a distributed environment, but if there are concurrent problems that are difficult to solve by technology or are costly to solve, we should first consider whether we can avoid concurrency through certain policies and business designs. For example, through reasonable time scheduling, avoiding the access conflicts of shared resources. In addition, in the design of parallel tasks, it is possible to ensure that there is no shared resource between tasks and tasks through appropriate policies.

Serialization

Sometimes serialization can result in concurrent problem operations, sacrificing performance and scalability to meet data consistency requirements. For example, the distributed message system can not guarantee the order of messages, but the distributed message system may be changed to a single system can guarantee the order of the message. In addition, when the receiver is unable to handle the call ordering, it is possible to cache the invocation information in a queue and then process the calls serially.

In short, the system optimization of the road, but also in the work of continuous learning, and constantly explore. Well, there are good ways to keep updating, and continue to write code. /(ㄒoㄒ)/~~

"Cloud Guardian" is the dominant brand of a series of security products which is the best Information Technology Co., Ltd. to provide customers with a full range of integrated, operational security products, including network attack tracking traceability system, network attack detection and Analysis system, network traffic situation awareness system, security threat situational awareness system, security operation support System, Distributed anti-Denial service attack system and so on.