About the architecture of high-performance and high-concurrency websites with tens of millions of PVS/IP addresses

Original Author:Old boy linux O & M training institution old boy QQ: 31333741
Note: A few months ago, the old boy sent a similar article. This time, I have improved my website architecture to participate in an activity invited by a friend.

Architecture diagram:

The core principle of High-concurrency access is to put forward all user access requests as far as possible ".

If we compare the visiting users to the "Enemies" of the criminals, we must block them out of 800 miles, that is, they cannot send requests to our headquarters (the headquarters is database and distributed storage ).

For example, users who can be cached locally on their computers should not be allowed to access CDN. If CDN servers can be cached, do not allow CDN to access the source (static server. If you can access static servers, do not access dynamic servers. Similarly, you must not access the database or storage without accessing the database or storage.

It is easy to say, but it is not easy to do. But it can be done with a little effort. Does Google's Daily Independent IP address exceed million? Our tens of millions of PV sites are no longer in a hut than Google. Let's start from our hut first! Haha! The following content starts with a PV site of tens of millions, and supports the website architecture of hundreds of millions of PVS.

High performance, high concurrency, and scalable website architecture access:

Some people may ask, how do we push forward the user's access to the business? Where is it? Next, the old boy will come here for everyone.

First, use Apache mod_deflate to compress and transmit data in the user's browser. For example, the expires, deflate, and expires functions are well utilized, this will greatly improve user experience, reduce website bandwidth, and reduce the pressure on backend servers. Of course, there are still many methods. I will not discuss them here.

Tip: software such as nginx/lighttpd for compression transmission and expires functions are also available.

Layer 2: page elements, slice/js/css, and other static data html. This layer is the webpage cache layer, such as CDN (the effect is better than the company's own squid/nginx deployment, they are more professional and cost-effective, such as fast network and CC (the price is 80 RMB/M/month or even lower) and cover more city nodes ), set up squid/nginx cache on your own for small-scale CDN selection (a very large-scale company may consider the risk issue to implement a combination of self-built and purchased services), unless it provides data source services for the front-end CDN, to reduce the data and Storage pressure on Our backend servers, rather than directly providing the cache service to end users. Taobao CDN once caused a high CDN pressure due to the size of some images, and even reduced the size of images to reduce traffic and bandwidth.

Tip: You can also set up a cache layer to provide data source services for the CDN we purchased. available software includes varnish, nginx, squid, and other caches, to reduce the pressure on the layer 3 static data layer. At this layer, we can also set up DNS servers to achieve cross-data center business expansion and smart resolution.

Layer 3: The static server layer is generally the image server, Video Server, and static HTML server. This layer is the connection link between the front cache layer and the dynamic server layer. News and other content published by large companies are directly distributed to various cache nodes by publishers (sina, 163, etc ), this may be different from the business of general companies. Therefore, there is no way to directly reference imitation, such as SNS for everyone.

We can use the Q queue method to achieve asynchronous distribution of access, while static storage of dynamic published data (data in the database. That is to say, you can access the database at the current layer or publish it to the cache nodes in other ways, instead of allowing all users to access the database directly, there are hundreds of thousands of news comments on the qq.com portal. If all users load all comments as soon as they read the news, it is strange that the database is not suspended. Their comments need to be reviewed (the naming convention is actually asynchronous, and the comments may be static or similar to static or memory cache ), this may be the site study like 51cto.com. When you open a blog post on 51CTO, you will find that the following comments are always displayed, or they may be paginated. However, the databases should be read directly. Once the access volume is high, the database pressure is inevitable. This is not to say that the 51cto website is not good. All websites are developed from a similar program architecture. The same may be true for CU.

Tip: we can set up a cache layer on the front end of the static data layer to provide data source services for the CDN we purchased. available software includes varnish, nginx, and squid caches. At this layer, we can also set up DNS servers to achieve cross-data center business expansion and smart resolution.

Layer 4: dynamic server layer: php, java, etc. The database and storage devices can be accessed only when the access requests passed through the first three layers reach this layer. After filtering through the first three layers of access, there are usually very few access requests to this layer. Generally, they are newly published content and newly published content first browsed, such as blog posts (including Weibo posts) and BBS posts.

Note: This layer can be used to write more articles on the program, such as accessing the cache layer, memcache, memcachedb, tc, mysql, and oracle in a program to implement distributed access and distributed read/write splitting, each db cache node with distributed access at the program level can be a Server Load balancer for multiple servers split from a group of businesses or a group of businesses. This architecture will greatly reduce the pressure on the subsequent databases and storage layers. Here, it is equivalent to the outer layer of the headquarters.

Layer 5: database cache layer, such as memcache, memcachedb, and tc.

Select a database suitable for a specific business based on different business needs. For memcache, memcachedb ttserver, and nosql databases, You can implement distributed access to this layer through the program on the fourth layer, each distributed access node may be a set of Load Balancing (dozens of machines ).

Layer 6: database layer. Generally, mysql Master/Slave structures are used for non-large sites, such as 163, sina, and kaixin. The program layer implements Distributed Database read/write splitting, one-master (or dual-master) Multi-slave mode, with a large access volume, you can perform cascading master-slave and cyclic multi-master-Multi-slave mode, and then achieve multi-group load balancing, for the frontend distributed program to call, if the traffic volume is high, you need to split the business. For example, when I work for another enterprise as a part-time employee, I find a site similar to 51cto and use the www Service, the blog service and bbs service are all placed on one server, and then the master. In this case, when the business traffic is high, you can simply split the www, blog, and bbs services with a group of servers. This operation and maintenance method is not difficult. Of course, when the traffic volume is high, you can continue to split a service, for example, www database splitting. Each database performs a set of load balancing, and you can also split the tables in the database. High availability can be achieved through tools such as drbd. For MYSQL with a large write volume, you can use the primary or multi-master mysql rep mode. For ORACLE, it is enough to use several oracle DG groups (one master and multiple salve methods, the 11g DG can support read/write splitting like mysql rep. Of course, the optional solutions include mysql cluster and oracle RAC. Playing with mysql cluster and oracle RAC requires more hardware and a large number of maintenance costs after deployment, the traffic here is huge, so congratulations, at least tens of millions or even hundreds of millions of PVS.

In addition to conventional mysql and oracle Database databases, giant companies like Baidu will use nosql databases in areas with higher performance requirements, and then add DNS and Server Load balancer at the front end, distributed read/write splitting still involves splitting businesses and databases ,... Step by step, and then each point can be a group or multiple sets of machines.

Note: The hardware quality at the database layer also determines the amount of access traffic, especially the disk I/O issue. Large companies often write articles on cost effectiveness, for example, the core business uses the hardware netapp/emc and san Optical Fiber architecture. For resource data storage and video, sas or solid state ssd disks are used. If the data is too large, you can use the hotspot shard storage method: for example, the most frequently accessed 10-20% uses ssd storage, and the 20-30% in the middle uses sas disks, the last 40-50% can use cheap sata.

Layer 7: If the design of tens of millions of PV sites is reasonable, one or two NFS servers are enough. I have maintained (part-time) or experienced tens of millions of PVS and used NFS and common servers for storage. There are still a large number of disks, such as SAS's 15 K * 6, you can also use dell6850 to create several NFS storage groups, which is sufficient for Small and Medium websites. Of course, it can be made into drbd + heartbeat + nfs + a/.

If you can meet the design requirements of this article, medium-scale websites, back-end databases and Storage pressure will be very small. For portal websites, such as XX, hardware storage devices such as hardware netapp and emc, or san optical fiber cables, or even cost-effective articles, for example, the core business uses the hardware netapp/emc and san Optical Fiber architecture. For resource data storage and video, sas or solid state ssd disks are used, you can use the hotspot shard storage method: for example, the most frequently accessed 10-20% uses ssd storage, and the 20-30% in the middle uses sas disks, the last 40-50% can use cheap sata.

Giant companies such as XX will adopt a distributed storage architecture such as hadoop. The front-end will be split based on the service by adding multi-layer CACHE and multi-level load balancing, such as crawler-layer storage, index layer storage, service layer storage... It can be finer and finer... All measures are used to cope with the pressure.

Special services, such as some SNS portal sites, including comments and microblogs of portal sites, are mostly written asynchronously, that is, there are very few concurrent accesses to the database regardless of reading and writing.

If the above Layer 1-7 is set up, the access from the dynamic server layer on the fourth layer will be skipped. Generally, medium websites do not impose too much pressure on databases. Distributed access at the program layer is evolving from 10 million and million PVS. Of course, special services also require special architectures to make rational use of databases and storage.