I'll talk about it. Large Web site architecture (i)

Original: http://www.cnblogs.com/chenlulouis/archive/2009/07/19/1526498.html

The system architecture of large-scale high-concurrency high-load web site

A small website, such as personal website, can use the simplest HTML static page to achieve, with some pictures to achieve beautification effect, all the pages are stored in a directory, such a site on the system architecture, performance requirements are very simple, with the Internet business is constantly enriched, Website related technology After these years of development, has been subdivided into very fine aspects, especially for large sites, the use of technology is very wide, from hardware to software, programming languages, databases, WebServer, firewalls and other fields have a very high requirements, is not the original simple HTML static site can be compared.

Large Web sites, such as portals. In the face of a large number of user access, high concurrent requests, the basic solution is focused on a number of aspects: the use of high-performance servers, high-performance databases, high-efficiency programming language, as well as high-performance web containers. But in addition to these aspects, there is no way to solve the large-scale web site is facing high load and high concurrency problems.

The above offers a few solutions to a certain extent also means a greater input, and such a solution to the bottleneck, not very good extensibility, below I from the low cost, high performance and high expansion of the perspective of my experience.

1. Static HTML
In fact, we all know that the most efficient, the least expensive is the pure static HTML page, so we try to make the page on our site using static pages to achieve, the simplest method is actually the most effective method. But for a lot of content and frequently updated sites, we can not all manually to achieve, so we have a common information distribution system CMS, like we often visit the various portals of the news channel, and even their other channels, are through the information distribution system to manage and implement, Information Publishing system can achieve the simplest information input automatically generated static pages, but also with channel management, rights management, automatic capture and other functions, for a large web site, has a set of efficient, manageable CMS is essential.

In addition to the portal and the type of information publishing site, for the interactive requirements of the Community type site, as much as possible static is also to improve the performance of the necessary means, the community posts, articles in real-time static, there is a renewal of the time and re-static is a lot of use of the strategy, A hodgepodge like mop is the use of such strategies, such as the NetEase community. At present, many blogs have also been implemented statically
At the same time, HTML static is also the use of some caching policies, for the system frequently using database queries but the content of small updates, you can consider the use of HTML static, such as forum public settings information, This information is currently the mainstream forum can be managed in the background and stored in the database, which is actually a lot of the foreground program calls, but the update frequency is very small, you can consider this part of the background update the time to static, so as to avoid a large number of database access requests.

In the process of HTML static can use a compromise method, that is, the front-end use of dynamic implementation, under certain strategies for the timing of static and timing judgment call, this can achieve a lot of flexibility of operation, I developed the billiard site of the People (http://www.8zone.cn/) It is using this method, I set some HTML static time interval to cache the dynamic site content, to share most of the pressure on the static page, can be applied to the architecture of small and medium-sized web sites. Home Site Address: http://www.8zone.cn/, by the way, there is a lot of friends like billiards support me this free site:)

2, Image server separation
You know, for the Web server, whether it is Apache, IIS or other containers, the picture is the most consumption of resources, so we have to separate the picture and the page, which is basically a large site will adopt the strategy, they have a separate picture server, and even many picture server. Such architectures can reduce the pressure on the server system that provides page access requests and ensure that the system does not crash due to picture problems.

In the application server and picture server, can be different configuration optimization, such as Apache in the configuration of contenttype can be as little as possible to support, as little as loadmodule, to ensure higher system consumption and execution efficiency.

My billiard site 8zone.cn also used the image server architecture on the separation, which is currently only architecturally separate, physically not separated, due to no money to buy more servers:), You can see the picture connection on the people's home is similar to img.9tmd.com or img1.9tmd.com URL.

In addition, in dealing with static pages or images, JS and other access, you can consider using LIGHTTPD instead of Apache, which provides a more lightweight and more efficient processing power.

3. Database cluster and library table hash
Large Web sites have complex applications, which must use databases, and in the face of a large number of accesses, the bottleneck of the database can soon be revealed, when a database will soon be unable to meet the application, so we need to use the database cluster or library table hash.

In the database cluster, many databases have their own solutions, Oracle, Sybase and so on have a good solution, the common MySQL provided by the Master/slave is a similar scenario, you use what kind of db, refer to the corresponding solutions to implement.

The database cluster mentioned above is constrained by the DB type used in architecture, cost, and extensibility, so we need to consider improving the system architecture from the perspective of the application, and the library table hashing is the most common and effective solution. We install the business and application in the application or function module to separate the database, different modules corresponding to different databases or tables, and then according to a certain policy on a page or function of a smaller database hash, such as the user table, according to user ID for the table hash, This makes it possible to improve the performance of the system at a low cost and has a good scalability. Sohu Forum is the use of such a framework, the Forum users, settings, posts and other information database separation, and then to the post, the user in accordance with the plate and ID hash database and table, finally can be configured in the configuration file simple configuration will allow the system at any time to add a low-cost database to supplement the system performance.

4. Cache
The word cache has been touched by technology, and caches are used in many places. Caching in the Web site architecture and Web development is also very important. Here we first describe the two most basic caches. The advanced and distributed caches are described later.

Architecture cache, people familiar with Apache can know that Apache provides its own Mod_proxy cache module, can also use additional squid for caching, both of which can effectively improve the access response of Apache.

Website program development Cache, Linux provides memcached is a common caching scheme, many Web programming languages provide memcache access interface, PHP, Perl, C and Java have, can be used in web development, can be in real-time or cron data, Objects and so on, the policy is very flexible. Some large communities have used such architectures.

In addition, in the use of web language development, all kinds of languages have their own cache modules and methods, PHP has pear cache module and eaccelerator acceleration and cache module, but also the well-known APC, XCache (developed by the Chinese, support!). PHP Cache module, Java is more.

5. Mirror
Mirroring is often used by large web sites to improve performance and data security, the mirror technology can solve the different network access providers and geographical user access speed differences, such as the difference between chinanet and edunet prompted a lot of websites in the education network to build mirror site, Data is scheduled to be updated or updated in real time. In terms of mirror detail technology, this is not too deep, there are many professional ready-made solution architectures and products to choose from. There are also inexpensive ways to implement software, such as the tools of Rsync on Linux.

6. Load Balancing
Load balancing will be the ultimate solution for large web sites to address high-load access and a large number of concurrent requests.

Load balancing technology has developed for many years, there are many professional service providers and products can be selected, I personally contacted a number of solutions, including two architecture can give you a reference. In addition, the primary load balancer DNS round robin and the more professional CDN architecture is not much to say.

6.1 Hardware four-layer switching
The fourth layer Exchange uses the header information of the third layer and fourth layer packets, according to the application interval to identify the business flow, the entire interval segment of the business flow distribution to the appropriate application server for processing. The fourth layer switch function is like a virtual IP, pointing to the physical server. It transmits services that comply with a variety of protocols, such as HTTP, FTP, NFS, Telnet, or other protocols. These operations are based on physical servers and require complex load balancing algorithms. In the IP world, the business type is determined by the terminal TCP or UDP port address, and the application interval in layer fourth switching is determined by the source and endpoint IP addresses, TCP, and UDP ports.

In the hardware four-layer switching product area, there are some well-known products to choose from, such as Alteon, F5, etc., these products are expensive, but value for money, can provide very good performance and very flexible management capabilities. Yahoo China at the beginning of nearly 2000 servers using three or four alteon to be done.

6.2 Software four-layer switching
When you know the principle of hardware layer four switch, the software four layer exchange based on the OSI model comes into being, so the solution achieves the same principle, but the performance is slightly worse. But to meet a certain amount of pressure or comfortable, some people say that the software implementation is actually more flexible, the ability to handle the full look at your configuration of the familiar ability.

Software four-layer switching we can use the common LVS on Linux to solve, LVs is Linux Virtual Server, he provides a real-time disaster response based on the Heart Line heartbeat solution, improve the system robustness, At the same time, the flexible virtual VIP configuration and management functions can meet a variety of application requirements, which is necessary for distributed systems.

A typical use of load balancing strategy is to build a squid cluster on the basis of software or hardware four-layer switching, which is adopted on many large Web sites including search engines, which have low cost, high performance and strong extensibility, and it is easy to add or subtract nodes to the architecture at any time. Such a structure I am ready to empty a special detail and discuss with you.

6.3 Seven-layer switching
We all know that TCP/IP seven layer protocol, four layer exchange is based on the transport layer, in this layer can only handle the management of the connection, but can not be associated with the business, usually only for TCP, UDP connection to processing, and the real business logic needs to be processed by the following server group, with the development of technology, today , we have seven layers of switching in many advanced applications.

Layer Seven Exchange is based on the seventh layer of TCP/IP application layer to achieve, in this layer, we can first distinguish between specific applications, such as HTTP, TELNET, FTP, DNS, etc., but also according to the content of the application to carry out the management of the policy, such as we have so two of the path of the site a.com/music/... and a.com/photo/... Originally based on the four-layer exchange can only send these two URL requests to the subsequent set of servers, but the seven-tier exchange can determine whether to access the music/or photo/path, and then distributed to the non-reachable server farm, thus achieving a more flexible system architecture design.

Of course, the seven-layer exchange is also divided into hardware and software implementation, where I do not elaborate, the hardware has the famous F5, Nortel, etc., the software has haproxy, of course, seven layers of software is still in the performance of the hardware is far from the difference in the implementation, you know, these hardware are expensive

Summarize:
For large sites, each of the previously mentioned methods may be used at the same time, the introduction is relatively simple, the implementation of a lot of detail in the process also need to be familiar with and experience, sometimes a very small squid parameters or Apache parameter settings, the impact on the system performance will be very large, I hope that we will discuss together to achieve the effect.

　　

We talk about the impact of the picture on the performance of the problem, in general, most of our visit page images are often more than the HTML code occupied by the traffic, in the same network bandwidth, the picture transmission takes longer, because the transmission needs to spend a lot of money on the connection, This will extend the user Client side and server side of the HTTP connection length, which for Apache, concurrency performance will certainly decline, unless your return is all static, it can be httpd.conf in the keepalives is off, which can reduce the connection processing time , but if too many pictures can cause more connections to be established, it also consumes performance.

In addition, the theory we mentioned is more about the case of large clusters, in such circumstances, the separation of images can effectively improve the structure, and thus affect the performance of the improvement, you know why we talk about architecture? The architecture may be for security, for resource allocation, and for more scientific development and management, but the end as far as are for performance.

It is also easy to find descriptions of the MIME type and content length sections in the HTTP protocol documentation for RFC1945, which is easy to understand the performance impact of a picture.

For large sites, his database and Web Server is generally distributed, in a number of regions have been deployed, when a user in a region will be mapped to a node, if the community posts in real-time static, there are updates and re-static, then how to synchronize between nodes immediately? How is the database side implemented? If the user does not see the words will think the post failed? Causes the duplicate to send, then how to lock the user on one node?


For a user to lock on a node is implemented through a four-layer exchange, generally so, if the application is relatively small can be implemented through program code. Large applications typically manage user connections through a four-layer switch like LVS and hardware, and policies can be developed to keep a user's connection on a node for a lifetime.

Static and synchronous strategy is more, the general approach is to centralize or distribute storage, but static is achieved through centralized storage, and then use the front-end proxy group to achieve the cache and share the pressure.

Generally for a medium-sized website, interactive operation is very much, day PV million, how to do a reasonable load?

If the interaction is very much, you can consider using the cluster memory cache method, the constantly changing and need to synchronize the data into the memory cache for reading, the specific scenario needs to be analyzed in combination with specific circumstances.

If a website is in a technological development period, then what should be implemented before these optimization tools are implemented?
Or in terms of cost (technology, human and financial costs), which first implementation can achieve maximum results?

First, from the server performance optimization, code performance optimization, including webserver, dbserver optimization configuration, HTML static and so easy to start, these links strive to extract the maximum utilization, and then consider the increase in the structure of investment, such as clustering, load balancing, etc. All these need to be considered more appropriately after some development and accumulation.

This img.9tmd.com is a virtual host, that is, a service provided by Apache, so that the performance of the improvement is also very meaningful? Or just foreshadowing, in order to facilitate the physical separation later?

Because there is only one server, so physically unable to achieve a real separation, the temporary use of virtual host to achieve, is for the design and Web site architecture flexibility, if there is a new server, I just need to mirror the image past or synchronized past, Then the img.9tmd.com DNS resolution to the new server on the natural implementation of the separation, if not now from the implementation of architecture and procedures, the future of such separation will be more painful:)

about how to spend as low as possible in the early days of the site, to maximize the use of performance, at the same time do a good job of planning the later system architecture, the problem can be said to be magnified beyond the technical category, but the technology-related parts are still a lot to consider.

The planning of a site is critical to the planning of the phased objectives, such as predicting what the user level, storage level, number of concurrent requests, and then in the next few months, these forecasts must be estimated and adjusted according to the specific business and market conditions, with these predictive data as a reference, It is possible to plan the technical architecture, otherwise it is not technically possible to properly design the architecture.

On the basis of website development planning, consider what kind of application will be provided in the future? What kind of domain name relationships do you have? What are the business logic and associations between the various applications? In the face of what geographical distribution of users to provide services? Wait a minute...

These issues help you plan your site's server and device inputs, and technically anticipate what the future will be, and what the requirements for each node under this architecture will be, which is the initial architecture.

Overall, there is no point in combining technical planning with specific business, so the first is business planning, that is, product design, then technology planning.

Image server separation from the master data issue.
is the picture stored on the hard drive good or stored in the database?
Please explain the following questions in two cases: hard disk and database.
What do I do when the server capacity of the image is not met?
What do you do when the server load that holds the picture does not meet the requirements?

Must be stored in the hard disk, appear stored in the database inside the statement is actually from some virtual host or rented space personal website and corporate website, because the site data volume is small, also for backup convenience, from large commercial website, no picture stored in the database of large applications. Database capacity and efficiency can be a huge bottleneck.

You mentioned two questions later. Capacity and load are basically at the same time to consider the problem, capacity, most of the solution is the use of mass storage, such as professional disk array, entry-level disk cabinets or advanced fiber optic disk array, LAN disk array, these are the main solution. Remember I originally said, if you consider the low cost, you must use a cheap single server to store, then you need to control from the logic of the program, such as you can more than the same server to store, respectively, to provide NFS partition for the front-end application to use, In the program logic of the front-end application, you control the NFS partition on which server is stored, such as hashing based on UserID or image ID, or other logic, similar to the logic that we plan to store hash tables or sub-libraries for large databases.

There are two solutions to the high load of the basic piece, front squid cache and mirroring, through the storage device (server or disk array) using the image, can be distributed across multiple servers to provide image services, and then with the squid cache to achieve lower load and improve user access speed.

1, Image server separation
This view is what I have always been very supportive of. In particular, if the program and pictures are placed under the same APAHCE server, each picture request may lead to a HTTPD process call, and HTTPD if it contains a PHP module, it will occupy too much memory, and this is not necessary.

The use of independent image server not only to avoid this situation, but also to different use of the nature of the picture set different expiration time, so that the same user on different pages to access the same image will not be again from the server (based on the cache server) to fetch data, not only fast, and save bandwidth. There is also, for the time of the cache can also be adjusted to adjust.

In my past management of the image server, not only to the image and the application and the page to separate, or for different properties of the image to enable different domain names. To alleviate the pressure of different nature pictures. For example photo.img.domain.com this domain name is for the photographic service, usually uses 5 caches, but after 5.1 long vacation, may need to increase independently for him to 10 sets. The additional 5 units can be transferred from other low-load image servers for temporary use.

2. DB Cluster
A set of ORACLE RAC cluster arrangement around 40W, this price for the general company, is not necessary. Because the WEB's application logic is relatively simple, the value of ORACLE's large databases lies in data mining, not in simple storage. So choosing MySQL or PostgreSQL will be practical.

Simple MySQL replication can achieve better results. Read the time from SLAVE read, write the time only to the MASTER update. In fact, MySQL has a very good replication performance, which basically does not bring too high an update delay. Using balance (http://www.inlab.de/balance.html) software, the local (127.0.0.1) listens on 3306 ports, and then maps multiple SLAVE databases, which can achieve load balancing of reads.

3. Is the picture saved on a disk or a database?
I have also seriously considered this issue. If it is under the Ext3 file system, the 3W directory is built to the limit, and with XFS there is no such limit. Picture storage, if necessary is a large number of save, must be separated into a number of small directories, otherwise there will be ext3 can only build 3W directory limit, and the number of files and the number of directories too many will affect disk performance. Not counting the waste of space.

More importantly, for a large number of small files of data backup, to occupy great resources and very long time. Before these issues, it might be a different option to save the picture in the database.

You can try to save the picture to the database, the front end with the PHP program to return the actual picture, and then put a SQUID on the front end of the server, you can avoid performance problems. The backup problem of the picture can also be achieved by using MySQL's data replication mechanism. This problem can be better solved.

4, the static page I will not say, I do my own WordPress has completely realized the static, at the same time can be a good balance of dynamic data generation.

5. Cache
I have also suggested using memcached before, but the actual use is not very special ideal. Of course, the inconsistent use of each application environment will have inconsistent results, which is not important. Just use it if you feel good about it.

6, software four layer Exchange
LVS performance is very good, I have friends of the site using LVS to do the balance of the scheduler, the amount of data can be easily supported. Of course, the way DR was used.

In fact, I also think I can use LVS to do CDN scheduling. For example, the BGP computer room in Beijing accepts the user's request, then, through the TUN method of LVS, dispatches the request to the actual physical server of the telecommunication or Netcom computer room, and returns the data directly to the user.

This is the WAN dispatch, F5 These hardware devices also apply such technology. However, the use of LVS to achieve the cost is greatly reduced.

I'll talk about it. Large Web site architecture (i)