System Architecture of High-concurrency and high-load websites

A small website, such as a personal website, can be implemented using the simplest html static page. With some images for beautification, all the pages are stored in a directory, such websites have very simple requirements on system architecture and performance. With the increasing diversity of Internet services, website-related technologies have been subdivided into many aspects after years of development, especially for large websites, the technology is widely used. From hardware to software, programming language, database, WebServer, firewall and other fields, there are high requirements, it is no longer comparable to a simple html static website.

Large websites, such as portal websites. In the face of a large number of user access and high concurrency requests, the basic solution focuses on the following aspects: use high-performance servers, high-performance databases, high-efficiency programming languages, and high-performance Web containers. However, in addition to these aspects, it is impossible to fundamentally solve the high load and high concurrency problems faced by large websites.

The solutions provided above also mean a greater investment to a certain extent, and these solutions have bottlenecks and do not have good scalability, I will talk about some of my experiences from the perspectives of low cost, high performance, and high scalability.

1. HTML static
As we all know, the most efficient and least consumed html pages are purely static html pages, so we try our best to make the pages on our website adopt static pages, this simplest method is actually the most effective method. However, for websites with a large amount of content and frequent updates, we cannot manually implement them all, so we have a common information publishing system CMS, news channels such as the portals we often visit, and even other channels, are managed and implemented through the information publishing system, the information publishing system can automatically generate static pages based on the simplest information input. It can also provide channel management, permission management, automatic crawling, and other functions. For a large website, having an efficient and manageable CMS is essential.

In addition to portal and information publishing websites, static websites with high interaction requirements are also a necessary means to improve performance, static Community posts and articles in real time, and re-static updates are also a lot of use strategies, such as Mop, the same is true for the Netease community. At present, many blogs also achieve static
At the same time, html static is also a method used by some caching policies. For applications that frequently use database queries but have little content updates in the system, you can consider using html static, for example, the public setting information of the Forum in the forum. Currently, mainstream forums can perform background management and store the information in the database. In fact, this information is frequently called by foreground programs, but the update frequency is very small, you can consider static content during background updates to avoid a large number of database access requests.

When making html static, you can use a compromise method, that is, the front-end uses dynamic implementation to regularly perform static and timed judgment calls under certain policies, this allows for a lot of flexible operations. I used this method to develop a billiards website. I set some static html time intervals to cache dynamic website content, to share most of the pressure on static pages, it can be applied to the architecture of Small and Medium websites. Address of the home site: By the way, many friends who like Billiards can support my free website :)

2. image server Separation
As we all know, for Web servers, images, whether Apache, IIS or other containers, consume the most resources. Therefore, it is necessary to separate images from pages, this is basically a strategy adopted by large websites. They all have independent image servers and even many image servers. This architecture reduces the pressure on the server system that provides page access requests and ensures that the system will not crash due to image problems.

Different configuration optimizations can be performed on the application server and image server. For example, Apache may support as few loadmodules as possible when configuring ContentType, ensures higher system consumption and execution efficiency.

My billiards website also uses the separation of image server architecture. Currently, it is only the separation of architecture and physical isolation, because there is no money to buy more servers :), you can see that the image links on the old people are URLs similar to img.9tmd.com or img1.9tmd.com.

In addition, lighttpd can be used to replace Apache for processing access to static pages, images, and js. It provides more lightweight and efficient processing capabilities.

3. Database Cluster and database table hash
Large websites have complex applications, and these applications must use databases. In the face of a large number of accesses, database bottlenecks will soon become apparent. At this time, a database will soon fail to satisfy applications, therefore, we need to use a database cluster or database table hash.

In terms of database clusters, many databases have their own solutions, and Oracle and Sybase all have good solutions. The commonly used MySQL Master/Slave is also a similar solution, you can refer to the corresponding solutions to implement the database.

As the database cluster mentioned above is limited by the DB type used in terms of architecture, cost, and expansion, we need to consider improving the system architecture from the perspective of applications, database table hashing is a common and most effective solution. We install business and application or function modules in the application to separate the database. Different modules correspond to different databases or tables, then, according to a certain policy, conduct a smaller database hash for a page or function, such as a user table and table hash by user ID, in this way, the system performance can be improved at a low cost and the scalability can be improved. Sohu's Forum adopts this architecture to separate the database of Forum users, settings, posts, and other information, and then hash the databases and tables of posts and users according to sections and IDs, the simple configuration in the configuration file allows the system to add a low-cost database at any time to supplement the system performance.

4. Cache
The word cache has been used in many areas. The cache in website architecture and website development is also very important. Here we will first describe the two most basic caches. Advanced and distributed caching are described later.

For architecture caching, anyone familiar with Apache can know that Apache provides its own mod_proxy cache module, or use an additional Squid for caching, both methods can effectively improve Apache's access response capabilities.

The Memcached provided on Linux is a common cache solution. Many web Programming Languages provide memcache access interfaces, which are available in php, perl, c, and java, it can be used in web development. It can cache data, objects, and other content in real time or Cron, with flexible policies. Some large communities use this architecture.

In addition, when developing using the web language, various languages basically have their own Cache modules and Methods. PHP has the Pear Cache module and the eAccelerator acceleration and Cache module, well-known Apc and XCache (developed by Chinese people, supported !) Php cache module, more Java.

5. Images
Images are often used by large websites to improve performance and data security. The image technology can solve the differences in user access speed caused by different network access providers and regions, for example, the difference between ChinaNet and EduNet has prompted many websites to set up image sites in CERNET to regularly update or update data in real time. In terms of image details, I will not elaborate too deeply here. There are many professional off-the-shelf solutions and product options. There are also low-cost software implementation ideas, such as rsync on Linux and other tools.

6. Server Load balancer
Server Load balancer is the ultimate solution for large websites to solve high-load access and a large number of concurrent requests.

Server Load balancer has been developing for many years. There are many professional service providers and products to choose from. I personally have some solutions, including two architectures for your reference. In addition, we will not talk much about the primary Server Load balancer DNS round robin and the professional CDN architecture.

6.1 hardware layer-4 Switching
The layer-4 Exchange uses the header information of the layer-3 and layer-4 information packets to identify business flows based on the Application interval and distribute the business flows of the entire interval segment to appropriate application servers for processing. The layer-4 switching function is like a virtual IP address pointing to a physical server. Its transmission services are subject to a variety of protocols, including HTTP, FTP, NFS, Telnet, or other protocols. These services require complex load balancing algorithms based on physical servers. In the IP address world, the service type is determined by the TCP or UDP port address of the terminal. The application interval in the layer-4 switch is jointly determined by the source and terminal IP addresses, TCP and UDP ports.

In the field of hardware layer-4 switching products, there are some well-known products to choose from, such as Alteon and F5. These products are expensive, but value for money, it provides excellent performance and flexible management capabilities. Yahoo China used three or four Alteon servers on nearly 2000 servers.

6.2 software layer-4 Switching
After learning about the principle of the hardware layer-4 switch, the four-layer switch based on the OSI model came into being. Such a solution achieves the same principle, but has a poor performance. However, it is easy to meet a certain amount of pressure. Some people say that the software implementation method is actually more flexible, and the processing capability depends entirely on the familiarity of your configuration.

We can use LVS, which is commonly used in Linux for software layer-4 Switching. LVS is a Linux Virtual Server. It provides a real-time disaster response solution based on heartbeat to improve system robustness, at the same time, it provides flexible virtual VIP configuration and management functions to meet a variety of application needs at the same time, which is essential for distributed systems.

A typical load balancing strategy is to build a squid Cluster Based on layer-4 software or hardware exchanges. This idea is adopted on many large websites, including search engines, this architecture is low-cost, high-performance, and highly scalable. It is easy to increase or decrease nodes in the architecture at any time. I have prepared a special detail for this architecture and will discuss it with you.

6.3 layer-7 Switching
As we all know, layer-4 Exchange is based on the transport layer. At this layer, you can only manage connections, but cannot associate them with services, generally, only tcp and udp connections can be processed. The real business logic needs to be handled by the server group. With the development of technology, today, we have seen layer-7 switching in many advanced applications.

Layer-7 switching is implemented based on the layer-7 Application Layer of TCP/IP. At this layer, We can first differentiate specific applications, such as HTTP, TELNET, FTP, and DNS, policy management can also be performed based on the content transmitted in the application. For example, we have two website paths: a.com/music /... And a.com/photo /... Originally, requests from these two URLs can only be distributed to the next group of Servers Based on layer-4 switching. However, layer-7 switching can determine whether the accessed url is music/or photo, then they are distributed to different server groups to achieve more flexible system architecture design.

Of course, layer-7 switching is also divided into hardware and software implementation methods. Here I will not elaborate on the hardware, such as F5 and Nortel, and Haproxy. Of course, the performance of layer-7 switching software is far different from that of hardware. You must know that these hardware are expensive.

Summary:
For large websites, each method mentioned above may be used at the same time. This is a simple introduction. You need to familiarize yourself with many details in the implementation process, sometimes a very small squid parameter or apache parameter setting will have a great impact on the system performance. I hope you can discuss it together to make it easier.

　　

When talking about the performance impact of images, most of them are from the fact that most of our webpage images often occupy more traffic than html code. With the same network bandwidth, image Transmission takes a longer time, because it takes a lot of time to establish a connection, which will prolong the http connection time between the user client and the server. For apache, concurrency performance will definitely drop, unless all your responses are static, You can take httpd. keepAlives in conf is off, which can reduce the connection processing time. However, if there are too many images, the number of established connections increases and performance is also consumed.

In addition, the theory we mentioned is more about cases of large clusters. In such an environment, image separation can effectively improve the architecture, and thus affect the performance improvement, you need to know why we want to talk about architecture? The architecture may be designed for security, resource allocation, and more scientific development and management, but the ultimate goal is performance.

In addition, it is easy to find descriptions about Mime Type and Content length in RFC1945's HTTP document, which makes it easy to understand the effect of images on performance.

For a large site, its database and Web Server are generally distributed and deployed in multiple regions. When a user accesses a region, they correspond to one node, if the post in the Community is static in real time and the post is static again when there is an update, how can we synchronize the posts between nodes immediately? How is the database implemented? If the user does not see it, the post will fail? How can I lock a user on a node when the user sends a duplicate message?

Locking a user on a node is implemented through layer-4 Switching. In general, if the application is small, it can be implemented through program code. Large applications generally manage user connections through layer-4 exchanges like LVS and hardware. policies can be formulated to keep user connections on a node during the life cycle.

There are many static and synchronous strategies. The general method is centralized or distributed storage, but static storage is implemented through centralized storage, then, the front-end proxy group is used to implement caching and relieve pressure.

Generally, for a medium-sized website, there are a lot of interactive operations, with a daily PV of about 1 million. How can we make a reasonable load?

If there are many interactions, you can consider adding Memory Cache to the cluster to put the constantly changing and synchronized data into the Memory Cache for reading, the specific solution must be analyzed based on the specific situation.

 

If a website is in technical development, which of the following optimization methods should be implemented first?
In terms of cost (technical, human, and financial cost), which implementation can achieve the maximum effect?

Starting with server performance optimization and code performance optimization, including webserver, dbserver configuration optimization, and html static configuration, these steps strive to maximize utilization first, then we should consider increasing investment in the architecture, such as clusters and Server Load balancer, which should be considered appropriately after a certain amount of development and accumulation.

This img.9tmd.com is a virtual host. That is to say, it is a service provided by apache. Does this make sense for performance improvement? Or is it just a foreshadowing to facilitate future physical separation?

Because there is only one server at present, physical isolation cannot be achieved physically. Currently, virtual hosts are used for the purpose of flexible program design and website architecture, if I have a new server, I only need to pass the image or synchronize it, And then resolve the dns of img.9tmd.com to the new server, which naturally achieves separation, if we do not implement the separation from the architecture and program, it will be more painful in the future :)

The question about how to invest as low as possible in the early stage of the website to maximize the performance and plan the later stage of the system architecture can be said to have been magnified beyond the technical scope, however, there are still many technical considerations.

The key to planning a website is to plan its stage objectives, such as predicting the user level, storage level, and number of concurrent requests in a few months. What will happen in a few months, these predictions must be estimated and constantly adjusted based on specific business and market conditions. With these forecast data as a reference, you can plan the technical architecture, otherwise, the architecture cannot be properly designed technically.

Based on the website development plan, what applications should be provided in the future? What is the domain name relationship? What is the business logic and association between applications? What regions do Users provide services? And so on...

The above problems help to plan website servers and device investment, and predict technically what architecture will be in the future, what are the conditions that each node in this architecture needs to meet? They are the requirements of the initial architecture.

In general, technical planning without combining specific services is meaningless. Therefore, first, business planning, that is, product design, and then technical planning.

Image server and master data separation issues.
Are images stored on hard disks or in databases?
Please explain the following questions in two cases: hard disk and database.
What should I do when the server capacity for storing images cannot meet the requirements?
What should I do when the server load for storing images cannot meet the requirements?

It must be stored in the hard disk. The statement stored in the database actually comes from some personal websites or enterprise websites that use virtual hosts or lease space. Because the website data volume is small, it is also convenient to back up data, for large commercial websites, there are no large applications where images are stored in databases. Both the database capacity and efficiency are bottlenecks.

The two questions you mentioned later. Capacity and load are basically considerations at the same time. In terms of capacity, most of the solutions use massive storage, such as professional disk arrays, entry-level cabinet, advanced optical fiber disk array, lan disk array, etc. These are the main solutions. As I said before, if low costs are considered, you must use a cheap single server for storage, and you need to control the program logic. For example, you can store multiple identical servers, the NFS partitions are provided for front-end applications respectively. In the program logic of the front-end applications, you can control the NFS partitions on which servers are stored, for example, hash is performed based on Userid, image id, or other logic, which is similar to the logic for storing hash tables or database shards in large databases.

There are two solutions for High-load basic slice: front-end squid caching and images. By using images for storage devices (servers or disk arrays), they can be distributed to multiple servers to provide image services, then, the squid cache is used to reduce the load and increase the user access speed.

1. image server Separation
I have always supported this idea. Especially if the program and the image are placed on the same APAHCE server, each image request may lead to an HTTPD process call. If HTTPD contains a PHP module, it will occupy too much memory, and this is not necessary.

Using an independent image server not only avoids the above situation, but also sets different expiration times for images of different usage properties, so that the same user will not retrieve data from the server (based on the Cache Server) again when accessing the same image on different pages, not only fast, but also saves bandwidth. In addition, you can adjust the cache time.

In the image server I managed in the past, not only does it separate images from applications and pages, but it also enables different domain names for different types of images. To ease the pressure on images of different types. For example, the photo.img.domain.com domain name is used for photography services and uses five caches at ordinary times. However, after a 5.1 long holiday, it may need to be added to ten independent hosts. These 5 servers can be temporarily transferred from other low-load image servers.

2. Database Cluster
The cluster layout of a set of oracle rac is about 40 W. This price is unnecessary for general companies. Because the WEB application logic is relatively simple, and the value of ORACLE large databases lies in data mining, rather than simple storage. Therefore, MySQL or PostgreSQL is actually used.

Simple MySQL replication can achieve better results. Read from SLAVE and update to MASTER only when writing. In actual situations, MySQL's replication performance is very good, basically it will not bring too high update latency. Using the balance (http://www.inlab.de/balance.html) software, listening to port 3306 locally (127.0.0.1), and then ing multiple SLAVE databases, you can achieve read load balancing.

3. Are images stored on disks or databases?
I have carefully considered this issue. If the ext3 file system is used, the limit is reached when directories are created, but xfs is not used. If you need to store a large number of images, you must separate them into many small directories. Otherwise, there will be a limit that ext3 can only create 3 W directories, in addition, too many files and directories will affect disk performance. There are no problems such as space waste.

More importantly, data backup for a large number of small files takes a lot of resources and a very long time. Before these problems, it may be another option to save images in the database.

You can try to save the image to the database, and use the PHP program to return the actual image at the front end, and then place a SQUID server at the front end to avoid performance problems. You can also use the data replication mechanism of MySQL to back up images. This problem can be effectively solved.

4. I will not talk about static pages, and my own wordpress will be completely static. At the same time, it can well take into account the generation of dynamic data.

5. Cache
I have also proposed using memcached before, but it is not very ideal in actual use. Of course, inconsistent use results may occur in different application environments, which is not important. As long as you think it is easy to use.

6. layer-4 software exchange
The LVS performance is very good. My friend's website uses LVS as the Scheduler for balancing. The data volume is very large and can be easily supported. Of course, DR is used.

In fact, I also thought about using LVS for CDN scheduling. For example, the BGP Data Center in Beijing accepts user requests, and then dispatches requests to the actual physical servers of the Telecom or China Netcom data center in the TUN mode of LVS, directly returning data to the user.

This is WAN scheduling, and F5 hardware devices also use this technology. However, the cost of using LVS is greatly reduced.

 

Refs: http://www.cnblogs.com/chenlulouis/archive/2009/07/19/1526498.html