Analysis of large-scale website system architecture with high operational efficiency and stability

With the acceleration of Informatization for large IT enterprises in China, the data volume and access volume of most applications have increased sharply. Large enterprise websites are facing pressure from performance and high data access volume, it also puts forward higher requirements for storage, security, and information retrieval. Websites visited by tens of millions of people at the same time generally have many databases working at the same time. The white point is Database Cluster and concurrency control. Such websites are also relatively real-time. These websites share some common features: large data volumes, large numbers of online users, high concurrent requests, high pageview, and fast response. The architecture of each major website is summarized, which mainly improves efficiency and stability, including: 1,Program Program Development is on the one hand, and system architecture design (hardware + network + software) is on the other hand. In terms of software architecture, website creation requires a lot of web servers to store static resources, such as movies, videos, and static pages. Do not put static resources together with application servers. A program written by a good programmer is very concise and has good performance. A junior programmer may make many low-level errors, which is also one of the reasons that affect website performance.
Websites must be highly efficient, not just for programmers. Database optimization and program optimization are required. Databases and programs must go hand in performance optimization! Cache also starts at the same time. First, database cache and database optimization are completed by DBA (and this has a great potential to be tapped, But we ignore it because we are all programmers ). Second, program optimization, this is very exquisite. For example, the important point is to standardize SQL statements, use less in OR, and use more preparestatement, in addition, avoid program redundancy, such as using less dual loops to find data. In addition, excellent open-source frameworks are used for support. I personally think that mid-and backend support is the most important. Spring + ibatis can be selected. Because ibatis directly operates SQL and has a caching mechanism. I don't need to talk about the benefits of spring. The IOC mechanism can avoid new objects, which also saves costs. According to my analysis, most of the overhead is generated when new is used and when connected to the database. Please avoid it as much as possible. In addition, you can use some memory testing tools for a demo to illustrate who is faster in Hibernate and ibatis! The front-end can be used for whatever you want, Struts and webwork. If you think you are awesome, try using tapestry. Using databases may not be able to solve the problem caused by huge access volumes. The addressing time for making a static file hard disk may not be less than the database's search time. Of course, it will take another time to index data. I personally think that portals often have a high click rate of popular materials on the day, and the maximum cache is 1 ~ 2 GB of data, for example: ◎ Netease news Http://news.163.com/07/0606/09/3GA0D10N00011229.html
Easy to understand: http: // domain name/year/month/day/category of news/news id.html
The hashtable (key: year-month-day-category-ID, value: News object ), put it in the memory statically (the speed is definitely faster than the hard disk addressing static pages ). Generally, the Oracle stored procedure and two WebLogic engines are used. The update mechanism is similar. Every time a piece of news is issued, a static page is generated and then sent to the front-end web server, the front-end web is used for load balancing. In addition, a scheduled program is automatically generated every 5-15 minutes. Cache data when publishing news. Of course, the cache will not get bigger and bigger, and expired data will be removed in a specific period of time (such as early morning. Building a large website is far less simple than imagined, and there are hundreds of servers. In this way, the processing speed of a computer can be greatly increased. If one machine cannot handle the problem, you can use the httpserver cluster to solve the problem. 2. Network China's networks are divided into China North and South China Telecom and China Netcom. The IP addresses accessed must be differentiated between China North and South China into different networks. 3. Clusters
CDN, gsbl, and DNS Server Load balancer are usually used. Each region has a group of front-end servers, such as Netease And Baidu, which use the DNS Server Load balancer technology. Each channel has a group of front-end servers, one search uses DNS load technology, and all channels share a group of Front-End Server clusters. Websites use Server Load balancer Based on Linux Clusters and fail to recover, including application servers and database servers, service status detection and High Availability Based on Linux-ha.
Application Server clusters can use Apache + Tomcat clusters and WebLogic clusters. Web Server clusters can use reverse proxy, Nat, or multi-domain name resolution. Squid can also be used, there are many methods, which can be selected as needed. 4. Databases
Because tens of millions of people access the website at the same time, many databases usually work at the same time. In other words, database clusters and concurrency control are used to distribute data to different data centers in different geographical locations, to avoid power failure. Another point is that the static Web pages of those websites are not true, but the illusion that dynamic web pages are exchanged with static Web pages, this can be implemented using an open source website er such as urlrewrite. This kind of website is also relatively real-time, because there is a process when copying data in the database, Hibernate and ecache can be used technically, but if you want to make the website work better, you can use a large server such as EJB, websphere, and WebLogic to support and use a large database such as oracle.
MySQL databases are not recommended for large portal websites unless you are very familiar with MySQL DATA optimization. The master-slave mode of the MySQL database server uses the database server to synchronize data between the master and slave servers. The application only writes data to the master server, when reading data, select a server or master server to read data based on the load, and divide the data into different servers (groups) based on different policies to distribute the database pressure.
For large websites, Oracle is used, and data operations should be performed with as many stored procedures as possible to definitely improve performance. At the same time, DBAs should optimize the database, and the optimized database should be different from the unoptimized one; at the same time, distributed databases can be expanded, and more research will be conducted in the future; 5. Page
From the very beginning, consider using a virtual storage/cluster file system. It allows you to access a large number of concurrent Io resources without any restructuring.
Page data calls must be carefully designed. Some data queries can be performed without the database method, while Lucene can be used for real-time requests. Even if real-time requests are required, Lucene can be used, lucene + compass is excellent. News websites can use static page storage and the regular update mechanism to reduce the burden on servers. Each small module on the home page can use the Oscache cache so that data is not pulled every time.
The front-end Web Accelerator based on static page cache. Its main applications include squid. Squid caches most static resources (images, JS, CSS, etc.) and returns them directly to visitors to reduce the load on the application server.
Static web pages of a website are not true, but the illusion that dynamic web pages are exchanged with static Web pages. This can be achieved by using an open-source website er such as urlrewrite, the suffix "htm" or "html" does not indicate that the program has generated a static page. It may be implemented through URL rewriting, just to improve the coverage of your website in the search engine.
The servers that generate static pages and WWW servers are two different groups of servers. After the pages are generated, they will be sent to the WWW server. Some databases are not relational databases, which is more suitable for information derivation, there are many WWW, mail servers, and routers. Load Balancing is mainly used to solve access bottlenecks.
◎ Disadvantages of static pages:
1) increases the complexity of the program.
2) information management is not conducive
3) The speed is not the fastest
4) hard disk injury 6. Cache
Cache should be used from the very beginning. High-speed cache is a better place to store temporary data, such as tracking temporary files generated by a session of a specific user on a web site, you no longer need to record the data in the database. You cannot use Lucene to implement caching. You can use memcached for Distributed caching. If you have money, you can use 10 machines for caching.> 10 Gb of storage space is sufficient; if you have no money, you can work on the page cache and data cache. You can also use Oscache and ehcache, or swarmcache, But it is said that synchronization is not very good; Memcache can be used for caching, and all the unchanged data can be cached with large memory, and cache expiration will be notified when the modification is made. memcache is a distributed cache product developed by LJ, for many large websites, we can install the cache server and app server together. Because the cache server does not consume much CPU, and with the support of the cache server, the app server does not require too much memory, so it can coexist and use resources more effectively. The above immature ideas can be gradually refined from a certain level to improve the product performance indicators. The following content is reprinted: Analysis of the architecture of large websites A small website, such as a personal website, can be implemented using the simplest HTML static page, in combination with some images, all pages are stored in one directory. Such websites have simple requirements on system architecture and performance. With the increasing diversity of Internet services, after years of development, website-related technologies have already been subdivided into very detailed aspects. Especially for large websites, the technologies used are widely used, from hardware to software, programming language , databases, webservers, firewalls, and other fields all have high requirements, it is no longer comparable to a simple HTML static website.
large websites, such as portal websites. In the face of a large number of user access and high concurrency requests, the basic solution focuses on the following aspects: use high-performance servers, high-performance databases, high-efficiency programming languages, and high-performance WEB containers. However, in addition to these aspects, it is impossible to fundamentally solve the high load and high concurrency problems faced by large websites.
the solutions provided above also increase investment to a certain extent, and these solutions have bottlenecks and do not have good scalability, I will talk about some of my experiences from the perspectives of low cost, high performance, and high scalability.
1. HTML static
as we all know, the most efficient and minimum consumption is the pure static HTML page, therefore, we try our best to make the pages on our website adopt static pages. This simplest method is actually the most effective method. However, for websites with a large amount of content and frequent updates, we cannot manually implement them all, so we have a common information publishing system CMS, news channels such as the portals we often visit, and even other channels, are managed and implemented through the information publishing system, the information publishing system can automatically generate static pages based on the simplest information input. It can also provide channel management, permission management, automatic crawling, and other functions. For a large website, having an efficient and manageable CMS is essential.
apart from portal and information publishing websites, for community websites with high interaction requirements, static as much as possible is also a necessary means to improve performance, the Community posts, articles for real-time static, there are also a lot of strategies to use the static method again when there are updates, such as the mop hodgedge is to use such a strategy, Netease community and so on.
at the same time, HTML static is also a method used by some cache policies. For applications that frequently use database queries but have little content updates in the system, you can consider using HTML static, for example, the public setting information of the Forum in the forum. Currently, mainstream forums can perform background management and store the information in the database. In fact, this information is frequently called by foreground programs, but the update frequency is very small, you can consider static content during background updates to avoid a large number of database access requests.
2. image server Separation
As we all know, for Web servers, images, whether Apache, IIS or other containers, consume the most resources. Therefore, it is necessary to separate images from pages, this is basically a strategy adopted by large websites. They all have independent image servers and even many image servers. This architecture can reduce the pressure on the server that provides page access requests, and ensure that the system does not crash due to image problems. Different configurations can be optimized on the application server and image server, for example, Apache supports as few loadmodules as possible when configuring contenttype to ensure higher system consumption and execution efficiency.
3. Database Cluster and database table hash
Large websites have complex applications, and these applications must use databases. In the face of a large number of accesses, database bottlenecks will soon become apparent. At this time, a database will soon fail to satisfy applications, therefore, we need to use a database cluster or database table hash.
In terms of database clusters, many databases have their own solutions, and Oracle and Sybase all have good solutions. The commonly used MySQL Master/Slave is also a similar solution, you can refer to the corresponding solutions to implement the database.
As the database cluster mentioned above is limited by the DB type used in terms of architecture, cost, and expansion, we need to consider improving the system architecture from the perspective of applications, database table hashing is a common and most effective solution. We install business and application or function modules in the application to separate the database. Different modules correspond to different databases or tables, then, according to a certain policy, conduct a smaller database hash for a page or function, such as a user table and table hash by user ID, in this way, the system performance can be improved at a low cost and the scalability can be improved. Sohu's Forum adopts this architecture to separate the database of Forum users, settings, posts, and other information, and then hash the databases and tables of posts and users according to sections and IDs, the simple configuration in the configuration file allows the system to add a low-cost database at any time to supplement the system performance.
4. Cache
The word cache has been used in many areas. The cache in website architecture and website development is also very important. Here we will first describe the two most basic caches. Advanced and distributed caching are described later.
For architecture caching, anyone familiar with Apache can know that Apache provides its own cache module, or use the plus squid module for caching, both methods can effectively improve Apache's access response capabilities.
The memory cache provided on Linux is a common cache interface that can be used in Web development, for example, during Java Development, you can call memorycache to cache and share data, and some large communities use this architecture. In addition, when using Web language development, various languages basically have their own cache modules and Methods. php has a pear cache module, and Java has more ,. net is not very familiar with, I believe there must be.
5. Images
Images are often used by large websites to improve performance and data security. The image technology can solve the differences in user access speed caused by different network access providers and regions, for example, the difference between Chinanet and EduNet has prompted many websites to set up image sites in CERNET to regularly update or update data in real time. In terms of image details, I will not elaborate too deeply here. There are many professional off-the-shelf solutions and product options. There are also low-cost software implementation ideas, such as rsync on Linux and other tools.
6. Server Load balancer
Server Load balancer is the ultimate solution for large websites to solve high-load access and a large number of concurrent requests.
Server Load balancer has been developing for many years. There are many professional service providers and products to choose from. I personally have some solutions, including two architectures for your reference.
Hardware layer-4 Switching
The layer-4 Exchange uses the header information of the layer-3 and layer-4 information packets to identify business flows based on the Application interval and distribute the business flows of the entire interval segment to appropriate application servers for processing. The layer-4 switching function is like a virtual IP address pointing to a physical server. Its transmission services are subject to a variety of protocols, including HTTP, FTP, NFS, telnet, or other protocols. These services require complex load balancing based on physical servers Algorithm . In the IP address world, the service type is determined by the TCP or UDP port address of the terminal. The application interval in the layer-4 switch is jointly determined by the source and terminal IP addresses, TCP and UDP ports.
In the field of hardware layer-4 switching products, there are some well-known products to choose from, such as Alteon and F5. These products are expensive, but value for money, it provides excellent performance and flexible management capabilities. Yahoo China used three or four Alteon servers on nearly 2000 servers.
Layer-4 software exchange
After learning about the principle of the hardware layer-4 switch, the four-layer switch based on the OSI model came into being. Such a solution achieves the same principle, but has a poor performance. However, it is easy to meet a certain amount of pressure. Some people say that the software implementation method is actually more flexible, and the processing capability depends entirely on the familiarity of your configuration.
We can use LVS, which is commonly used in Linux for software layer-4 Switching. LVS is a Linux virtual server. It provides a real-time disaster response solution based on heartbeat to improve system robustness, at the same time, it provides flexible virtual VIP configuration and management functions to meet a variety of application needs at the same time, which is essential for distributed systems.
A typical load balancing strategy is to build a squid Cluster Based on layer-4 software or hardware exchanges. This idea is adopted on many large websites, including search engines, this architecture is low-cost, high-performance, and highly scalable. It is easy to increase or decrease nodes in the architecture at any time. I have prepared a special detail for this architecture and will discuss it with you.
For large websites, each method mentioned above may be used at the same time. I will introduce it more easily here. You need to be familiar with and understand many details during the implementation process, sometimes a very small squid parameter or Apache parameter setting will have a great impact on the system performance. I hope you can discuss it together to make it easier.