Analysis of large-scale portal website architecture

Tens of thousands of people at the same time to visit the site, generally there are many databases at the same time work, the white point is the database cluster and concurrency control, such a site real-time is also relative. These sites have some common features:

Large amount of data, more people online, more concurrent requests, PageView high, fast response. Summarizes the structure of the major sites, mainly to improve the efficiency and stability of several places include:

1. Procedure
program development is on the one hand, system architecture design (Hardware + network + software) is on the other hand.
Software architecture, do the site first need a lot of Web servers to store static resources, such as pictures, video, static pages, etc., do not put static resources and application server together.
A good programmer will write a program that is very concise and performance is very good, a novice programmer may make a lot of low-level errors, which is one of the factors that affect the performance of the site.
Web site to achieve high efficiency, not only the programmer's Things, database optimization, program optimization This is necessary, in the performance optimization of the database and procedures go hand in hand. Caching is also a two-pronged approach.

First, database Caching and database optimization, this is done by the DBA (and this has great potential to dig, just because we are programmers and ignore him).

Second, the program optimization, this is very fastidious, for example, the important point is to standardize SQL statements, less in multi-use or, multi-use preparestatement stored procedures, in addition to avoid program redundancy, such as finding less data with double loops. In addition, the use of excellent open source framework to support, I personally think that the support in the background is the most important, you can choose Spring+ibatis. Because Ibatis directly operates SQL and has a caching mechanism. The benefits of spring do not need me to say more, the IOC mechanism can avoid new objects, which also saves overhead. According to my analysis, most of the overhead is generated when new and connected to the database, please try to avoid. You can also use some memory testing tools to make a demo description of Hibernate and Ibatis who is faster. Front desk you want to use what, struts,webwork all become, if feel oneself very ox x can try tapestry.
With the database may not be able to solve the huge number of access problems caused by the static file hard disk address time may not be less than the database search time, of course, the index of the data to the next turn. I think the portal is often the same day, popular data click-through rate is higher, it does not cache the most also 1~2g the amount of data, for example:

Take NetEase news to say http://news.163.com/07/0606/09/3GA0D10N00011229.html
Format, easy to understand: HTTP///year/month/news category/News id.html
Can be released on the day, the popular, large browser to make a cache, with Hashtable (key: Year-month-day-category-id,value: News object), statically put it into memory (speed absolutely faster than the hard drive addressing static page).

Typically with Oracle stored procedures + 2 WebLogic, the update mechanism is almost the same as every news issue, a static page is generated, and then to the front-end Web server, the front-end of the web is load-balanced. There are also timed programs that are automatically generated every 5-15 minutes. The data is cached while the news is being published. Of course, the cache is not getting bigger, and deleting outdated data at a specific time period (such as the wee hours). Make a big web site is not as simple as imagined, the server basically must Bashing.
This can greatly increase the processing speed of a computer, if a machine cannot handle, you can use the Httpserver cluster to solve the problem.

2. Network
China's network sub-South Telecom and North Netcom, access to the IP to distinguish between North and south into different networks.

3. Cluster
Typically using CDN with GSBL and DNS load Balancing technology, a group of front-office servers in each region, such as Sina and Sohu, and NetEase, Baidu uses DNS load balancing technology, a set of front-desk servers per channel; A search using DNS load technology, all channels share a set of foreground server clusters.
The site uses Linux cluster-based load balancing, failure recovery, including application server and database server, Linux-ha-based service status detection, and high availability.
The application server cluster can adopt APACHE+TOMCAT cluster and WebLogic cluster, etc., the Web server cluster can use the reverse proxy, can also use the way of NAT, or multi-domain name resolution can be, squid also can, method many, can choose according to the situation.

4. Database
Because it is tens of thousands of people visit the site at the same time, so there are many databases working at the same time, the white point is the database cluster and concurrency control, data distribution to a different geographical location of the data center, in order to avoid a power outage accident.

The main databases are the sun's MySQL and Oracle.
Oracle is an excellent, widely-used business database management software. There are powerful features and security that can handle relatively large amounts of data. And MySQL is a very good open-source database management software, is very suitable for multiple PC server to make up a multipoint storage node array (here I am not referring to MySQL itself provides the cluster function), each unit of data storage cost is very low. Install MySQL with multiple PC server to form a storage node array, through the replication of MySQL itself or application of its own processing, can be well guaranteed fault tolerance (allow partial node failure), to ensure the robustness and reliability of the application. It can be said that, in the choice of relational database management system, the application itself can be considered to determine the situation.

MySQL database server master-slave mode, the use of database server synchronization between the master and slave servers, the application only writes data to the master server, while reading the data according to the load select a server or the primary server to read, the data by different policies divided into different servers (groups), Decentralized database pressure.

Another thing is that the static web pages of those sites are not real, but through dynamic Web pages and static Web site exchange of the appearance of the illusion, which can be used urlrewrite such an open source URL mapper implementation. The real-time nature of such a site is also relative, because there is a process of copying data in the database, generally technically can use Hibernate and Ecache, but if you want to make the site work better, you can use EJB and WebSphere, WebLogic such a large server to support, and to use a large database such as Oracle.
The MySQL database is not recommended for large portal sites unless you are familiar with the optimization of MySQL data. MySQL database server master-slave mode, the use of database server synchronization between the master and slave servers, the application only writes data to the master server, while reading the data according to the load select a server or the primary server to read, the data by different policies divided into different servers (groups), Decentralized database pressure.
Large sites to use Oracle, data-related operations as much as possible with stored procedures, absolutely improve performance, but also to enable DBAs to optimize the database, the optimized database and not optimized, there is a great difference, but also can expand the distributed database, the future of this research will be more and more;

5. Page
Consider using the virtual storage/cluster file system from the start. It allows you to access large amounts of parallel Io, and you can increase the required disk without any reorganization.
Page data call more careful design, some data query can not through the database, the real-time requirements can be used in Lucene, even if there is real-time requirements can be used Lucene (full-text indexing/retrieval engine based on Java), lucene+ Compass is still very good.
The News class website can use the static page to store, uses the regular updating mechanism to alleviate the server burden; the home page each widget can use the Oscache cache so that it doesn't have to pull the data every time.
The front-end of the Web accelerator based on static page caching, mainly used squid and so on. Squid caches most of the static resources (images, JS,CSS, etc.) and returns them directly to the visitor, reducing the load on the application server.
The static web page of the website is not true, but through the Dynamic Web page and static Web site Exchange to make the appearance of the illusion, which can be implemented with urlrewrite such an open source URL mapper, the suffix of the HTM or HTML does not explain that the program generated a static page, possibly through a URL rewrite to achieve, Only in the search engine to enhance their site coverage.
Static page generation server and WWW server are two sets of different servers, the page will be generated to the WWW server, part of the database is not a relational database, which is more suitable for information derivation, www, mail servers, routers, mainly with load balancing to solve the access bottleneck.
Disadvantages of static pages:
1) Increase the complexity of the program
2) not conducive to the management of information
3) speed is not the fastest
4) Hurt hard drive

6. Cache
Caching should be used from the start, and caching is a better place to store temporary data, such as a temporary file that is tracked by a particular user's session on a Web site and is no longer required to be logged into the database.
Can not be implemented with Lucene cache, distributed cache can be used memcached, if the money with 10来 machine to do the cache,> the storage of 10G is believed to be enough; If you don't have the money, you can work on the page cache and the data cache. Multi-use Oscache and Ehcache,swarmcache can also, but it is said that synchronization is not very good;
You can use Memcache (distributed cache) for caching, using large memory to cache all these unchanging data, and when modified to notify the cache expires, Memcache is a distributed cache developed by LJ, many large Web sites in the application, we can put the cache The server is installed with the app server. Because the cache server is not CPU-intensive, and with the support of the cache server, the memory requirements of the APP server are not too high, so it can coexist peacefully and make more efficient use of resources.

The strategies for single-machine memory caching, file caching, database caching, and so on, can be easily implemented, such as using Microsoft's caching application Block, but how to make multiple caches, multi-tier caches, and save synchronization in a clustered environment is a major issue. Large Web sites typically use a cache server farm and use a multilayer cache. The most commonly used in the industry are:

Squid Cache,squid server farm, as a Web server-side cache server caching related requests to improve the speed of the Web server. Squid caches most of the static resources (images, JS,CSS, etc.) and returns them directly to the visitor, reducing the load on the application server.

Memcache,memcache server farm, a distributed cache products, many large web sites in the application; It can handle any number of connections, using non-blocking network IO. Because it works by opening up a space in memory, it then builds a hashtable,memcached self-managing these hashtable. Because the most time-consuming task in a Web application is to retrieve the data in the database, and when multiple users query the same SQL, the database pressure increases, and the query cache hits through Memcache, and the data is taken directly from the memcache memory. Each cache hit replaces one round trip to the database server, with fewer requests to the database server, which indirectly improves the performance of the database server, allowing the application to run faster. It improves the response of Web site systems by reducing database queries based on memory cache objects, and one of its most appealing features is support for distributed deployments. For Memcache, the following articles can be consulted: reference 1, reference 2, reference 3 official site.

E-accelerator, more special, PHP cache and accelerator. is a free open source PHP accelerated, optimized, compiled and dynamically cached project that can improve the performance of PHP scripts by caching the compiled results of PHP code, making PHP scripting problems that are always complex and far from us completely resolved. By using Eaccelerator, you can optimize your PHP code execution speed, reduce server load, and increase your PHP application execution speed up to 10 times times.

7. Server operating system and Web server
The bottom layer is the operating system first. Good operating systems can improve performance, stability, and security, which are critical to the performance, security, and stability of large web sites. Taobao (Alibaba): Linux operating system + Web server: Apache Sina: FreeBSD + Web server: Apache YAHOO:FREEBSD + Web server: Own Google: Some Linux + Web servers: self- Baidu: Linux + Web server: Apache NetEase: Linux + Web server: Apache ebay:windows server 2003/8 (large number) + Web server: Microsoft IIS MySpace: Windows Server 2003/8 + WEB server: Microsoft IIS

This shows that open source operating system to do Web applications is the first choice is already a established fact. In the open-source operating system, Linux and FreeBSD difference is not too much, it is difficult to say which one must be better than the other, can comprehensively surpass the opponent, should be the strengths. But the technical staff who are familiar with Linux are more advantageous to system management, optimization, etc., so Linux is used more widely. Windows Server and IIS, while some sites use, but do not open source, and need to buy Microsoft's range of application products, limiting its use. In short, open source operating systems, especially Linux for Web applications are the first choice is already a given fact.
Common system architectures are: Linux + Apache + PHP + MySQL Linux + Apache + Java (WebSphere) + Oracle Windows Server 2003/2008 + IIS + c#/asp. NET + Database

Above some of the less mature ideas, can start from a certain level, gradually refine, the product performance indicators up.

The following content is reproduced: analysis of the structure of large web sites
A small website, such as personal website, can use the simplest HTML static page to achieve, with some pictures to achieve beautification effect, all the pages are stored in a directory, such a site on the system architecture, performance requirements are very simple, with the Internet business is constantly enriched, Website related technology After these years of development, has been subdivided into very fine aspects, especially for large sites, the use of technology is very wide, from hardware to software, programming languages, databases, WebServer, firewalls and other fields have a very high requirements, is not the original simple HTML static site can be compared.
Large Web sites, such as portals. In the face of a large number of user access, high concurrent requests, the basic solution is focused on a number of aspects: the use of high-performance servers, high-performance databases, high-efficiency programming language, as well as high-performance web containers. But in addition to these aspects, there is no way to solve the large-scale web site is facing high load and high concurrency problems.
Some of the solutions provided above also mean greater input, and there are bottlenecks and no extensibility in this approach, and I have some experience in terms of low cost, high performance and scalability.
1. Static HTML
In fact, we all know that the most efficient, the least expensive is the pure static HTML page, so we try to make the page on our site using static pages to achieve, the simplest method is actually the most effective method. But for a lot of content and frequently updated sites, we can not all manually to achieve, so we have a common information distribution system CMS, like we often visit the various portals of the news channel, and even their other channels, are through the information distribution system to manage and implement, Information Publishing system can achieve the simplest information input automatically generated static pages, but also with channel management, rights management, automatic capture and other functions, for a large web site, has a set of efficient, manageable CMS is essential.
In addition to the portal and the type of information publishing site, for the interactive requirements of the Community type site, as much as possible static is also to improve the performance of the necessary means, the community posts, articles in real-time static, there is a renewal of the time and re-static is a lot of use of the strategy, A hodgepodge like mop is the use of such strategies, such as the NetEase community.
At the same time, HTML static is also the use of some caching policies, for the system frequently using database queries but the content of small updates, you can consider the use of HTML static, such as forum public settings information, This information is currently the mainstream forum can be managed in the background and stored in the database, which is actually a lot of the foreground program calls, but the update frequency is very small, you can consider this part of the background update the time to static, so as to avoid a large number of database access requests.
2, Image server separation
You know, for the Web server, whether it is Apache, IIS or other containers, the picture is the most consumption of resources, so we have to separate the picture and the page, which is basically a large site will adopt the strategy, they have a separate picture server, and even many picture server. This architecture can reduce the server system pressure to provide page access requests, and can ensure that the system does not crash due to picture problems, on the application server and picture server, can be different configuration optimization, such as Apache in the configuration of contenttype can be as little as possible to support, LoadModule as little as possible to ensure higher system consumption and execution efficiency.
3. Database cluster and library table hash
Large Web sites have complex applications, which must use databases, and in the face of a large number of accesses, the bottleneck of the database can soon be revealed, when a database will soon be unable to meet the application, so we need to use the database cluster or library table hash.
In the database cluster, many databases have their own solutions, Oracle, Sybase and so on have a good solution, the common MySQL provided by the Master/slave is a similar scenario, you use what kind of db, refer to the corresponding solutions to implement.
The database cluster mentioned above is constrained by the DB type used in architecture, cost, and extensibility, so we need to consider improving the system architecture from the perspective of the application, and the library table hashing is the most common and effective solution. We install the business and application in the application or function module to separate the database, different modules corresponding to different databases or tables, and then according to a certain policy on a page or function of a smaller database hash, such as the user table, according to user ID for the table hash, This makes it possible to improve the performance of the system at a low cost and has a good scalability. Sohu Forum is the use of such a framework, the Forum users, settings, posts and other information database separation, and then to the post, the user in accordance with the plate and ID hash database and table, finally can be configured in the configuration file simple configuration will allow the system at any time to add a low-cost database to supplement the system performance.
4. Cache
The word cache has been touched by technology, and caches are used in many places. Caching in the Web site architecture and Web development is also very important. Here we first describe the two most basic caches. The advanced and distributed caches are described later.
Architecture cache, people familiar with Apache can know that Apache provides its own cache module, can also use the addition of Squid module for caching, both of which can effectively improve the access response of Apache.
Web application development cache, the memory cache provided on Linux is a common cache interface, can be used in web development, such as Java development can call MemoryCache to some data caching and communication sharing, some large communities use such a framework. In addition, in the use of web language development, all kinds of languages have their own cache modules and methods, PHP has pear cache module, Java more,. NET is not very familiar with, I believe there is certainly.
5. Mirror
Mirroring is often used by large web sites to improve performance and data security, the mirror technology can solve the different network access providers and geographical user access speed differences, such as the difference between chinanet and edunet prompted a lot of websites in the education network to build mirror site, Data is scheduled to be updated or updated in real time. In terms of mirror detail technology, this is not too deep, there are many professional ready-made solution architectures and products to choose from. There are also inexpensive ways to implement software, such as the tools of Rsync on Linux.
6. Load Balancing
Load balancing will be the ultimate solution for large web sites to address high-load access and a large number of concurrent requests.
Load balancing technology has developed for many years, there are many professional service providers and products can be selected, I personally contacted a number of solutions, including two architecture can give you a reference.
Hardware four-layer switching
The fourth layer Exchange uses the header information of the third layer and fourth layer packets, according to the application interval to identify the business flow, the entire interval segment of the business flow distribution to the appropriate application server for processing. The fourth layer switch function is like a virtual IP, pointing to the physical server. It transmits services that comply with a variety of protocols, such as HTTP, FTP, NFS, Telnet, or other protocols. These operations are based on physical servers and require complex load balancing algorithms. In the IP world, the business type is determined by the terminal TCP or UDP port address, and the application interval in layer fourth switching is determined by the source and endpoint IP addresses, TCP, and UDP ports.
In the hardware four-layer switching product area, there are some well-known products to choose from, such as Alteon, F5, etc., these products are expensive, but value for money, can provide very good performance and very flexible management capabilities. Yahoo China at the beginning of nearly 2000 servers using three or four alteon to be done.
Software four-layer switching
When you know the principle of hardware layer four switch, the software four layer exchange based on the OSI model comes into being, so the solution achieves the same principle, but the performance is slightly worse. But to meet a certain amount of pressure or comfortable, some people say that the software implementation is actually more flexible, the ability to handle the full look at your configuration of the familiar ability.

Software four-layer switching we can use the common LVS on Linux to solve, LVs is Linux Virtual Server, he provides a real-time disaster response based on the Heart Line heartbeat solution, improve the system robustness, It also provides flexible virtual VIP configuration and management functions, which can meet the needs of multiple applications simultaneously, which is essential for distributed systems.
A typical use of load balancing strategy is to build a squid cluster on the basis of software or hardware four-layer switching, which is adopted on many large Web sites including search engines, which have low cost, high performance and strong extensibility, and it is easy to add or subtract nodes to the architecture at any time. Such a structure I am ready to empty a special detail and discuss with you.
For large web sites, each of the previously mentioned methods may be used at the same time, I introduced here is relatively simple, the implementation of a lot of details of the process needs to be familiar with and experience, sometimes a very small squid parameter or Apache parameter settings, the impact on the system performance will be very large, I hope that we will discuss together to achieve the effect.

=================================== Why do the mirror server. ==================================

The primary purpose of the mirror server is to load balance servers.
In our country, the mirror server is more used to solve the problem of the North-south Line (Netcom Telecom interconnection slow).

How to do the mirror server.

1. Load balancing based on specific server software
This technique uses the redirection function of the network protocol to achieve load balancing, for example, to support the location directive in the HTTP protocol, and the browser that receives the instruction is automatically redirected to the other URL indicated by the directive. Because the Send location directive has a much smaller load on the Web server than executing service requests, you can design a load-balanced server based on this feature. Once the Web server thinks its load is large, it is no longer sent directly back to the Web page requested by the browser, but instead sends back a location directive that allows the browser to go to the other servers in the server cluster to obtain the required pages. In this way, the server itself must support this functionality, but there are many difficulties in implementing it, such as how a server can guarantee that the server it redirects is relatively idle and does not send the location directive again. Neither the location directive nor the browser has this support capability, which makes it easy to create a dead loop on the browser. Therefore, this method is seldom used in practice, and the server cluster software implemented in this way is less.

2. DNS-Based load balancing
DNS Load Balancing technology is the first load-balancing solution, which is implemented by random name resolution in the DNS service, where the same name can be configured for multiple different addresses in a DNS server, and the client that eventually queries the name will get one of the addresses when parsing the name. Therefore, for the same name, different customer opportunities to get different addresses, they also access different addresses on the Web server, so as to achieve load balancing purposes.

The advantage of this technology is that the implementation is simple, easy to implement, low cost, suitable for most TCP/IP applications, but its drawbacks are obvious, first of all, this scenario is not a real load balancer, the DNS server distributes HTTP requests evenly to the backend Web servers. Regardless of the current load situation of each Web server, if the background of the Web server configuration and processing power, the slowest Web server will become the bottleneck of the system, the processing power of the server is not fully functioning, second, not consider fault tolerance, if a Web server in the background fails, The DNS server will still assign DNS requests to this failed server, causing the client to fail to respond. The last point is fatal, potentially causing a significant portion of customers to be unable to enjoy Web services, and the consequences of DNS caching for a considerable period of time (general DNS refresh period of approximately 24 hours). So in the latest foreign Construction Center Web site scenario, it has rarely been used.

3. Load balancing based on four-layer switching technology
This technique is to set the virtual IP address of the Web service on a layer fourth switch, which is the IP address of the Web server resolved to the DNS server and is visible to the client. When a customer accesses this web app, the client's HTTP request is first received by the fourth switch, which detects the load of the background Web server in real-time based on the fourth layer switching technology and quickly exchanges it based on the algorithm set. Common algorithms are polling, weighting, least connection, random and response time, and so on.

4. Load balancing based on seven-layer switching technology
Load balancing technology based on seventh exchange is mainly used to realize load balance and service quality assurance of Web application. It has many advantages over layer fourth switches: The seventh switch can not only check TCP and UDP port numbers of TCP/IP packets, but also forward to a server in the background, and can parse the HTTP request URL from above the session layer. Depending on the URL, different HTTP requests are given to different servers for processing (specific to a certain type of file, up to a file), and even the same URL request can have multiple servers respond to load sharing (when a customer accesses a URL and initiates an HTTP request, It actually establishes multiple session connections with the server, resulting in multiple objects, such as. txt/.gif/.jpg documents, when these objects are downloaded locally to form a full page.

5. Site Mirroring Technology
The above load balancing technology is mainly used in a site of the server farm, but because a site access to the Internet bandwidth is limited, so you can load balancing technology to start to apply to different network sites, this is the site mirroring technology, site mirroring technology actually leverages the DNS load balancing technology.

=============================== How to Do mirror server--related issues ==================================
Question 1: Because of the current telecommunications and Netcom's reasons, a lot of Netcom's friends are always slow, depressed ah, I see some stations can choose is Netcom or telecom, do not know, this function is how to achieve.
Answer:
[HTML]
[/html]

Then put 1 larger picture 1.gif in the root directory of each space (preferably about three KB)

Question 2: There are netcom, telecom server, want to put two servers inside the site of the whole station program, I add the file is in the telecom website, want to let Netcom also automatic synchronization and telecommunications, I how to do AH

Answer:

1. Introduced

Now the site with the increase in traffic, a single server can not afford a huge amount of traffic, there is no convenient and quick way to solve the problem, the answer is "yes"!

For example, set up a server farm for a balanced load. But if you want to solve the problem such as telecom Netcom (Chinese netizens sorrow), this solution will be powerless!

The most convenient way to solve this problem is to set up a mirror site! The fastest Web site for your visitors to choose! This solves the problem of the line, but also can solve the traffic problem!

2. Data classification for Web site synchronization

Web site data is basically divided into two categories:
A type of file, such as html,asp,php, such as Web files, or Rar,zip,rm,avi and other downloadable files!
To achieve their synchronization is very simple, with FTP synchronization software on it! As for which I will be in detail in the back.

One type is database data files, such as Mysql,sql server and so on!
Database synchronization method is also many, the simplest way is simply to synchronize the database directory is OK!
In the back I will also do a detailed explanation!

3. Synchronization of Web site files

The main tool used here is the FTP, Web site file synchronization in two cases, one is local to remote, one is remote to remote (FXP)! The first is to say, the second remote to remote that is FXP, support it's many software, but really suitable for multi-site synchronous mirror is not much!

Below I introduce a few I think good software!

Below I introduce a few I think good software!

1. First of all, I would like to recommend the domestic FTP software "network of Vivid", the function is very powerful, especially in the site's synchronous mirroring aspect, unfortunately, this very classic software is no longer updated, the last update time is March 2003, the last version is 3. 12! Although this is still very useful! Here is an official introduction:

The network vividly absorbed all the functions of cuteftp and Updatanow, and added many functions that the other software did not: support the Web site synchronization (updata now), support the Post platform (multi-threaded upload multiple files), can open multiple sites at the same time , multi-site plan upload function, support the mirror site, support the macro operation to support scheduled operations, support file advanced comparison upload, support directory-hidden filtering (for friends who use Forntpage as a homepage); Server auto-recognition function; Resource Manager browsing mode; can customize commands Support the RFC959 standard with better stability, complete information return mechanism and error monitoring mechanism complete Chinese help.

2. The second paragraph is made by reget the same development company dedicated to the website synchronization software "Websynchronizer", with this software, you will experience the site synchronization is convenient, easy and easy. The latest version is 1. 3. 62, the net can find the last version of XX is 1. Version 1! Here is a brief introduction:

File Synchronization Tool-Websynchronizer, by the well-known continuing software ReGet Software Publishing company launched, is the site synchronization, file mirroring, file backup of the excellent tool, you can perform the following major tasks: 1) Synchronization of the local folder and remote folder; 2) Synchronization of data folders in two remote computers, 3) synchronization of two native data.

3. Other software, such as the synchronous Shuttle (AUTOSYNCFTP), also enables simple Web site synchronization, but the software is very unstable and has ceased to develop in 2001 years. So, do not think about it! There is a friend mentioned last time Sitemirro, because the Internet can not find a version to use, so there is no way to test!