[Turn] Various large-scale web site technology architecture

Introduction in recent times, through contact with the massive data processing and search engine of many technologies, often see a lot of exquisite architecture diagram. In addition to every exclamation on the surface of each picture of the fine, more than the structure behind the hidden design ideas are amazed. Individuals these two days have been collecting the structure of the large web site design, one for feast, to understand the various types of large-scale Web site architecture and design of the wonderful, and secondly also can be used for leisure time to ponder over the experience, why not? hereby, summarize and collate such as foreign wikipedia,facebook,yahoo! , Youtube,myspace,twitter, the technical architecture of large-scale websites such as Youku (This paper focuses on the technology architecture of Youku), readers. This paper highlights the highlights of each picture and the meaning behind it, while the illustrative text of the figure is withheld. OK, enjoy the architecture feast. Of course, if you have any suggestions or questions, please do not hesitate to correct me. Thank you.

• 1. WikiPedia Technology Structure

WikiPedia Technical Architecture copy @Mark Bergsma

1. Data from Wikipedia: Spikes of 30,000 HTTP requests per second for 3Gbit traffic, nearly 375MB 350 PC servers.
2. Geodnsa:40-line patch for bind to add geographical filters-the existent views in BIND ", takes the user to the nearest server. GeoDNS's role in the Wikipedia architecture is, of course, determined by the nature of Wikipedia's content--for every country, every region.
3. Load balancer: LVS, see:

.

• 2. Facebook Architecture

the architecture of the Facebook search feature

Careful readers will be able to find that the upper-secondary architecture diagram appears in this article: stealing from a few architectural drawings of the ocean data processing experience. This article and the largest difference is that the previous article only a few, this article series will have hundreds of architectural drawings, let you enjoy.

• 3. Yahoo! Mail Architecture

Yahoo! Mail Architecture

The Yahoo! Mail Architecture deploys Oracle RAC to store Meta data related to Mail services.

• 4. Twitter technology architecture

Twitter's overall architecture design diagram

The Twitter platform is broadly comprised of twitter.com, mobile phones, and third-party applications, as shown (where traffic is primarily based on mobile phones and third-party sources):

Caching plays an important role in large Web projects, after all, the closer the data gets to the faster the CPU accesses. It's Twitter's cache architecture: about the cache system, you can also look at the following image:

• 5. Google APP Engine Technology architecture

the architecture diagram of Gae

In simple terms, the above Gae architecture is divided into three parts: Front end, datastore and service group.

1. The front end consists of 4 modules: Front end,static files,app Server,app Master.
2. Datastore is a distributed database based on BigTable technology, although it can also be understood as a service, but because it is the only place for the entire app engine to store persistent data, it is a very central module in App engine. The specifics will be discussed in the next article.
3. The entire service group includes many services for app server calls, such as memcache, graphics, users, URL crawls, and task queues.

• 6. Amazon Technology architecture

Amazon's dynamo key-value storage architecture Diagram

Some readers may not be familiar with Amazon, which is now the world's largest online retailer and the world's 2nd largest Internet company. And it was just a small online bookstore. OK, below, let's see how it's structured. Dynamo is Amazon's key-value-mode storage platform with good usability and scalability and good performance: 99.9% response times in read-write access are within 300MS. Divide the data by hashing algorithms commonly used in distributed systems and put them on different node. The read operation is also based on the hash value of key to find the corresponding node. Dynamo uses the consistent hashing algorithm, node is no longer a definite hash value, but a hash value range, key hash value falls in this range, then clockwise along the ring, encountered the first node is required. Dynamo the improvement of the consistent hashing algorithm is that it is placed on the ring as a node is a set of machines (rather than memcached a machine as node), a set of machines that ensure data consistency through a synchronization mechanism. is a distributed storage system that readers can observe:

Amazon's cloud architecture diagram looks like this:

Amazon's cloud architecture diagram

• 7, the technology structure of Youku

From the outset, Youku has built a CMS to solve the front-end page display, the separation between the modules is more appropriate, the front-end extensibility is very good, the separation of the UI, so that development and maintenance has become very simple and flexible, is the Youku front-end Module call relationship: So, according to module, Method and params to determine the relative independence of the module, appears very concise. Is Youku's front-end local architecture diagram: Youku's database architecture has gone through a lot of twists and turns, from a single MySQL server (Just Running) to simple MySQL master-slave replication, SSD optimization, vertical sub-Library, and horizontal sharding sub-Libraries.

2. simple MySQL master-slave replication. MySQL master-slave replication to solve the database read and write separation, and improve the read performance, the original figure is as follows: its master-slave replication process as shown: However, master-slave replication also brings a number of other performance bottlenecks:
   1. Write cannot be extended
   2. Write cannot be cached
   3. Replication delay
   4. Lock Example Rise
   5. Table becomes larger, cache rate drops
   The problem has to be solved, which results in the following optimization scheme.
4. MySQL Vertical Partitioning If the business is cut enough to be independent, it would be a good idea to put different business data into a different database server, and if one of the services crashes, it will not affect the normal operation of the other business, and it also plays a role of load shunt. Greatly improves the throughput capacity of the database. After the vertical partitioning of the database schema diagram is as follows: However, although the business is sufficiently independent, but some of the business between more or less always a little connection, such as the user, basically will be associated with each business, and this partitioning method does not solve the problem of single-sheet data volume explosion, So why not try the level sharding?
6. MySQL horizontal shard (sharding) This is a very good idea, the user according to a certain rule (by ID hash) group, and the group of users of the data stored in a database shard, that is, a sharding, so as the number of users increased, As long as a simple configuration of a server, the schematic is as follows: How to determine the Shard of a user, you can build a user and shard corresponding data table, each request from this table to find the user's Shard ID, and then from the corresponding shard query the relevant data, as shown: however, Youku is how to solve the cross-shard query, this is a difficult point, according to the introduction of Youku is as far as possible not cross-shard query, really not through multi-dimensional Shard index, distributed search engine, the worst is a distributed database query (this is very troublesome and consumption performance).
8. cache policy seemingly large systems have a "cache", from the HTTP cache to the memcached memory data cache, but Youku means that there is no memory cache, for the following reasons:
   1. Avoid memory duplication and avoid memory locks
   2. If the Big Brother notice to take a video off, if in the cache is more troublesome
   and squid write () user process space consumption, Lighttpd 1.5 of AIO (asynchronous I/O) read files to user memory resulting in inefficient. But why would our visit to Youku be so smooth that the video loading speed of Youku is notch above compared to potatoes? This is due to Youku established a relatively perfect content distribution network (CDN), it is distributed across the country in a variety of ways to ensure that users in the vicinity of the visit-the user click on the video request, Youku will be based on the user's location, the closest to the user, the best service status video server address to the user, So that users can get a fast video experience. This is the advantage that CDN brings, the nearest visit.

Note: 1, this segment of the technology structure of Youku is organized here:/system-analysis/20110918/264936.html; 2, recommend a very good site:http://www.dbanotes.net/). From the hundreds of architectural plans to learn a bit of large-scale website construction experience (above), finish. PostScript This article finally finished, from yesterday to organize this article motives, to this morning to find the computer online and not, and then to the moment in the Internet bar to complete this article. I really appreciate what it's like to be a tech fanatic. Large Web site architecture is a very strong real thing, and you and I may be a temporary outside to see a lively layman just now. However, it doesn't matter, small fish shrimp can still swim in the ocean, not to mention the future can also grow into big fish sharks.

[Turn] Various large-scale web site technology architecture