Technical preparations for millions of website visits (in progress)

Document directory

• VII. Database
• 8. cache.
• 9. queue.
• 10. File storage.

VII. Database

Almost all operations will end up on the database, and it is the most difficult to expand (storage is also difficult ). For MySQL, determine which tables use MyISAM and InnoDB before development. You must also confirm the replication policy and sharding policy. For Table engines, MyISAM can be used for tables with few updates and no transactions. InnoDB is used for tables with row locking and transaction support. The MyISAM lock table is not necessarily the root cause of low performance, and InnoDB is not always a row lock. For details, see the relevant documents and get familiar with the engine features for better use. Modern web applications are becoming more and more complex. We often design a lot of redundancy when designing table structures. Although it does not conform to the traditional paradigm, it is worthwhile to consider the speed. In the case of high requirements, we even need to eliminate joint queries. Pay more attention to data consistency during programming.

In terms of replication policies, it is best to design the multi-master and multi-slave structure from the very beginning. The code is written directly according to the multi-master and multi-slave structure, and some tips are used to avoid replication latency issues, in addition, it also solves the problem of consistency of multi-database data. You can write or find a ready-made O & M tool by yourself.

Partition policy. There will always be a few tables with a large amount of data, and sharding is required. There are many sharding policies, from simple partitioning to automatic adjustment based on heat, select a suitable one based on the specific business. Avoid auto-increment ID as the primary key, which is not conducive to sharding.

It is difficult to use stored procedures for expansion. This often happens in traditional C/S, especially for developers who have switched from the OA system. A low-cost website does not use one or two minicomputers to run a database to process all services. The convenience of horizontal scaling is more important than the pre-analysis time and network transmission traffic.

Nosql. This is just a concept. In practical applications, websites are increasingly concentrated on writing operations, reading hundreds of millions of simple relational data, and hot standby. This is not what traditional relational databases are good, as a result, many non-relational databases, such as redis/TC & TT/MongoDB/memcachedb, were generated. During the test, these write operations almost reached at least 10 thousand times per second, more than 50 thousand of memory type. For example, in MongoDB, you can create a replication + automatic sharding + failover environment with a few configurations. The document-based storage also simplifies the development mode of the traditional design library structure. Many businesses can use such databases to replace MySQL.

8. cache.

The database is very fragile and must be slowed down. In fact, our optimization speed is almost the optimization of the cache. Where the cache can be used, we should not go to the backend database. The cache includes persistent cache and memory cache, and generating static pages is the most understandable persistent cache. There are also many examples such as varnish block cache and memcachedb mentioned earlier. The memory cache and memcached bear the brunt. Cache updates can be passively updated and actively updated. The advantage of passive update is that it is easy to design. When the cache is empty, it automatically retrieves data from the database and fills in the cache. However, it is easy to cause an avalanche effect. Once a large area of cache fails, the database pressure is likely to rise sharply. Active cache can avoid this but may cause the program to fail to obtain data. How the two work together requires more brains in programming.

9. queue.

A user operation may trigger a series of resource and function movements. If these actions occur at the same time, the stress cannot be controlled, and the user experience is poor, you can put these operations into the queue, it is executed asynchronously by several other modules, such as sending emails and sending SMS messages. There are many open-source queue servers with low performance requirements. You can also use a database as a queue. As long as the interface of the program's read/write queues remains unchanged, the underlying queue service can be changed at any time, similar to the zend_queue class in Zend framework, Java. util. queue interface.

10. File storage.

In addition to structured data, we often need to store other data, such as images. This type of data is massive and has a high access volume. Typically, images, from user portraits to Photos Uploaded by users, must be generated with different thumbnail sizes. The distribution of storage is almost as difficult as database expansion. When professional storage is not used, it is basically based on its own NAs. This involves the structure. Taking image storage as an example, images are very prone to hot spots. Some images are no longer viewed after they are uploaded, and some images may be scanned several hundred thousand times a day, asynchronous backup of a large number of small files also takes a lot of time.

In order to prepare images for CDN in the future, it is best to separate the image domain names from the primary domain names. Many websites set cookies. domain. ltd. If the image is under this domain name, it is likely that the cache will become invalid because of cookies and occupy excessive traffic. It may also cause slow access due to browser concurrent thread restrictions.

If you use a common file system to store images, there is a simple method. Calculate the hash value of a file, such as MD5. Take the first result as the first level directory, so that the first level has 16 directories. From 0 to F, you can use this letter as the domain name, 0.yourimg.com to f.yourimg.com (the client DNS pressure increases), and expand to a maximum of 16 NAS clusters. Second-level available year, for example, 201011, third-level use day, Level 4 optional, based on the upload volume, such as AM/PM, or even hour. The final directory structure may be E/201008/25/AM/e43ae391c839d82801920cf.jpg. During rsync backup, you can use scripts to synchronize only files of a certain date of a certain year to avoid overhead of computing a large number of files. Of course, it is best to use a dedicated distributed file system or a more professional storage solution.

Next, let's talk about the code.