Learning the technical architecture development history of Douban.com

At this qcon conference, the topic "the development course of Douban.com technical architecture" is almost the most important. After listening to the speech, we will find that Douban is also a bit of a detour in the development process. These are valuable treasures in the development process of the website. They can take out some of their best practices, is a valuable thing. Although Douban has revealed a lot of architectural details, it will not (or impossible) Any company can make a Douban and surpass Douban as soon as they get these things. To some extent, this reflects the popularity of Douban students, which makes most companies in China feel ashamed. Many companies only want to ask for it, but do not want to give it to them even a little bit. Using this closed mindset to treat technology is actually a little bit of childishness, and it is a kind of miser thinking. Technology can only be used by more people.

After the discussion, let's talk about it again. Experience on Douban architecture. Manipulation changes. Some items are used by everyone (nginx), but some are used well, and some are used worse than others. Therefore, we need to gradually summarize and improve it. Learning others' architectural designs is not just about copying them, but learning from their ideas.

Technology Selection

For a long time, Douban is technically cutting-edge. It seems like something new to use. Actually, it is not. They have always been "solving problems with existing technologies ", if the existing things are sufficient, there is no need to migrate them to the new one, and the conversion is often to solve the current problem. In addition, to use new things, you must have sufficient control capabilities. I learned from the speech that, douban has found bugs in the basic library several times before it went online (for example, libmemcached's consistency hash-related bug). The technical team can fix the bug immediately and submit it to the open-source community. Otherwise, it becomes a wrong decision.

Disk speed

Small topic. If possible, you can buy a 15000-to-disk. A 10000-to-disk may save money, but it is almost impossible to upgrade it after deployment. Therefore, if you are a start-up company, I suggest you buy high-speed disks. If your business is growing fast, your previous machine positioning may also change.

Eliminate remote I/O

Do not write data remotely in a common TCP/IP network. The latency of cross-network operations does not seem to be a big deal, but once it reaches the critical point, it will not work. This is basically because you don't hit the south wall and don't look back. Some technicians have to experience it before they can stop.

Maintain a URL-friendly style

Consistent URLs have been mentioned many times in the speech. In fact, when Douban attaches importance to URL rewrite, structure adjustment, or application changes, it is best to make URLs "user friendly. This is a "soft technology", but we should pay the greatest attention to it.

Database Replication latency

In the MySQL replication environment, if there is a read operation on the slave, the user may be confused in some cases because the data on the master and slave nodes is inconsistent. From the perspective of consistency, it is not complicated :, you only need to provide consistency for "users who know data has changed" (basically users who initiate changes ), users who do not know the data changes have a certain degree of "tolerance" for data inconsistency. Of course, it is simple to say, and it still requires skill and skill to implement it.

Synchronization of a large number of small files: Merkle tree

Many websites with large numbers of small files will encounter synchronization issues. If the design is poor or relatively lazy, use traditional methods (such as the old mode of rsync) it is easy to trigger problems and waste resources. Doubanfs uses Merkle tree (hash tree) for data synchronization. For a detailed description of this problem, see real-time synchronization solution for a large number of small files. Merkle tree is an exquisite idea. ZFS is in use (refer), and Amazon dynamo systems are also in use.