Questions about the initial Website planning in 2.0 [reprinted]

I have worked as a technical consultant for over 2.0 companies and briefly talked about the problems encountered by 2.0 companies (I used a B c d instead of privacy ), here, we will not go into details about the well-known issues of page static, caching, and code security. CTO of some technical 2.0 companies know these things. Let's talk about things that have evolved.

Company

Company A is working on an SNS website. The program is made by two young men with a goal of 51. The program development is smooth sailing and has more functions than 51, promotion is also smooth sailing (Company A has its own unique promotion method. But when Alexa reached, the problem came out. Every day around four o'clock P.M., the website speed was so slow that it could not be opened. The company's three servers cpu100 %, the depressing thing is that the company's network configuration method is actually a dual-web cluster, and a separate DB database. The whole bottleneck lies in the database, so I suggest you create a database cluster and analyze the data structure. md, a typical web programmer's work, does not have any database design specifications, so function implementation is acceptable, if expansion is required, the cluster is basically impossible. What should I do? You can't do it. So, after a month of modifying the program, the data structure basically changed to the hundreds of thousands of times that had been dumped in the early stage, and the user went away.

Conclusion: During the preliminary design of Web2.0, we should not only consider functions, but also carefully consider the underlying and data structures.

Company B

Company B is also a SNS website. The program is developed by three people. The CEO is a master of economics from a famous university, and has some characteristics. To be honest, the company has good potential. The CEO has a strong operational ability and feels promising. The system architecture is okay, but --- but the system crashes, why? The system does not take into account the massive number of users, but also the massive number of files. users' photo albums and images are all stored in one partition of the web server, with each user having a directory, however, when the performance monitor is turned on, the disk's Io is surprisingly high and basically has no response time. As we all know, the file system is also a database, and it doesn't matter if you have a single large file. The key is that the entire file is more than 300 GB of fragmented files, massive read/write operations, system crashes, and data loss, A chain in the file system is broken, and all user data is lost !!! This is a very heavy problem. The system has stopped for a whole month to recover data (it is easy to separate files, but there is currently no software to organize the software architecture for massive files ). Solution: modify the program architecture and perform distributed file storage (it took 8 days for the program to be modified, but the file transfer took nearly a month), and 0.2 million users were lost.

Conclusion: In the early stage of Web2.0, there should be considerations for dealing with massive storage. The whole process involves the modification of the program architecture. If the preliminary planning is poor, there is basically a line of thought.

Company C

Company C is a respectable company with a CEO's technical background. like Bill Gates, the University has not graduated from the university and has made a fortune in text messages from 01 to 03 years, I was very impressed with the small projects I made later. The company focuses on alumni, but focuses more on the MySpace style, personal homepage, and promotion. The cause of the system crash is actually very simple. Because Microsoft's sqlserver is used, Microsoft directly told us that sqlserver does not support clusters, and their databases are overloaded, so 100% will not survive, configuration can only be added horizontally. A 4-Core 4-core CPU system is used, but the system crashes... high interaction is destined for high load. Solution: we will start from the basics to solve several large program energy-consuming users, adopt horizontal cutting for the database, group every 0.1 million users, and hash the database system, split multiple tables vertically and group files at the same time to solve the problem. because the data structure has been modified, the program has basically moved a bit. Fortunately, the system did not make a big mistake, and the loss was not very large, but it had a bad impact on the user experience.

Conclusion: the preliminary design of Web2.0 should be well hashed, and the program should be expandable in line with the database expansion.

Company D

Company D is a good company in all aspects, with CDN acceleration, and N servers and a good database (CTO is a database expert ), the reason for the system crash is that the Web can easily be used as a cluster. However, when a cluster is discovered and cannot be resolved, only four web clusters can be used as the cluster, however, all four of them are replaced. After careful analysis, find out the cause. I guess it is also the most common mistake that most ctos make, or the problem they don't think of at all, is the problem of Web upload, during the upload process, the thread maintains the link due to the time, and the 300 thread can delete a web server. Solution: this is the easiest way to separate uploading from other energy-consuming users. Program changes are not very large, but the loss of user experience cannot be underestimated if the speed of the previous half month is full.

Conclusion: There is no conclusion. After all, there are not many CTO engineers with massive access experience, that is, those websites.

Conclusion: It is easy to imitate because it is not just a cold water. It is easy to find a few web programmers, and the speed may be very efficient, because Web2.0 is nothing more than dealing with databases, will operate the database. However, it is not easy to really expand, because it is not easy to cope with the massive access of the program. The current programmers are too pretentious. In fact, there are not many real experiences, do not believe that a programmer with a monthly salary of 5 k-10 K can surprise you. It is not the price that programmers can cope with massive access. If you want to increase the performance by 2.0, there are several suggestions:

1. Ask DBMS experts to design databases. Most programmers do not know partition views, data hashes, and the concept of data groups.

II. well-designed program architecture (this is not difficult, but it will be fine if you have an expert guide) to maintain good scalability. For cost consideration, you can find a part-time system architecture designer to do a good job in system architecture, determine future development bottlenecks.

3. Consider file storage issues. The technical content of file storage seems very low, but it is actually very high. You can consider the reverse proxy solution. When there is a problem with file storage, the site is basically finished, not only the raid problem and the storage server problem, but the truth is broken.

4. Considering China's national conditions, this is the most critical issue. We need to consider the issue of telecommunications and China Netcom. CDN cannot solve all problems. CDN is not very effective for interactive things. The most important thing is that the existing dual-Data Center suffers from DDoS attacks. The reason is very simple. The dual-data center is a private data center, so there will be no high bandwidth, if you attack it at will, you can just drop it (along with a joke, I know that the boss of a dual-line data center has bought a 4G Alibaba Cloud security wall with a total bandwidth of 1 GB, it's easy to handle MB of attacks ).

5. network latency. This is an issue that must be taken into account in Distributed Systems. Programs must tolerate data latency between 0 and 100 seconds, that is, synchronization. Don't underestimate the dozens of seconds. The problem is big. If your site has interactive functions, such as instant chat, you can imagine what the result is. For instant chat, you can use reverse proxy (high cost ). But it has little impact on messages and comments, but if the system caches and static content for robustness, this may be disastrous.

6. Spread your programs. If you don't have much money to build millions of servers, we recommend that you spread the functions, such as one server for photo albums and one server for messages.

7. optimistic about your programmers. Without good incentives, your programmers can easily write perfunctory code, which may be a major problem in the future, it may take a lot of effort to modify the program architecture after it is finalized. It is best for your CTO to be 100% responsible and 100% responsible for you.

8. the file synchronization problem may be unnecessary. If you look at the TTL of China Netcom and China Telecom, you will understand that synchronization must support continuous transmission and cannot be sustained, otherwise, your cost will be n times higher. Don't expect it to be implemented through your software. Hand it over to your programmer. Tell him what he said above and he will know how to do it.

9. one of the hardest problems is the biggest loss. No matter how good your relationship with the cyber police is, optimistic about your users and review your stuff, it may be fatal if your services are shut down, I have suffered N losses.

10. Finally, I wish you all the masters a great success.

# Software Engineering