Build a distributed file system using scripts

Last weekend, ADCs @ Ming Peng presided over the discussion on the open-source topic of TFS. Some external users of TFS went to the site to ask questions about the use process. It was not easy for these small companies to maintain large systems, especially when the internal implementation is unclear. Some people share their company's practices. When saving files, they store them on multiple servers, and then record the correspondence between the files and the server to the document (you can't lose them, it's easy to find them ), when an exception occurs, manually copy the file to increase the number of file copies, and then update the correspondence between the file and the server. The practice looks very earthy, but for small companies, as long as the maintainer can "Never get tired of it", this is also a good way, after all, everything is under control.

 

At present, many internet startups are faced with data storage problems, such as generating a large number of unstructured data (pictures, music, videos, documents, structured data, such as registration information, can be stored directly in MySQL). The data volume is very embarrassing. Generally, servers cannot store and expensive servers cannot afford it; as a result, everyone is eyeing open-source solutions that use cheap pcs to build distributed file systems, such as TFs and moosefs. However, using these open-source solutions is risky. It is of course a pleasure to run normally every day, if something goes wrong, you can't solve it if you don't do some research. In the initial stage of your business, you don't have the human resources to do this research, or you can't find someone who can do this.

 

So I want to consider whether a startup company can use scripts to build a storage solution when it needs to store a large number of files. The advantage of this solution is that script is used, the learning cost is low, the R & D cost is low, and self-developed, strong controllability. Compared with the preceding manual solution, scripts can be used to automate many situations and reduce the frequency of manual intervention. The specific considerations are as follows:

 

• On the web server, mount the storage directory of the storage server (for example, each disk of each server) to the local through NFS.
• When the Web server stores files, select two or more directories (not on one server) from these Mount directories to store the files on these servers, and record the ing between files and disks (MySQL or local + NFS ).
• Use scripts (Python, Perl, and so on) to monitor the disk of the server (many methods can be used freely). When the disk or machine is detected to be broken (it is usually OK when the machine is restarted ), you can get the list of files on the disk from the ing information and add a copy of these files.
• When the Web server is reading the file, it randomly selects a server that stores the target file. For file hotspots, You can manually intervene to increase the number of file copies.

 

Many of the above functions only require a small number of scripts.CodeIn the initial stage, it does not need to be improved, and the O & M process continues to improve. In the end, it is possible that a set of good solutions will be developed.