The storage System stuff.

Storage systems are hard to escape from the magic of complex systems from their innate mission. Whether it is from the stand-alone file system, or later, C/s or B + + structure of the database such as storage middleware rise, or today's hot cloud storage services, storage is very complex, and is increasingly complex.

Why storage is complex, what is storage to talk about. Storing this word is very trivial, storage + computing (operation) constitutes a simple computer model. In simple terms, storage is the unit responsible for maintaining the state of the computing system. From the standpoint of maintaining the state, we will have the simplest reliability requirements. For example, a stand-alone file system, machine power outages, program failures, system restart, and other general anomalies, file systems must be able to correctly respond to disk sector damage, and file systems also need to consider as far as possible to minimize the loss. For most business applications, you only need to focus on the normal branching process of your business, and for unexpected situations, you usually just throw an error and tell the user that you shouldn't be playing. But for the storage system, you need to spend most of your time dealing with all sorts of anomalies, and even you should think that these complex and diverse error-branching processes are the "normal business logic" of the storage system.

To the internet age, with C/s or b/s structure, the storage system has a new indicator: usability. In order to ensure quality of service, those invisible server programs must always stay online, preferably logically not downtime (availability 100%). How can a server program achieve high availability? The answer is storage middleware. Without storage middleware, it means that all business processes must consider persisting each step to the state, so that they can hang on to another server (or reboot later), know where the work was before, and what to do next. But the persistence of the state (that is, storage) is cumbersome, and if every business is implemented on its own, the burden is undoubtedly heavy. However, if there is a high availability of storage middleware, server-side business programs only need to operate the storage middleware to update the state, by starting multiple instances of business processes to do the standby and load balancing, it is easy to achieve business logic is not downtime.

Therefore, the emergence of storage middleware such as database is basically a historical necessity. Although the database is common, it is never the only storage middleware. For example, the rich media used in the business (images, audio and video, Office documents, etc.), we rarely go to store in the database, more times we will put them in the file system. But is the file system born in a stand-alone era really the best place to store these rich media data? No, the file system needs to change because:

Scalability. The first problem with a stand-alone file system is the limited capacity of a single computer, and what to do when the storage scale is more than one machine can manage.

Performance bottlenecks. Typically, the performance of a stand-alone file system decreases rapidly when the number of files reaches a critical point. This tipping point is fairly easy to reach today, with the growing popularity of large-capacity disks in 4TB.

Reliability requirements. Stand-alone file systems are usually only a single copy of the scenario, but today's single copy of the storage has long been unable to meet the business reliability requirements. The data needs to be redundant (a classic 3 copy), and repair the lost data as soon as the disk is damaged to avoid all replica corruption causing data loss.

Availability requirements. A stand-alone file system is usually only a single copy of the scenario, and when the machine is down, the data is unreadable and not writable.

Prior to the advent of distributed storage systems, some modified versions based on stand-alone file systems were adopted by some applications. For example, in the stand-alone file system plus RAID5 to do data redundancy, to solve the problem of the reliability of stand-alone file system. Assuming that the RAID5 data repair time is 1 days (often not, especially if the business system itself is more stressful, the disk read and write bandwidth left to the RAID fix is limited), the reliability of the stand-alone program is probably 100 years of loss of data (i.e., reliability is 2 9). Looks fair? But you have to be careful of two situations. One is the size of your cluster, you still use this earth method, for example, you now have 100 of these machines, then it will become 1 years lost data. Another situation is that if the actual data repair time is 3 days, then the reliability of the stand-alone machine will be reduced to 4 years lost data, 100 will be 15 days lost data. This figure is clearly unacceptable.

Google GFS is the first distributed storage paper that many people have read, and this paper lays out the status of 3 copies in a distributed storage system. Hadoop then referenced this paper to achieve the open source version of the Gfs--hdfs. But the HDFS of Hadoop is actually a lot of misunderstanding in the industry. The GFS design has a strong business background and is designed to be a search engine. HDFS is more suitable for log storage and log analysis (data mining) rather than storing large volumes of rich media files. Because:

The block size of the HDFS is 64M and will take 64M if the file is less than 64M. Most of the rich media files are still very small, such as the general size of the picture around 100K. One might say that I can adjust the size of the block to fit, but this is not the correct way, the HDFS architecture is designed for large files, it is impossible to simply adjust the block size to meet the needs of mass small file storage.

HDFS is a single Master structure, which determines that the number of metadata entries it can store is limited and scalability is problematic. Of course, as large file log storage, this bottleneck will be encountered very late, but if as a mass of small file storage, this bottleneck will soon encounter.