Why we want to use Distributed File system (experience after feeling)

1 , why do Distributed file systems use a specific organizational structure to store files?

Store and copy directly in the original path of the file, so you can perform static access directly through the app service to dramatically improve performance. How's that for a good idea?

Wait, we seem to be winding back again ...

Such a system, presumably a shared file system ? or a file distribution system.

What if I just share the file system and there are too many files? File access pressure is too big what to do? What if the file is missing? What if the file is wrong? What if the file server is hung?

How to do, how to do?

Not so much to do, so we have experienced these practices and use, the conclusion is that with the Distributed file system, this is done.

2 what the Distributed file system can do for us (pros)

It is possible to build a mass storage system with a large number of inexpensive servers, which is not easy for file distribution and synchronization;

It is very important to ensure the availability of files through internal redundancy, and in the mass storage system, the fault-tolerant ability is very significant.

Scalability is very strong, it is easier to add storage nodes and trackers;

Load balancing across multiple copies of files can be scaled out to ensure a performance boost.

Perform the calculation of specific index files, etc.;

...

3 , a lack of distributed file systems, or shortcomings .

Low latency Access

Distributed file systems are less suitable for applications that require low latency (dozens of milliseconds) to access because Distributed file systems are designed for massive data processing, at the cost of a certain delay. For low-latency access, the classic and traditional approach is the database, and our favorite ORACLE is very good at doing it.

for Example a payment system, for its core payment system, backend with P590 minicomputer +oracle, when the scale of payment is getting larger and bigger. The payment system for minicomputer +oracle can be very painful. The scale of the database, horizontal segmentation is a method, but directly to the core payment module to switch to the Distributed file system, there are challenges, did not succeed in the year. (Opinion, maybe you can handle it, for reference only)

The application of frequently modified files

currently commonly used Distributed file system, basically are "write once multiple read" mode, if it involves a large number of frequent changes in data, then this problem is relatively troublesome;

Massive Small files

The Distributed File system places the file system's metadata in memory, so the number of files the file system can hold is limited. In general, each file, folder, and Block needs to occupy about the space of about four bytes, so if you have thousands of files, With each Block, you need at least 300MB of RAM. Currently, millions of of the files are still viable, and when scaled to billions of, it is painful for the current hardware level. Therefore, due to the large amount of metadata, the distributed file system is relatively weak in treating large amounts of small files.

Other

some directly through http files that are accessed, such as scripts, CSS and so on.

To carry out complex calculations, such as the calculation to launch rockets to Mars, by the way in the spaceship to configure a table mahjong, the need to calculate how much capacity, how much of the effort point;

Why we want to use Distributed File system (experience after feeling)