Key points for improving efficiency of massive small file storage

In WEB development, you may often encounter file writing. The most common case is to save image files. If the number of files is small, there is no need to worry about its efficiency. However, when you have a large number of files, you will often encounter file writing in WEB development. The most common is to save image files. If the number of files is small, there is no need to worry about its efficiency. However, when you have a large number of users and images, how to store image files directly affects the efficiency of the entire image storage system.

Generally, there is a saying that, for example, if a directory contains 10000 sub-files, the speed of reading a file will be significantly reduced. Which of the following statements is true? Let's take a look:

Q: Why does too many sub-files in a single directory affect performance? For example, if a directory contains 10000 sub-files, the reading speed of a file will be obviously slow? Is this related to the file index? How to organize these nodes in the index?

A: Yes, it depends on the index. Not too many. We can see from the millions. However, we recommend that you set no more than 10000 instances.

Q: How is the relationship between the file system and the current directory? A file system that supports millions of files is nothing

A: I mean that a directory contains hundreds of thousands or millions of files without any sub-directories. in this case, it is very resource-intensive to search for this directory index.

The supported quantity is limited because the size of the directory object is limited. The directory is a container that contains the inode number corresponding to the file name and file, then, the contained entries are limited.

The speed of reading a file is not affected. However, searching is hard. The indexing mechanism of some file systems is not complete, and there is no optimization algorithm, resulting in more time for each search.

As we can see from the above Q & A, "If a directory contains 10000 sub-files, the speed of reading a file will be significantly reduced ." Is true. How can we split directories?

In fact, it is relatively simple. you can divide by month, hash, and time plus hash. as to which method to use, it depends on your project requirements...