Brief analysis of Fastdfs and the problems encountered in the collection

I. Introduction of FASTDFS

Fastdfs (Latest version v3.04) is an open-source, lightweight, distributed file system that is implemented in pure C to support UNIX systems such as Linux, FreeBSD, and Aix. It can only access the files via proprietary APIs, does not support POSIX interface, cannot mount, is not a system-level Distributed file system, but is an application-level Distributed file storage service.

Second, Fastdfs use

1) Fastdfs mainly solves the problem of large-capacity file storage and high concurrent access, and it realizes load balance when file is accessed.

2) Fastdfs realizes the software RAID, can use the inexpensive IDE hard disk to carry on the storage, supports the storage server online expansion.

3) Fastdfs is especially suitable for large and medium-sized websites to store resource files (e.g. pictures, documents, audio, video, etc.).

Three, Fastdfs principle

Storage nodes are grouped (group) in a way. The storage system consists of one or more group, the files between group and group are independent, and the file capacity of all group is the file capacity of the whole storage system. A group can consist of one or more storage servers, and the files in a storage server under a group are the same, and multiple storage servers in group play a role of redundant backup and load balancing (one group has storage capacity of the smallest storage server in the group. Different groups of storage servers do not communicate with each other, and storage servers within the same group are connected to each other for file synchronization.

When adding servers to the group, synchronizing existing files is done automatically by the system, and after synchronization is complete, the system automatically switches the new server to the online service.

You can add group dynamically when there is not enough storage space or is about to run out. Only one or more servers need to be added and configured as a new group, which increases the capacity of the storage system.

Fastdfs has only two roles: Tracker Server and Storage server. Tracker server is a central node, and its main role is load balancing and scheduling. Tracker server records information such as group and storage server status in memory, does not log file index information, and consumes little memory. Additionally, when the client (app) and storage server access Tracker server, Tracker server scans the in-memory grouping and storage server information, and then gives the answer. From this we can see that tracker server is very lightweight and does not become a system bottleneck.

Storage server in Fastdfs is often referred to as trunk server or data server in other file systems. Storage server uses the OS file system to store files directly. Fastdfs does not block the file, the client uploads the file and file one by one on storage server (the file ID in Fastdfs is divided into two parts: the group name and the file name, both of which are indispensable).

Four, Nginx environment call FASTDFS requirements

1) Compile and install Nginx with Fastdfs-nginx-module module –ADD-MODULE=/ROOT/FASTDFS-NGINX-MODULE/SRC

2) Modify Nginx configuration file to increase ngx_fastdfs_module;

3) Compiling fastdfs_client.so PHP Extension Module

4) Modify php.ini let PHP-FPM support fastdfs_client expansion module extension = fastdfs_client.so

Note: If you want to enable FASTDFS to support web and power-on automatic scripting (which is not supported by default), you need to modify the make.sh file

#WITH_HTTPD =1

#WITH_LINUX_SERVICE =1

(Because the Fastdfs module is added to Nginx, there is no need for Fastdfs to support HTTP so there is no need to remove the comment #WITH_HTTPD = 1).

v. Problems encountered

1. Fastdfs applicable scenarios and scenarios that are not applicable?

Fastdfs is a set of distributed file storage systems tailored for Internet applications, ideal for storing user images, videos, documents, and more. For Internet applications, the advantages are obvious compared to other distributed file systems. Fastdfs does not block storage of files, so it is not suitable for distributed computing scenarios.

2. What is the compilation and operating environment required by the FASTDFS?

Fastdfs server only supports UNIX systems, passed in Linux and FreeBSD tests. There are some problems with network communication under the Solaris system. Other library files required for compilation are Pthread and Libevent. Pthread using the system comes with. The version requirement for libevent is 1.4.x, and it is recommended to use the latest stable version, such as 1.4.14b. Note that you should never use Libevent 2.0 stable previous version.

3. When you start storage server, you are always in a zombie state.

Starting storage Server,storage will connect to tracker server and will retry if not connected. Until the connection is successful, the startup is truly complete.

If this occurs, check why the connection is not on tracker server.

Friendly tip: From V2.03 onwards, multi-tracker server will do time detection at startup, to determine whether it is necessary to synchronize 4 system files from another tracker server.

The trigger time is the first Storage server connection on tracker server, and a join request is initiated.

If there are 2 tracker servers in the cluster, and one of the tracker does not start, it may cause storage server to remain in a zombie state.

4. When performing fdfs_test or fdfs_test1 uploading a file, the server returns an error number 2

Error number 2 indicates that storage server does not have an active state. You can perform fdfs_monitor to view server status.

5. How do I delete an invalid storage server?

You can use Fdfs_monitor to delete. The command line is as follows:

/usr/local/bin/fdfs_monitor Delete

For example:

/usr/local/bin/fdfs_monitor/etc/fdfs/client.conf Delete group1 192.168.0.100

Note: If the state of the storage server that is being deleted is active, that is, the storage server is also in the service online, it cannot be deleted.

6. The nginx and Apache extension modules correspond to the FASTDFS server version

Extension Module 1.05: For Fastdfs server v2.x, requires server version greater than or equal to v2.09

Extension modules 1.07 and later: for Fastdfs server v3.x

7. Upload file failed, return error code 28, what's going on?

The error code 28 is returned, indicating that there is insufficient disk space. Note that there is a concept of reserved space in the Fastdfs, set in tracker.conf, the configuration entry is: Reserved_storage_space, the default value is 4GB, which is reserved 4GB space. Please set the Reserved_storage_space parameter as appropriate, for example, can be set to about 20% of the total disk space.

8. nginx Extension module, does not display the picture problem correctly

In the configuration file/etc/fdfs/mod_fastdfs.conf, the default setting is this: Http.need_find_content_type=false This parameter needs to be set to true in Nginx, should be set to False in Apache

Brief analysis of Fastdfs and the problems encountered in the collection