Distributed File Management System _ FastDFS cluster, _ fastdfs

Distributed File Management System _ FastDFS cluster, _ fastdfs

Brief Introduction

1. Relationship between client storage tracker

First, we use a diagram to explain how users can access a file managed by DFS.

 

Generally, one server has only one storage server, and multiple storage servers can form one group. The data of storage servers in the same group is automatically synchronized (backup and recovery ).

Data in different groups is isolated from each other. One tracker can manage multiple groups or multiple to multiple groups.

The client is used to manage tracker server and storage server.

FAST_DFS Installation

Download and decompress the DFS package. You can see the installed text file. The following installation steps are based on this.

# Step 1. download the libfastcommon package from the following link and install the https://github.com/happyfish100/libfastcommon.git#step by default 2. download the DFS package and decompress it. tar xzf FastDFS_v5.x.tar.gz # for example: tar xzf FastDFS_v5.08.tar.gz # step 3. go to the FastDFS folder cd FastDFS # step 4. run the following command :. /make. sh # step 5. run the installation command :. /make. sh installtip: If no installation path is specified by default, the script is started in the/usr/bin/directory after the installation is complete, and the related configuration files are under/etc/fdfs. # Step 6. edit/modify the config file of tracker and storage (you can skip this step to modify the configuration file) # step 7. start DFS service # first start the tracker server:/usr/bin/fdfs_trackerd/etc/fdfs/tracker. conf restart # then start the storage server:/usr/bin/fdfs_storaged/etc/fdfs/storage. conf restart # step 8. test Service Availability with a test program. # For example, upload a file:/usr/bin/fdfs_test/etc/fdfs/client. conf. sample upload/etc/fdfs/client. conf. the following information is printed in the sample: This is FastDFS client test program v5.08Copyright (C) 2008, Happy Fish/YuQingFastDFS may be copied only under the terms of the GNU GeneralPublic License V3, which may be found in the FastDFS source kit. please visit the FastDFS Home Page http://www.csource.org/for more detail. [21:03:35] DEBUG-base_path =/home/yuqing/fastdfs, connect_timeout = 30, network_timeout = 60, tracker_server_count = 1, anti_steal_token = 0, duration length = 0, use_connection_pool = 0, g_connection_pool_max_idle_time = 3600 s, use_storage_id = 0, storage server id count: 0tracker_query_storage_store_list_without_group: server 1. group_name =, ip_addr = YOUR_TRACKER_IP, port = 23000group_name = group1, ip_addr = YOUR_TRACKER_IP, port = region = group1, remote_filename = M00/00/00/latest ip address: Export timestamp =: 03: 35 file size = 1460 file crc32 = 2429950965 example file url: http: // YOUR_TRACKER_IP/group1/M00/00/00/rules = group1, remote_filename = M00/00/00/latest ip address: YOUR_TRACKER_IPfile timestamp = 21: 03: 35 file size = 1460 file crc32 = 2429950965 example file url: http: // YOUR_TRACKER_IP/group1/M00/00/00/rbfuji keaywsjaaaftjdwg_u_big.sample # step 9. view service running details in the console/usr/bin/fdfs_monitor <client_conf_filename>

Integrate Nginx to support external network access

Nginx installation is not described here.

After Nginx is installed, modify the configuration file:

Server {listen 80; server_name YOUR_HOST_NAME; location/M00 {# Set the Host header and the real client address so that the server can obtain the real client IP address proxy_set_header host $ Host; proxy_set_header X-Real-IP $ remote_addr; proxy_set_header X-Forwarded-For $ proxy_add_x_forwarded_for; proxy_buffering off; root/home/yuqing/fastdfs/data ;}}

The root path is the base_path specified in storage. conf.

After Nginx is started successfully, access the successfully uploaded test text from the Internet using URL: YOUR_IP/M00/00/00/rbfuji ljrlmmai7zoaaaftjdwg_u_big.sample. The RESPONSE 404 is inaccessible.

View Nginx log error. log

2017/03/21 21:41:49 [error] 20562#0: *1 open() "/home/yuqing/fastdfs/data/M00/00/00/rBFUJljRLMmAI7ZOAAAFtJDWG_U_big.sample" failed (2: No such file or directory), client: ......

Cause analysis:

The file directory opened through the URL is/home/yuqing/fastdfs/data/M00/00/00, and the actual storage directory is/home/yuqing/fastdfs/data/00/00.

A non-existing Directory M00 is found: FastDFS supports multiple disks (base path), so Mxx should be used to distinguish between them. Among them, xx is the disk serial number, which is based on 0.

Solution: Create a soft connection for/home/yuqing/fastdfs/data/M00/00/00

ln -s /home/yuqing/fastdfs/data /home/yuqing/fastdfs/data/M00

Add hard disks to storage nodes

The goal is to expand and improve some performance.

Modify storage. conf attributes # Number of storage paths count (store_path) store_path_count = 2 # specify the storage path store_path0 =/home/yuqing/fastdfs # add another hard disk path store_path1 =/usr/local/dfsdata to modify tracker. conf attributes # storage server storage path selection method #0: round robin #2: server Load balancer store_path = 2

Restart two services

./fdfs_trackerd /etc/fdfs/tracker.conf restart./fdfs_storaged /etc/fdfs/storage.conf restart

After a slight delay, you can see the directory where generated files are stored in the store_path1 directory.
If you use the same test method to upload a file again, the file path starting with M01 will be generated.
/M01/00/04/rBFUJljR4zOAaxHOAAAAVJyyfIA543.txt
It is found that this is stored in the store_path1 path, exactly in line with the meaning of the previously explained M00.

To support Internet access to new Disks

Method 1

Modify some nginx. conf attributes. Location/M01 {# Set the Host header and client Real address so that the server can obtain the Real IP address proxy_set_header host $ Host of the client; proxy_set_header X-Real-IP $ remote_addr; proxy_set_header X-Forwarded-For $ proxy_add_x_forwarded_for; # disable cache proxy_buffering off; root store_path1/data ;}

Add soft connection
Ln-s/usr/local/dfsdata/data/M01

Restart the Nginx Service

Recommendation-method 2

Install the fastdfs-Nginx-module plug-in for nginx

Decompress the package and install it in the embedded INTALL file.
Note: copy the configuration file http. conf mime. types in the FastDFS installation package and mod_fastdfs.conf in the fastdfs-nginx-module package to the/etc/fdfs/directory.

Modify nginx. conf attributes: delete/M00 and/M01 add suffix match (add as needed) location ~ *. (Gif | jpg | jpeg | txt) $ {proxy_set_header Host $ host; proxy_set_header X-Real-IP $ remote_addr; proxy_set_header X-Forwarded-For $ handle; # disable cache proxy_buffering off; ngx_fastdfs_module;} modify some attributes of mod_fastdfs.conf: store_path_count = 2store_path0 =/home/yuqing/fastdfsstore_path1 =/usr/local/dfsdata

ValueRequiredConsistent with the storage. conf configuration value.
Restart the Nginx Service

Data synchronization between the same group

When the storage server defines the same group_name, they are considered to be the same group, and files in the same group are updated and synchronized to each other. Therefore, backup data is often used to improve system reliability.

(During testing, it is not feasible to run multiple storage servers by modifying the port on the same server. This will make one of the servers unavailable .)

Adding storage nodes to a group does not need to modify the tracker server configuration.

When a node is added, the system synchronizes data immediately. Therefore, the system load is increased. We recommend that you operate the node when the system is idle.

When adding a node, modify group_name to specify a group for the storage server.

The number of storage paths configured for the added nodes must be the same as the number of other nodes in the same group. That is, the value of store_path_count is equal. The specific path can be different.

Otherwise, the tracker server has the following error message:

[2017-03-22 15:56:14] ERROR - file: tracker_mem.c, line: 4592, client ip: 172.17.84.39, store_path_count 1 less than that of the group "group1", the group store_path_count is 2

Start storage server 2

View logs (Omitted part of information)

data path: /home/yuqing/fastdfs2/data, mkdir sub dir done.[2017-03-22 15:58:57] DEBUG - file: tracker_client_thread.c, line: 225, report thread to tracker server *:22122 started[2017-03-22 15:58:57] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server *:22122, as a tracker client, my ip is *[2017-03-22 15:58:57] INFO - file: tracker_client_thread.c, line: 1235, tracker server *:22122, set tracker leader: 172.17.84.38:22122[2017-03-22 15:58:57] DEBUG - file: storage_sync.c, line: 2605, sync thread to storage server *:23000 started[2017-03-22 15:58:57] INFO - file: storage_sync.c, line: 2698, successfully connect to storage server *:23000[2017-03-22 16:00:56] DEBUG - file: storage_service.c, line: 3384, client ip: 172.17.84.38, storage server id: *

As you can see, when new nodes are added to the group, their workflow is roughly as follows:

INIT

! -- 1 generate a storage directory in the specified store_path

! -- 2 enable tracker_client_thread to notify tracker server that a new node is added.

! -- 3 successfully connected to the tracker server

SYCNING

! ------- 1 enable sync thread and prepare Data Synchronization

! ------- 2 data synchronization (Time consumed)

! ------- 3. All data is synchronized successfully.

ACTIVE

! ---------- 1. become a member of the group

Test reliability
! -- S 1
If one of the storage servers in the group is lost due to rm commands or bad disk paths, other storage servers in the group will not execute synchronization commands.
After the storage server is restarted, a complete storage server is selected from the Group for incremental updates.
! -- S 2
If one of the storage servers in the group is down, the system still runs normally. (Fastdfs-nginx-module plug-in must be installed and correctly configured for each storage server)
View Nginx logs:

172.17.84.38 - - [22/Mar/2017:16:46:15 +0800] "GET /M01/00/00/rBFUJ1jSOaCAMPUNAAAAVJyyfIA114.txt?redirect=1  HTTP/1.0" 200 84 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.104 Safari/537.36 Core/1.53.2372.400 QQBrowser/9.5.10548.400"

View Cluster running status on the console

# Run the command. /fdfs_monitor/etc/fdfs/client. conf. sample # view information (Omitted) tracker server is 172.17.84.38: 22122 group count: 1 Group 1: group name = group1storage server count = 2 active server count = 2 disk total space = 80630 MBdisk free space = 67252 MB Storage 1: ip_addr = always ACTIVE join time = 10:08:23 up time = 17:08:13 hour = 16:44:55 last_source_update = 11:20:13 hour = 08:00:00 hour = 08:00:00 hour = Storage 2: ip_addr = 172.17.84.39 ACTIVE join time = 15:32:58 up time = 15:58:47 hour = 17:03:56 last_source_update = 16:45:19 last_sync_update = 16:00:02 hour = 11:20:13 (0 s delay)