Fastdfs introduction and related knowledge, and cluster construction

Fastdfs Related knowledge

What is Fastdfs?

FAstdfs is an open source, lightweight, distributed File system. It solves problems such as large data volume storage and load balancing. Particularly suitable for small and medium-sized files (recommended scope: 4KB < file_size <500MB) as the carrier of online services, such as photo album sites, video sites and so on.

Fastdfs 's role:

Tracker server:tracker Server as a central node, its main role is load balancing and scheduling.  Tracker server records information such as group and storage server status in memory, does not log file index information, and consumes little memory. the main role of Storage server in Storage Server:fastdfs is to store files, which directly use the OS file system to store files, Fastdfs not block the files.

Fastdfs The grouping mechanism:

Fastdfs uses a grouped storage method. A cluster is comprised of one or more groups, and the total capacity of the cluster storage is the sum of the storage capacity of all the groups in the cluster. A group is made up of one or more storage servers, and the storage server in the same group is a mutual-prepared relationship, and the files on the same set of storage servers are fully consistent. File upload, download, delete, etc. can be done on any storage server in the group. The storage capacity of a group is the one with the smallest storage server capacity in the group.

Fastdfs Role Relationships:

Tracker Server in a Fastdfs cluster can also have more than one, and neither Tracker server nor storage server has a single point of issue. Tracker Server is a peer relationship, and the storage server within the group is also a peer relationship.

Fastdfs the cluster architecture:

650) this.width=650; "src=" Http://s4.51cto.com/wyfs02/M02/82/DD/wKioL1djWmPj55bEAAC6caGTrEM127.gif "title=" 1.gif " alt= "Wkiol1djwmpj55beaac6cagtrem127.gif"/>

Message passing between roles:

The client and storage server actively connect to tracker server. Storage server proactively reports its status information to tracker server, including statistics such as disk space remaining, file synchronization status, file upload download times, and so on. Storage Server connects all tracker servers in the cluster and reports their status to them. A group contains storage server that is not set by the configuration file, but is obtained through tracker server.

Different groups of storage servers do not communicate with each other, and storage servers within the same group are connected to each other for file synchronization.

Storage Server synchronization of files between:

File synchronization only in the same group between the storage server, using push mode, that is, the source server synchronization to the target server, only synchronize the source data, backup data is not synchronized. When a new storage server is added, all existing data (including source data and backup data) is synchronized to the new server by an existing storage server.

Storage Server uses Binlog file to record file upload, delete and other update operations. Only file names are recorded in Binlog, and the contents of the files are not recorded. File synchronization takes the form of incremental synchronization, where the system records the synchronized location (binlog file offset) to the identity file.

The interactive process for file upload and download:

file upload steps:

1,client asked tracker server to upload the storage server;

2,tracker server returns an available storage server with the data returned as the IP address and port of the storage server;

3,client directly and the Storage server to establish a connection, file upload, Storage server returns the newly generated file ID, file upload end.

file download steps:

1 , the client asks tracker server to download the storage server for the specified file, with the file ID (containing the group name and file name);

2 , Tracker server returns one of the available storage servers;

3 , the client establishes a connection directly to the storage server and completes the file download.

Fastdfs the file ID:

When the client uploads the file, it returns a Fileid. The file ID is not specified by the client, but is returned to the client after it is generated by storage server. The file ID contains the group name, the file-relative path, and the file name, and Storage server can navigate directly to the file based on the file ID.

Fastdfs expansion of:

Fastdfs Expansion and horizontal expansion. Vertical refers to the addition of servers in the same group to achieve data redundancy. Horizontal refers to the addition of a group that increases the storage space of the entire FASTDFS.

Fastdfs cluster installation

Lab Environment:

Hardware environment:

tracker:192.168.189.146

storage:192.168.189.145 (group1)

storage:192.168.189.144 (group2)

Software Environment:

Libevent-2.0.18-stable

fastdfs_v5.02

One: Fastdfs installation of: ---------Tracker with the Storage consistent.

1 : Download libevent with the Fastdfs the installation package to the server. /usr/local/src directory under :

2 : Installation libevent:

Libevent version of the system is low, need to be removed, or the installation Fastdfs will be error: #rpm-e--nodeps ibevent-1.4.13-4.el6.x86_64 decompression, installation libevent#cd/usr/local/src# TAR-ZXVF LIBEVENT-2.0.18-STABLE.TAR.GZ#CD CD libevent-2.0.18-stable#./configure--prefix=/usr/local/libevent#make && make install associated Libeven library file to System (64-bit system is placed under Lib64) #ln-S/usr/local/lib/libevent*/lib64/or #ln-s/usr/local/lib Event/lib/libevent-2.0.so.5/usr/lib64/libevent-2.0.so.5

3 : Installation Fastdfs :

Unzip, install fastdfs#cd/usr/local/src && tar-zxvf fastdfs_v5.02.tar.gz#cd fastdfs_v5.02 Edit make.sh file, modify compilation options #vi make.sh Target_prefix=/usr/local/fastdfs target_conf_path=/usr/local/fastdfs/conf#./make.sh C_INCLUDE_PATH=/usr/ local/libevent/includelibrary_path=/usr/local/libevent/lib#./make.sh Install

Two: Tracker with the Storage the configuration:

  1 : Configure tracker :

#vi/usr/local/fastdfs/conf/tracker.conf disabled=false bind_addr=192.168.189.146             (not configured to bind 0.0.0.0)  port=22122 base_path=/data/fastdfs           (storing logs and data paths)  store_lookup=2               (Specify the rules for uploading files, 2 for balanced uploads, file uploads to large groups of remaining space)  store_group=group2               (only effective when store_lookup=1)  use_ storage_id = true             (Enable storage  id  convenient storage address replacement)  storage_ids_filename = storage_ids.conf  copy Storage_ Ids.conf to the configuration directory because storageid#cp -p /usr/local/src/fastdfs/conf/storage_ids.conf /usr/local/is enabled fastdfs/conf/Edit storage_ids.conf (if storage server information does not match the file, Storage server will not start) #vi/usr/local/fastdfs/ conf/storage_ids.conf  100001   group1  192.168.189.145 100002   group2   192.168.189.144

2: Configuration storagegroup1:192.168.189.145

#vi/usr/local/fastdfs/conf/storage.conf disabled=false group_name=group1 (specifies the group to which the server belongs) bind_addr=192.168.189      .145 client_bind=true port=23000 Base_path=/data/fastdfs (storing logs and data) Store_path0=/data/fastdfs/storage (Data storage path, no configuration specified to Base_path) tracker_server=192.168.189.146:22122 (Specify tracker server) upload_priority=10 (upload excellent First level)

3: Configuration storagegroup2:192.168.189.144---- configuration with 145

#vi/usr/local/fastdfs/conf/storage.conf disabled=false group_name=group2 bind_addr=192.168.189.144 Client_bind=true port=23000 Base_path=/data/fastdfs tracker_server=192.168.189.146:22122 upload_priority=10

          

Three: Start up Tracker with the Storage :

Tracker (146): #/usr/local/fastdfs/bin/fdfs_trackerd/usr/local/fastdfs/conf/tracker.confstoragegroup1 (145): #/usr /local/fastdfs/bin/fdfs_storaged/usr/local/fastdfs/conf/storage.confstorage group1 (144): #/usr/local/fastdfs/bin /fdfs_storaged/usr/local/fastdfs/conf/storage.conf

Four: Test:

1 : Test Tracker with the Storage the connection

#netstat-an | grep 22122

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M01/82/DD/wKioL1djXPWzONssAAAQ7tmRitI224.png "title=" 1.png " alt= "Wkiol1djxpwzonssaaaq7tmriti224.png"/>

You can also view the storage information using the Fastdfs 's own monitor tool : ( using client.conf)

#vi/usr/local/fastdfs/conf Base_path=/data/fastdfs tracker_server=192.168.189.146:22122 #/usr/local/fastdfs/bin/ Fdfs_monitor/usr/local/fastdfs/conf/client.conf

2 : Test File Upload:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/82/DE/wKiom1djXDPiKleZAAARjfEYrJs301.png "title=" 2.png " alt= "Wkiom1djxdpiklezaaarjfeyrjs301.png"/>

You can see that the file is uploaded to the relevant directory in group1.

Fastdfs can go to the storage storage directory directly to find the relevant upload file.

This article from the "Play God Clown" blog, reproduced please contact the author!

Fastdfs introduction and related knowledge, and cluster construction