FastDFS Distributed File System, fastdfs

FastDFS Distributed File System, fastdfs

I. FastDFS:

Is a distributed file system with file storage, file synchronization, and file access to solve a large number of file storage problems.

II.FastDFS: Architecture

Client:
Client
Tracker:
Tracker is a tracking server that keeps an eye on the Storage server and serves as a bridge between storage and client.
Storage:
File storage server, used to store files

　　Workflow:

<1> stand-alone edition:

1. Single-Host Configuration:

Because FastDFS must be compiled before it can be installed (written in C) after being installed and decompressed, we need to rely on gcc.

A: yum install gcc-c ++ perl-devel pcre-devel openssl-devel zlib-devel wget // This dependency is used to compile programs written in c language.

After FastDFS is compiled, make install is required for installation. During the installation process, it depends on libfastcommon.

B: wget https://github.com/happyfish100/libfastcommon/archive/V1.0.35.tar.gz

Start downloading the FastDFS File

C: wget https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz // finally built with nginx later than version 5.11

The above operations a, B, and c are all preparations.

Now we will decompress the downloaded V1.0.35.tar.gz and V5.11.tar.gz respectively.

D: tar-zxvf V1.0.35.tar.gz unzip to the decompressed file cd libfastcommon-1.0.35/execute./make. sh to complete the compilation./make. sh install to complete the installation

E: tar-zxvf V5.11.tar.gz unzip to the decompressed file cd fastdfs-5.10/run./make. sh to complete the compilation./make. sh install to complete the installation

The above operations d and e have completed FastDFS installation. Next, let's see where the file is installed.

After the installation is complete, the FastDFS Distributed File System will be divided into configuration file Startup Script Commands

Configuration file --> stored in the/etc/fdfs/directory

Here we need to put the client. conf. sample, storage. conf. sample and tracker. conf. sample configuration files

Run the cp command (cp/etc/fdfs/tracker. conf. sample/etc/fdfs/tracker. conf) to rename the three files named client. conf.

It is very important that he will be our Startup File

Start the script --> store it in the/etc/init. d/directory.

/Etc/init. d/fdfs_tracker start // start

Related commands --> the commands that are stored in the/usr/bin directory start with fdfs _ are similar to uploading, downloading, and deleting file operations.

In this step, do not worry. First, create a file as the data storage location generated during the FastDFS operation.

Run the mkdir-p/data/fastdfs/{tracker, storage/store} command/data/fastdfs/storage/store to store the file directory.

The last step is configured.

Go to our configuration file, open the tracker. conf file, modify its base_path =/data/fastdfs/tracker, save and exit.

Open storage. conf (configure the storage server)

Vi/etc/fdfs/storage. conf

Base_path =/data/fastdfs/storage

Store_path0 =/data/fastdfs/storage/store/

Group_name = group2

Tracker_server = 192.168.128.141: 22122 // The tracking server's ip: port points to the tracker server and can be configured with multiple

　　　　　　　　Configure the client Configurator

Vi/etc/fdfs/client. conf
Base_path =/tmp
Tracker_server = 192.168.128.141.22122 // The tracking server's ip Address: port pointing to the tracker server can be configured with multiple

　　<2> cluster version construction:

The cluster version is easy to build: you only need to configure multiple tracker tracking services on the file configuration in the last step of the standalone version.

Summary Process Execution Process:

The client sends a request to the tracker, which monitors multiple storage file storage servers (from the storage. in the conf file, you can know that the file storage server is managed by XXX tracker, but the file server can also build a cluster, that is, the storage server is handed over to multiple trackers for management, but how can we know which tracker trackers can be known from the client. One final point is to build a FastDFS cluster, which starts from a tracker, three trackers, and six storage servers, storage servers are divided into three groups, one group, two storage servers (they can automatically complete the backup), and then divided by groups, each group has three trackers to achieve high performance, high concurrency and high availability. The client is used to allocate which tracker is called)

Iii. Integrate FastDFS with nginx

A. problems that occur when nginx is not used:
1. FastDFS's HTTP service is relatively simple and cannot provide high-performance services such as server Load balancer.

2. There is a delay in file backup between the same volumes. If the client accesses the file during this period, the problem may occur.

3. When the file is accessed, the IP address is always changed.

B. Benefits of using nginx:

To solve the above problem, nginx is used as a reverse proxy to shield the IP addresses from differences. It has a high-performance load balancing, And the FastDFS module nginx solves the errors caused by file backup delay.

Redirect to the source server to retrieve files

C. Build Nginx

1. Download

Yum install git

# Fastdfs needs to be greater than or equal to 5.11

Git clone https://github.com/happyfish100/fastdfs.git
./Make. sh clean
./Make. sh &./make. sh install
# Fastdfs module dependency
Https://github.com/happyfish100/fastdfs-nginx-module.git
# Nginx
Wget http://nginx.org/download/nginx-1.11.1.tar.gz

2. Installation

　　　　　　　　Tar-zxvf nginx-1.11.1.tar.gz

Cd nginx-1.11.1
# Nginx and fastdfs module Compilation
./Configure -- add-module =/root/fastdfs-nginx-module/src/-- prefix =/usr/local/nginx
# Install nginx
Make & make install
# Copy mod_fastdfs.conf to the/etc/fdfs directory
Cp/root/fastdfs-nginx-module/src/mod_fastdfs.conf/etc/fdfs/
# Copy the fastdfs-5.10/confanti-steal.jpg, http. conf mime. types File
Cd/root/fastdfs/conf/
Cp http. conf mime. types anti-steal.jpg/etc/fdfs/

3. Configuration

　　　　　　　　# Modify nginx. conf

　　　　　　　　Location ~ /Group ([0-2])/M00 {

Root/data/fastdfs/storage/store;
Ngx_fastdfs_module;
}
# Modify the/etc/fdfs/mod_fastdfs.conf configuration file
Tracker_server = 192.168.128.141: 22122
Url_have_group_name = true
Group_name = group1
Store_path0 =/data/fastdfs/storage/store

4. Start nginx

Cd/usr/local/nginx/sbin
./Nginx-t
./Nginx-c/usr/local/nginx/conf/nginx. conf

After completing nginx and FastDFS setup, you can access the folder through HTTP. However, when accessing the folder, you need to change the IP address at any time. Next, you need to build a proxy service for nginx.

Upstream fdfs_group1 {

Server 192.168.128.20.: 80 weight = 1 max_fails = 2 fail_timeout = 30 s;

Server 192.168.128.151: 80 weight = 1 max_fails = 2 fail_timeout = 30 s;

}

Upstream fdfs_group2 {

Server 192.168.128.150: 80 weight = 1 max_fails = 2 fail_timeout = 30 s;

Server 192.168.128.152: 80 weight = 1 max_fails = 2 fail_timeout = 30 s;

}