Distributed building-Simple version file upload download server Fastdfs

I. Introduction of FASTDFS

Fastdfs Open Source Address: https://github.com/happyfish100

Reference: Fastdfs design principles for Distributed file systems

Reference: Fastdfs Distributed File System

Fastdfs Java Api:https://github.com/bojiangzhou/lyyzoo-fastdfs-java for personal encapsulation

1. Introduction

Fastdfs is an open source, high performance Distributed File System (DFS). Its main features include: file storage, file synchronization and file access, as well as high capacity and load balancing. The main solution to the massive data storage problem, especially suitable for small and medium files (recommended scope: 4KB < file_size <500MB) as the carrier of online services.

The FASTDFS system has three roles: a Tracking Service (Tracker server), a storage server (Storage server), and a client.

TrackerServer: Tracking server, mainly to do scheduling work, play a balanced role, responsible for the management of all storage server and group, each storage after launch will connect Tracker, inform oneself belong to group and other information , and maintain a periodic heartbeat.

StorageServer: A storage server that mainly provides capacity and backup services, in group, each group can have more than one Storage server, the data are backed up.

Client: Clients, the server that uploads the downloaded data, which is where our own projects are deployed.

2. Fastdfs Storage Policy

To support large capacity, storage nodes (servers) are organized in a sub-volume (or grouping) manner. The storage system consists of one or more volumes, and the files between the volumes and volumes are independent of each other, and the cumulative file capacity of all volumes is the file capacity of the entire storage system. A volume can consist of one or more storage servers, and the files in a storage server under a volume are the same, and multiple storage servers in the volume play a role of redundant backup and load balancing.

When adding servers to a volume, synchronizing existing files is done automatically by the system, and after synchronization is complete, the system automatically switches the new server to the online service. You can add volumes dynamically when storage space is low or is about to be exhausted. You only need to add one or more servers and configure them as a new volume, which increases the capacity of the storage system.

3, the Fastdfs upload process

FASTDFS provides users with basic file access interfaces, such as upload, download, append, delete, etc., which are provided to the user in the form of client libraries.

Storage server periodically sends its own storage information to the tracker server. When there is more than one tracker server in the tracker server cluster, the relationship between the tracker is peer-on, so the client can select any tracker when uploading.

When tracker receives a request from the client to upload a file, the file is assigned a group that can store the file, and when group is selected, it is decided to assign the client to which storage server in the group. When storage server is assigned, the client sends a write file request to storage, and storage assigns a data store directory to the file. Then assign a fileid to the file, and finally generate a file name to store files based on the information above.

4, Fastdfs file synchronization

When a file is written, the client writes the file to a storage server within the group that the file is considered successful, and after the storage server finishes writing the file, the file is synchronized by the background thread to the other storage servers in the same group.

Each storage write a file, at the same time will write a copy of the Binlog,binlog does not include the file data, only the file name and other meta-information, the Binlog for background synchronization, storage will record the progress of other storage synchronization in the group, So that the last progress can be resumed after the restart, the progress is recorded in a timestamp, so it is best to keep the clocks of all servers in the cluster in sync.

Storage synchronization progress will be reported to the tracker as part of the metadata, Tracke will use synchronization progress as a reference when choosing a read storage.

5, fastdfs file download

After the client UploadFile succeeds, it will get a file name generated by the storage, and the client can then access the file based on the file name.

As with upload file, the client can select any tracker server when downloadfile. Tracker send download request to a tracker, must bring the file name information, Tracke from the file name to resolve the group, size, creation time and other information, and then select a storage for the request to service read requests.

Second, installation FASTDFS environment 0, preface

Operating environment: CentOS7 X64, the following operations are a stand-alone environment.

I downloaded all the installation packages to/softpackages/and extracted them into the current directory.

First do one thing, modify the hosts, the file server's IP and Domain name Mapping (single-machine trackerserver environment), because many of the configuration inside to configure the server address, IP changed, only need to modify the hosts.

# vim/etc/hosts Add the following line, this is my IP192.168.51.128 file.ljzsg.com if you want to natively access the virtual machine, in the C:\Windows\System32\drivers\etc\ The hosts also added a row 1, download install Libfastcommon

Libfastcommon is the public C function library extracted from Fastdfs and FASTDHT, and the basic environment can be installed.

① Download Libfastcommon

# wget https://github.com/happyfish100/libfastcommon/archive/V1.0.7.tar.gz

② Decompression

# TAR-ZXVF v1.0.7.tar.gz# CD libfastcommon-1.0.7

③ Compiling, installing

#./make.sh#./make.sh Install

④libfastcommon.so is installed to/usr/lib64/libfastcommon.so, but the Lib directory fastdfs the main program settings is/usr/local/lib, so you need to create a soft link.

# ln-s/usr/lib64/libfastcommon.so/usr/local/lib/libfastcommon.so# ln-s/usr/lib64/libfastcommon.so/usr/lib/ libfastcommon.so# ln-s/usr/lib64/libfdfsclient.so/usr/local/lib/libfdfsclient.so# ln-s/usr/lib64/ LIBFDFSCLIENT.SO/USR/LIB/LIBFDFSCLIENT.SO2, download and install Fastdfs

① Download Fastdfs

# wget https://github.com/happyfish100/fastdfs/archive/V5.05.tar.gz

② Decompression

# TAR-ZXVF v5.05.tar.gz# CD fastdfs-5.05

③ Compiling, installing

#./make.sh#./make.sh Install

④ the appropriate files and directories after installation of the default installation method
A, Service script:

/etc/init.d/fdfs_storaged/etc/init.d/fdfs_tracker

B, configuration file (these three are the sample configuration files given by the author):

/etc/fdfs/client.conf.sample/etc/fdfs/storage.conf.sample/etc/fdfs/tracker.conf.sample

C, command tools in the/usr/bin/directory:

Fdfs_appender_testfdfs_appender_test1fdfs_append_filefdfs_crc32fdfs_delete_filefdfs_download_filefdfs_file_ Infofdfs_monitorfdfs_storagedfdfs_testfdfs_test1fdfs_trackerdfdfs_upload_appenderfdfs_upload_ filestop.shrestart.sh

The bin directory for the ⑤fastdfs service script settings is/usr/local/bin, but the actual command is installed under/usr/bin/.

Two different ways:

"One is to modify the corresponding command path in the Fastdfs service script, which is to change the/usr/local/bin of the/etc/init.d/fdfs_storaged and/etc/init.d/fdfs_tracker two scripts to/usr/bin.

# Vim Fdfs_trackerd
Using the Find and replace command in Unified modification:%s+/usr/local/bin+/usr/bin
# Vim Fdfs_storaged
Using the Find and replace command in Unified modification:%s+/usr/local/bin+/usr/bin

　　　　

The second is to establish/usr/bin to/usr/local/bin soft link, I was in this way.

# ln-s/usr/bin/fdfs_trackerd/usr/local/bin# ln-s/usr/bin/fdfs_storaged/usr/local/bin# ln-s/usr/bin/stop.sh/usr/lo cal/bin# ln-s/usr/bin/restart.sh/usr/local/bin3, configuring Fastdfs Tracker (Tracker)

Configuration file Details Reference: Fastdfs configuration file Detailed description

① into/etc/fdfs, copy the Fastdfs Tracker sample configuration file tracker.conf.sample and rename it to tracker.conf.

# cd/etc/fdfs# CP tracker.conf.sample tracker.conf# vim tracker.conf

② edit tracker.conf, red need to modify the other, the default can be.

# The configuration file does not take effect, false is valid Disabled=false
# Port for service port=22122
# Tracker data and log directory address (root directory must exist, subdirectories are created automatically) Base_path=/ljzsg/fastdfs/tracker
# HTTP Service Port http.server_port=80

③ Create tracker base data directory, which is the base_path corresponding directory

# mkdir-p/ljzsg/fastdfs/tracker

④ Open the trace port in the firewall (default 22122)

# Vim/etc/sysconfig/iptables
Add the following port line:-a input-m State--state new-m tcp-p TCP--dport 22122-j ACCEPT
Restart Firewall: # service iptables restart

⑤ Start Tracker

For the first successful start, data, logs two directories are created under/ljzsg/fdfsdfs/tracker/(configured Base_path).

This can be used to start the #/etc/init.d/fdfs_trackerd start can also be started in this way, provided that the above created a soft link, followed by this way # service Fdfs_trackerd start

To see if the Fastdfs Tracker has been successfully started and port 22122 is being monitored, the installation of the Tracker service was successful.

# Netstat-unltp|grep Fdfs

To close the tracker command:

# Service Fdfs_trackerd Stop

⑥ Setting up Tracker boot

# Chkconfig Fdfs_trackerd on

Or:
# vim/etc/rc.d/rc.local
To join the configuration:
/etc/init.d/fdfs_trackerd start

⑦tracker Server directory and file structure

After the tracker service starts successfully, the data, logs two directories are created under Base_path. The directory structure is as follows:

${base_path} |__data | |__storage_groups.dat: storage grouping Information | |__storage_servers.dat: Storage server List |__logs | |__trackerd.log:tracker Server log file 4, configuring Fastdfs storage (Storage)

① into the/etc/fdfs directory, copy the FASTDFS memory sample configuration file Storage.conf.sample, and rename it to storage.conf

# cd/etc/fdfs# CP storage.conf.sample storage.conf
# Vim Storage.conf

② Edit storage.conf

Red needs to be modified, the other default.

# If the profile does not take effect, false for Disabled=false # Specifies this storage server group (volume) group_name=group1# Storage Server service port port=23000# heartbeat interval, single Bits are seconds (here is the active heartbeat to tracker server) heart_beat_interval=30# Storage data and log directory address (root directory must exist, subdirectories are automatically generated) Base_path=/ljzsg/fastdfs /storage# Storage Server supports multiple paths when storing files. This configures the number of base paths where the files are stored, usually with only one directory. store_path_count=1# each configuration Store_path_count path, the index number is based on 0. # If you do not configure Store_path0, it will be the same as the base_path corresponding to the path. store_path0=/ljzsg/fastdfs/file# Fastdfs store files, a level two directory is used. This configures the number of directories where the files are stored. # If this parameter is only n (for example: 256), storage server will automatically create a subdirectory of n * n stored files under Store_path when it is first run. subdir_count_per_path=256# a list of tracker_server, each tracker server writes a line when the tracker_server# has multiple tracker servers actively connected Tracker_se rver=file.ljzsg.com:22122

# The time period that allows the system to synchronize (default is all day). Generally used to avoid peak synchronization to create some problems and set.
sync_start_time=00:00
sync_end_time=23:59# Access Port
Http.server_port=80

③ Create storage base data directory, corresponding to Base_path directory

# Mkdir-p/ljzsg/fastdfs/storage# This is the configured Store_path0 path # mkdir-p/ljzsg/fastdfs/file

④ Open the memory port in the firewall (default 23000)

# Vim/etc/sysconfig/iptables Add the following port line:-a input-m State--state new-m tcp-p TCP--dport 23000-j accept reboot firewall: # service IP Tables restart

⑤ Start Storage

Make sure the tracker is started before starting storage. For the first successful launch, data and logs two directories will be created in the/ljzsg/fastdfs/storage directory.

You can start it this way #/etc/init.d/fdfs_storaged start can also be used in this way, followed by this # service fdfs_storaged start

To see if Storage started successfully, Port 23000 is being monitored, even if Storage is successful.

# Netstat-unltp|grep Fdfs

To close the storage command:

# Service Fdfs_storaged Stop

To see if storage and tracker are communicating:

/usr/bin/fdfs_monitor/etc/fdfs/storage.conf

⑥ Setting up Storage boot

# Chkconfig fdfs_storaged on

Or:
# vim/etc/rc.d/rc.local
To join the configuration:
/etc/init.d/fdfs_storaged start

⑦storage Directory

After successful startup with Tracker,storage, the data and logs directories are created under Base_path and the information of Storage Server is recorded.

Under the Store_path0 directory, a n*n subdirectory is created:

5. File Upload Test

① modifying client configuration files in the Tracker server

# cd/etc/fdfs# CP client.conf.sample client.conf# vim client.conf

Modify the following configuration, other default.

# Client data and log directory base_path=/ljzsg/fastdfs/client# tracker Port tracker_server=file.ljzsg.com:22122

② Upload Test

Execute the following command inside Linux upload namei.jpeg image

#/usr/bin/fdfs_upload_file/etc/fdfs/client.conf Namei.jpeg

Return file ID number after successful upload: Group1/m00/00/00/wkgz6lnduteamdrcaaeormxzpp870.jpeg

The returned file ID is stitched together by group, storage directory, level two subdirectory, Fileid, file suffix name (specified by the client, primarily for distinguishing file types).

Third, install Nginx

The file upload was successful, but we can't download it. Therefore, Nginx is installed as a server to support HTTP access to files. At the same time, the installation of Fastdfs in the rear Nginx module also requires nginx environment.

Nginx only needs to be installed on the server where Storageserver is located to access the files. I am here because it is a single machine, Trackerserver and Storageserver on a server.

1, installation of the environment required Nginx

①GCC Installation

# yum Install gcc-c++

②pcre Pcre-devel Installation

# yum Install-y pcre Pcre-devel

③zlib Installation

# yum Install-y zlib zlib-devel

④OPENSSL Installation

# yum Install-y OpenSSL openssl-devel2, installing Nginx

① Download Nginx

# wget-c Https://nginx.org/download/nginx-1.12.1.tar.gz

② Decompression

# TAR-ZXVF nginx-1.12.1.tar.gz# CD nginx-1.12.1

③ using the default configuration

#./configure

④ Compiling, installing

# make# make Install

⑤ start Nginx

# cd/usr/local/nginx/sbin/#./nginx other commands #./nginx-s stop#./nginx-s quit#./nginx-s Reload

⑥ Set Boot start

# vim/etc/rc.local Add a line:/usr/local/nginx/sbin/nginx

# Set Execution permissions
# chmod 755 rc.local

⑦ view Nginx version and module

/usr/local/nginx/sbin/nginx-v

⑧ open Nginx port in firewall (default 80)

Once added, it can be accessed using port 80 on the local computer.

# Vim/etc/sysconfig/iptables Add the following port line:-a input-m State--state new-m tcp-p TCP--dport 80-j accept reboot firewall: # service Iptab Les restart

3. Access to Files

Simple test access to files

① modifying nginx.conf

# vim/usr/local/nginx/conf/nginx.conf Add the following line to map/group1/m00 to/ljzsg/fastdfs/file/datalocation/group1/m00 {alias/ Ljzsg/fastdfs/file/data;}

# Restart Nginx
#/usr/local/nginx/sbin/nginx-s Reload

② uploaded pictures Before browser access, success.

Http://file.ljzsg.com/group1/M00/00/00/wKgz6lnduTeAMdrcAAEoRmXZPp870.jpeg

Four, FASTDFS configuration Nginx Module 1, Installation and configuration Nginx module

①fastdfs-nginx-module Module Description

Fastdfs the file is stored on the Storage server through the Tracker server, but file replication is required between the same set of storage servers and there is a problem of synchronization latency.

Assuming the Tracker server uploads the file to 192.168.51.128, the file ID has been returned to the client after the upload is successful.

At this point the FASTDFS storage cluster mechanism will synchronize this file to the same set of storage 192.168.51.129, if the file is not yet complete, the client if the file ID on the 192.168.51.129 to fetch files, there will be a file inaccessible error.

Instead, Fastdfs-nginx-module can redirect files to the source server to fetch files, preventing the client from being unable to access errors due to replication delays.

② Download Fastdfs-nginx-module, unzip

# Why this is so long a string, because the latest version of Master and the current Nginx some version of the problem. # wget Https://github.com/happyfish100/fastdfs-nginx-module/archive/5e5f3566bbfa57418b5506aaefbe107a42c9fcb1.zip # unzip # unzip 5e5f3566bbfa57418b5506aaefbe107a42c9fcb1.zip# Rename # MV FASTDFS-NGINX-MODULE-5E5F3566BBFA57418B5506AAEFBE107A42C9FCB1 Fastdfs-nginx-module-master

③ configuration Nginx

adding modules to Nginx

# First stop the Nginx service
#/usr/local/nginx/sbin/ngix-s Stop

Go to unpack Package directory # cd/softpackages/nginx-1.12.1/# Add Module #./configure--add-module=. /FASTDFS-NGINX-MODULE-MASTER/SRC recompile, install # make && make install

④ Viewing Nginx modules

#/usr/local/nginx/sbin/nginx-v

The following is a description of the success of the Add module

⑤ Copy the configuration files in the Fastdfs-nginx-module source to the/etc/fdfs directory, and modify

# CD/SOFTPACKAGES/FASTDFS-NGINX-MODULE-MASTER/SRC

# CP mod_fastdfs.conf/etc/fdfs/

Modify the following configuration, other default

# Connection Time-out
connect_timeout=10# Tracker servertracker_server=file.ljzsg.com:22122
# storageserver Default Port storage_server_port=23000# If the URI of the file ID contains/group**, set to Trueurl_have_group_name = true# Storage The configured Store_path0 path must be consistent with the storage.conf Store_path0=/ljzsg/fastdfs/file

⑥ copy some of the Fastdfs configuration files to the/etc/fdfs directory

# cd/softpackages/fastdfs-5.05/conf/# CP anti-steal.jpg http.conf mime.types/etc/fdfs/

⑦ configuration Nginx, modify the nginx.conf

# vim/usr/local/nginx/conf/nginx.conf

Modify the configuration, the other default

Add the Fastdfs-nginx module under Port 80

Location ~/group ([0-9])/m00 {ngx_fastdfs_module;}

Attention:

The Listen 80 port value is to correspond to the http.server_port=80 in/etc/fdfs/storage.conf (previously changed to 80). If you change to a different port, you need to unify and open the port in the firewall.

Location configuration, if more than one group is configured with location ~/group ([0-9])/m00, no group is used.

⑧ creates a soft connection in the/ljzsg/fastdfs/file file store directory, linking it to the directory where the data is actually stored, which can be omitted.

# ln-s/ljzsg/fastdfs/file/data//ljzsg/fastdfs/file/data/m00

⑨ start Nginx

#/usr/local/nginx/sbin/nginx

Print at the following even if the configuration is successful

⑩ is accessed in the Address bar.

Can download the file even if the installation is successful. Note and the 3rd in the direct use of Nginx routing access is different, here Configuration Fastdfs-nginx-module module, can redirect files linked to the source server to fetch files.

Http://file.ljzsg.com/group1/M00/00/00/wKgz6lnduTeAMdrcAAEoRmXZPp870.jpeg

Final deployment diagram (stolen diagram): The environment can be built according to the following structure.

V. Java Client

The previous file system platform is set up, now it is necessary to write the client code in the system to implement upload and download, here is just a simple test code.

1, first need to build FASTDFS client Java development environment

Using Maven for dependency management in the ① project, you can introduce the following dependencies in the Pom.xml:

<dependency> <groupId>net.oschina.zcx7878</groupId> <artifactid>fastdfs-client-java</ Artifactid> <version>1.27.0.0</version></dependency>

Other ways, refer to the Official document: Https://github.com/happyfish100/fastdfs-client-java

② introducing a configuration file

You can directly copy the Fastdfs-client.properties.sample or fdfs_client.conf.sample under the package to your project and remove the. Sample.

Here I directly copy the configuration in Fastdfs-client.properties.sample to the Project Profile Config.properties, modify the tracker_servers. Just load this configuration file to

2. Client API

Fastdfs Java API for personal encapsulation to sync to Github:https://github.com/bojiangzhou/lyyzoo-fastdfs-java.git

Vi. Control of permissions

The previous use of Nginx support HTTP access to the file, but everyone can directly access the file server, so do permission control.

Fastdfs's permission control is to turn on token authentication on the server, the client obtains tokens based on the file name, the current UNIX timestamp, the secret key, and takes the token parameter in the address to access the file in HTTP mode.

① Service-side open token verification

Modify http.conf# vim/etc/fdfs/http.conf set to True to turn on token verification http.anti_steal.check_token=true

Sets the time unit for token invalidation to seconds (s)
http.anti_steal.token_ttl=1800
The key is consistent with the Fastdfs.http_secret_key of the client configuration file http.anti_steal.secret_key=fastdfs1234567890 If the token check fails, the returned page http.anti_ Steal.token_check_fail=/ljzsg/fastdfs/page/403.html

Remember to restart the service.

② Configuring the Client

The client only needs to set the following two parameters, the keys on both sides remain the same.

# token anti-theft chain function fastdfs.http_anti_steal_token=true# key fastdfs.http_secret_key=fastdfs1234567890

③ Client Generation Token

Access to the file requires the resulting token and the Unix timestamp, so the token returned is a concatenation of token and timestamp.

After that, you can access the token by stitching it in the address: file.ljzsg.com/group1/m00/00/00/wkgzgfnkaxqaifxyaaeormxzpp878.jpeg?token= 078d370098b03e9020b82c829c205e1f&ts=1508141521

④ precautions

If the generated token verification fails, make the following two checks:
A. Confirm that the token generation function (Protocommon.gettoken) is called, and that the file ID passed does not contain group name. The file ID is passed in the form of a format such as: M00/00/00/wkgzgfnktpyaiaugaaeormxzpp876.jpeg

B. Confirm that the server time is basically the same, note that the server time can not vary too much, do not differ to the minute level.

⑤ comparison found that if the system file privacy is high, you can directly through the Fastdfs-client provided API to access, no longer configure Nginx go HTTP access. The main purpose of the configuration Nginx is to quickly access the server's files (slices), if you want to add permission validation, you need to generate tokens, in fact, it does not make much sense.

The point is, I didn't find Fastdfs. How to add token verification to some resources, partially open. Have to know also please leave a message.

Distributed building-Simple version file upload download server Fastdfs