Distributed Architecture Learning: Installation and use of 017--fastdfs Distributed File System (single node)

Fastdfs is a lightweight open source Distributed File system
Fastdfs mainly solves the problem of large capacity file storage and high concurrency access, and realizes load balance in file access.
FASTDFS implements a software-style raid that can be stored using a cheap IDE hard disk
Support Storage Server Online expansion
Files that support the same content save only one copy, saving disk space
Fastdfs can only be accessed through the client API and does not support the POSIX access method
Fastdfs is especially suitable for medium and large Web sites to store resource files (e.g. pictures, documents, audio, video, etc.)

System architecture-Upload file flowchart

1. Client asked tracker upload to the storage, do not need additional parameters;
2. Tracker returns a usable storage;
3. Client direct and storage communication complete file upload.

System architecture-Download file flowchart

1. Client Inquiry Tracker Download file storage, parameter for file identification (group name and filename);
2. Tracker returns a usable storage;
3. Client direct and Storage communication complete file download.

Tracker server: Tracking servers, mainly to do scheduling work, in the access load balanced role. Records the status of the storage server and is the hub for connecting client and storage servers.
Storage Server: Storage servers, files, and meta data are saved to the storage server
Group: Groups can also be called volumes. Files on servers in the same group are exactly the same
File identification: Includes two parts: group name and file name (including path)
Meta data: File-related properties, key value pairs (key values Pair), such as: width=1024,heigth=768

Storage servers in the same group are peers, and file uploads, deletions, and other operations can be performed on any storage server;
The file synchronization is only carried out between the storage server in the same group, and the source server is synchronized to the target server by the push method;
source data need to sync, backup data does not need to sync again, otherwise it will constitute a loop;
The exception to the second rule is that when a new storage server is added, an existing storage server synchronizes all existing data, including source data and backup data, to the new server.

The protocol package consists of two parts: header and body
Header A total of 10 bytes, the format is as follows:
8 bytes Body Length
1 byte command
1 byte status
The body packet format depends on the specific command, the body can be empty

Runtime directory Structure-tracker server

Runtime directory Structure-storage server

Fastdfs Chinese: http://www.csource.org/
Fastdfs English: http://code.google.com/p/fastdfs/

installation and use of Fastdfs Distributed File System (single node)

Tracking Device: 192.168.4.121 (edu-dfs-tracker-01)

Storage server: 192.168.4.125 (edu-dfs-storage-01)

Environment: CentOS 6.6

User: Root

Data directory:/fastdfs (Note: Data directory According to your data disk mount path depends)

Install package:

Fastdfs v5.05

Libfastcommon-master.zip(is the public C extracted from Fastdfs and FASTDHT) function library)

fastdfs-nginx-module_v1.16.tar.gz

nginx-1.6.2.tar.gz fastdfs_client_java._v1.25.tar.gz

Source Address: Https://github.com/happyfish100/

Download Address: http://sourceforge.net/projects/fastdfs/files/

Official forum: http://bbs.chinaunix.net/forum-240-1.html

One, all tracking servers and storage servers perform the following actions

1, compile and install the required dependency pack:

# yum install make CMake gcc gcc-c++

2, installation Libfastcommon:

(1) Upload or download libfastcommon-master.zip to/usr/local/src directory

(2) Decompression

# cd/usr/local/src/

# Unzip Libfastcommon-master.zip

# CD Libfastcommon-master

(3) Compiling, installing

#./make.sh

#./make.sh Install

Libfastcommon is installed by default.

/usr/lib64/libfastcommon.so

/usr/lib64/libfdfsclient.so

(4) Because the Lib directory set by the FASTDFS main program is/usr/local/lib, you need to create a soft link.

# ln-s/usr/lib64/libfastcommon.so/usr/local/lib/libfastcommon.so

# ln-s/usr/lib64/libfastcommon.so/usr/lib/libfastcommon.so

# ln-s/usr/lib64/libfdfsclient.so/usr/local/lib/libfdfsclient.so

# ln-s/usr/lib64/libfdfsclient.so/usr/lib/libfdfsclient.so

3. Installation Fastdfs

(1) Upload or download Fastdfs source package (fastdfs_v5.05.tar.gz) to/usr/local/src directory

(2) Decompression

# cd/usr/local/src/

# TAR-ZXVF Fastdfs_v5.05.tar.gz

# CD Fastdfs

(3) Compile, install (before compiling to ensure that the Libfastcommon has been successfully installed)

#./make.sh

#./make.sh Install

Installed by default installation, the corresponding files and directories after installation:

A, service scripts in:

/etc/init.d/fdfs_storaged

/etc/init.d/fdfs_tracker

B, configuration file in (sample configuration file):

/etc/fdfs/client.conf.sample

/etc/fdfs/storage.conf.sample

/etc/fdfs/tracker.conf.sample

C, the command tool in the/usr/bin/directory:

Fdfs_appender_test

Fdfs_appender_test1

Fdfs_append_file

Fdfs_crc32

Fdfs_delete_file

Fdfs_download_file

Fdfs_file_info

Fdfs_monitor

Fdfs_storaged

Fdfs_test

Fdfs_test1

Fdfs_trackerd

Fdfs_upload_appender

Fdfs_upload_file

stop.sh

restart.sh

(4) Because the Fastdfs service script set the bin directory is/usr/local/bin, but the actual command is installed in/usr/bin, you can enter the/user/bin directory to view the FDFS commands using the following command:

# cd/usr/bin/

# ls | grep Fdfs

It is therefore necessary to modify the corresponding command path in the Fastdfs service script, which is to modify the/usr/local/bin in the/etc/init.d/fdfs_storaged and/etc/init.d/fdfs_tracker two scripts to/usr/ Bin

# VI Fdfs_trackerd

To unify the changes using the Find substitution command:%s+/usr/local/bin+/usr/bin

# VI Fdfs_storaged

To unify the changes using the Find substitution command:%s+/usr/local/bin+/usr/bin

Second, configure Fastdfs Tracker (192.168.4.121)

1, copy the Fastdfs Tracker sample configuration file, and rename:

# cd/etc/fdfs/

# CP Tracker.conf.sample tracker.conf

2. Edit Tracker configuration file:

# vi/etc/fdfs/tracker.conf

The revised content is as follows:

Disabled=false

port=22122

Base_path=/fastdfs/tracker

(Other parameters retain the default configuration, please refer to the official documentation for specific configuration explanations:

http://bbs.chinaunix.net/thread-1941456-1-1.html)

3, create the basic data directory (refer to the basic directory Base_path configuration):

# mkdir-p/fastdfs/tracker

4, open the tracker port in the firewall (default is 22122):

# Vi/etc/sysconfig/iptables

Add the following port line:

-A input-m state--state new-m tcp-p TCP--dport 22122-j ACCEPT

Reboot the firewall:

# Service Iptables Restart

5, start Tracker:

#/etc/init.d/fdfs_trackerd Start

(The first successful startup will create data, logs two directories under the/fastdfs/tracker directory) to see if the Fastdfs tracker has started successfully:

# Ps-ef | grep Fdfs

6. Close Tracker:

#/etc/init.d/fdfs_trackerd Stop

7, set the Fastdfs tracker boot:

# vi/etc/rc.d/rc.local

Add the following:

# # Fastdfs Tracker

/etc/init.d/fdfs_trackerd start

Third, configure Fastdfs storage (192.168.4.125)

1, copy FASTDFS Storage sample configuration file, and rename:

# cd/etc/fdfs/

# CP Storage.conf.sample storage.conf

2. Edit the Memory sample configuration file:

# vi/etc/fdfs/storage.conf

The revised content is as follows:

Disabled=false

port=23000

Base_path=/fastdfs/storage

Store_path0=/fastdfs/storage

tracker_server=192.168.4.121:22122

http.server_port=8888

(Other parameters retain the default configuration, please refer to the official documentation for specific configuration explanations:

http://bbs.chinaunix.net/thread-1941456-1-1.html)

3, create the basic data directory (refer to the basic directory Base_path configuration):

# mkdir-p/fastdfs/storage

4, open the memory port in the firewall (default is 23000):

# Vi/etc/sysconfig/iptables

Add the following port line:

-A input-m state--state new-m tcp-p TCP--dport 23000-j ACCEPT

Reboot the firewall:

# Service Iptables Restart

5, start Storage:

#/etc/init.d/fdfs_storaged Start

(First successful startup, create data, logs two directories under the/fastdfs/storage directory) to see if Fastdfs storage started successfully

# Ps-ef | grep Fdfs

6. Close Storage:

#/etc/init.d/fdfs_storaged Stop

7, set up Fastdfs memory boot:

# vi/etc/rc.d/rc.local

Add to:

# # Fastdfs Storage

/etc/init.d/fdfs_storaged start

iv. File Upload Test (192.168.4.121)

1, modify the client configuration file in the Tracker server:

# cp/etc/fdfs/client.conf.sample/etc/fdfs/client.conf

# vi/etc/fdfs/client.conf

Base_path=/fastdfs/tracker

tracker_server=192.168.4.121:22122

2, execute the following upload file command

do not use kill-9 to kill the process, may cause binlog data loss

Client Java Code Test fdfs_client.conf

Connect_timeout =
Network_timeout =
charset = UTF-8
http.tracker_http_port = 8080
http.anti_ Steal_token = no
Http.secret_key = FastDFS1234567890

tracker_server = 192.168.4.121:22122

encapsulated client-side code

Import Java.io.ByteArrayInputStream;
Import Java.io.File;
Import Java.io.FileInputStream;
Import java.io.IOException;

Import Java.io.InputStream;
Import Org.apache.commons.lang3.StringUtils;
Import Org.apache.log4j.Logger;
Import Org.csource.common.NameValuePair;
Import Org.csource.fastdfs.ClientGlobal;
Import Org.csource.fastdfs.StorageClient1;
Import Org.csource.fastdfs.StorageServer;
Import org.csource.fastdfs.TrackerClient;

Import Org.csource.fastdfs.TrackerServer;
 /** * * Description: Fastdfs Distributed File System Operation client.
 * @ Author: Wushuicheng.
 * @ creation time: 2015-3-29, 8:13:49.
 * @ Version number: V1.0. */public class Fastdfsclient {//private static final String conf_filename = Thread.CurrentThread (). getcontextclassload
	ER (). getresource (""). GetPath () + "fdfs_client.conf";
	private static final String Conf_filename = "src/main/resources/fdfs/fdfs_client.conf";

	private static StorageClient1 storageClient1 = null;

	private static Logger Logger = Logger.getlogger (Fastdfsclient.class);
	 /** * Only one load at a time. * * Static {try {logger.info ("= = Conf_filename:" + conf_filename);
			Clientglobal.init (Conf_filename);
			Trackerclient trackerclient = new Trackerclient (clientglobal.g_tracker_group);
			Trackerserver trackerserver = Trackerclient.getconnection ();
			if (Trackerserver = = null) {Logger.error ("getconnection return null");
			} storageserver storageserver = Trackerclient.getstorestorage (trackerserver);
			if (Storageserver = = null) {Logger.error ("getstorestorage return null");
		} storageClient1 = new StorageClient1 (Trackerserver, storageserver);
		catch (Exception e) {logger.error (e); }/** * * @param file * * @param filename * filename * @return return NULL for failure * * Publ
		IC static string uploadfile (file file, String fileName) {FileInputStream FIS = null;
			try {namevaluepair[] meta_list = null;//new namevaluepair[0];
			FIS = new FileInputStream (file);
			byte[] File_buff = null; if (FIS!= null) {int len = fIs.available ();
				File_buff = new Byte[len];
			Fis.read (File_buff);
			String Fileid = Storageclient1.upload_file1 (File_buff, Getfileext (fileName), meta_list);
		return Fileid;
			catch (Exception ex) {logger.error (ex);
		return null;
				}finally{if (FIS!= null) {try {fis.close ();
				catch (IOException e) {logger.error (e); /** * Deletes a file based on the group name and remote file name * * @param groupname * For example, "group1" If you do not specify this value, the default is group1 * @para M FileName * For example "m00/00/00/wkgxgk5hblvfp86raaaachd9x1y736.jpg" * @return 0 for success, non 0 for failure, specifically for error codes/public STA Tic int DeleteFile (string groupname, String fileName) {try {int result = Storageclient1.delete_file (groupname = = Nu ll?
			"Group1": GroupName, FileName);
		return result;
			catch (Exception ex) {logger.error (ex);
		return 0; }/** * According to Fileid to delete a file (we are now in this way, upload files directly to the Fileid in the database) * * @param Fileid * file_id Source explanation fi LE_ID the file ID (includING group name and filename), such as Group1/m00/00/00/ooybafm6mpmahm91aaaegdpirc0012.xml * @return 0 for success, not 0 for failure, specific error code * p
			ublic static int DeleteFile (String fileid) {try {int result = Storageclient1.delete_file1 (Fileid);
		return result;
			catch (Exception ex) {logger.error (ex);
		return 0; }/** * Modify an existing file * * @param Oldfileid * Original Fileid of old files, file_id source code explanation file_id the file ID (includi NG group name and filename), such as Group1/m00/00/00/ooybafm6mpmahm91aaaegdpirc0012.xml * @param file * New files * @param filePath * New file path * @return return NULL is failed/public static string Modifyfile (string oldfileid, File file
		, string filePath) {string fileid = null;
			try {//upload first Fileid = uploadfile (file, FilePath);
			if (Fileid = = null) {return null;
			}//Then remove int delresult = DeleteFile (Oldfileid);
			if (delresult!= 0) {return null;
			} catch (Exception ex) {logger.error (ex);
		return null; } return Fileid;
		/** * File Download * * @param fileid * @return return a stream */public static InputStream DownloadFile (String fileid) {
			try {byte[] bytes = Storageclient1.download_file1 (Fileid);
			InputStream InputStream = new Bytearrayinputstream (bytes);
		return inputstream;
			catch (Exception ex) {logger.error (ex);
		return null;
	 }/** * Gets the file suffix name (without a dot).
	 * * @return such as: "JPG" or "". * * private static string Getfileext (String fileName) {if stringutils.isblank (filename) | |!filename.contains ("."))
		{return ""; else {return filename.substring (Filename.lastindexof (".") + 1);//No Last point}}

Test code

Import Java.io.File;
Import Java.io.InputStream;

Import Org.apache.commons.io.FileUtils;

Import wusc.edu.demo.fdfs.FastDFSClient;
Import Org.junit.Test;

/**
 * 
 * Description: Fastdfs test.
 * @ Author: Wushuicheng.
 * @ creation Time: 2015-3-29, afternoon 8:11:36.
 * @ Version number: V1.0.
* * public class Fastdfstest {
	
	/**
	 * upload test.
	 * @throws Exception * *
	 

	@Test

public void Upload () throws Exception {String FilePath = "E:/workspacespr10.6/edu-demo-fdfs/testfile/dubbovideo.jpg"; File File = new file (FilePath); String Fileid = fastdfsclient.uploadfile (file, FilePath); System.out.println ("Upload local file" + FilePath + "OK, fileid=" + Fileid);/fileid:group1/m00/00/00/wkgefvuypiead6a0 aap3btxj__e335.jpg//url:http://192.168.4.125:8888/group1/m00/00/00/wkgefvuypiead6a0aap3btxj__e335.jpg}/** * Download Test . * @throws Exception * *

        @Test

public static void Download () throws Exception {String Fileid = "Group1/m00/00/00/wkgefvuypiead6a0aap3btxj__e335.jpg"; Nputstream InputStream = Fastdfsclient.downloadfile (Fileid); File DestFile = new file ("E:/workspacespr10.6/edu-demo-fdfs/testfile/downloadtest.jpg"); Fileutils.copyinputstreamtofile (InputStream, destfile);} /** * Delete Test * @throws Exception * *

	@Test

public static void Delete () throws Exception {String Fileid = "group1/m00/00/00/wkgefvuypiead6a0aap3btxj__e335.jpg"; int result = Fastdfsclient.deletefile (Fileid); System.out.println (Result = = 0?) "Delete succeeded": "Delete failed:" + result);} /** * @param args * @throws Exception * *

	@Test

public static void Main (string[] args) throws Exception {//upload ();//download ();d elete ();}