Java EE Advanced--FASTDFS Implementation of Distributed File system

Fastdfs

Fastdfs is an open source Distributed file system written in C language. Fastdfs for the Internet, fully consider the redundant backup, load balancing, linear expansion and other mechanisms, and pay attention to high availability, high performance and other indicators, the use of Fastdfs easily set up a high-performance file server cluster to provide file upload, download and other services.

The FASTDFS server has two roles: Tracker (tracker) and storage node (storage). The tracker mainly does the dispatching work, and the function of load balancing on the access.
Storage node storage file, complete all the functions of file management: This is the storage, synchronization and the provision of access interface, Fastdfs at the same time the file metadata management. The so-called meta data of a file is the relevant property of the file, expressed as a key value pair (key), such as: width=1024, where the key is width,value to 1024. File metadata is a list of file attributes that can contain multiple key-value pairs.
Both trackers and storage nodes can be composed of one or more servers. Servers in trackers and storage nodes can be added or offline at any time without impacting online services. All servers in the tracker are peers, and can be increased or decreased at any time depending on the pressure of the server.
To support large capacity, storage nodes (servers) are organized in a sub-volume (or grouping) manner. The storage system consists of one or more volumes, and the files between the volumes and volumes are independent of each other, and the cumulative file capacity of all volumes is the file capacity of the entire storage system. A volume can consist of one or more storage servers, and the files in a storage server under a volume are the same, and multiple storage servers in the volume play a role of redundant backup and load balancing.
When adding servers to a volume, synchronizing existing files is done automatically by the system, and after synchronization is complete, the system automatically switches the new server to the online service.
You can add volumes dynamically when storage space is low or is about to be exhausted. You only need to add one or more servers and configure them as a new volume, which increases the capacity of the storage system.
The file ID in Fastdfs is divided into two parts: volume name and file name, both of which are indispensable.

Upload Interactive Process

1. Client asked tracker upload to the storage, no additional parameters required;
2. Tracker returns a usable storage;
3. Client directly and storage communication to complete the file upload.

Download Interactive Process

Build Environment File Download list

1. Fastdfs_v5.05.tar.gz
2. Nginx-1.14.0.tar.gz
3. Fastdfs-nginx-module_v1.16.tar.gz
4. Fastdfs_client_v1.20.jar
5. Libfastcommonv1.0.7.tar.gz

Build FASTDFS Environment 1, upload all downloaded compressed source package

Select the CentOS6.4 system, the specific installation see Java EE Advanced--centos Development Environment construction

2. Before installing Fastdfs, install the libevent kit first

yum -y install libevent

• 1

3, install Libfastcommon kit.

tar -zxvf libfastcommonV1.0.7.tar.gzcd libfastcommon1.0.7 ./make.sh./make.sh installcp /usr/lib64/libfastcommon.so /usr/lib/ll |grep libfast* # 查看lib下是否存在libfastcommon.so

• 1
• 2
• 3
• 4
• 5
• 6

4. Installation Tracker Service

tar -zxvf FastDFS_v5.05.tar.gzcd FastDFS./make.sh./make.sh install  cd /usr/bin/ll fdfs_*  # 查看编译结果是否产生fdfs相关文件cp FastDFS/conf/* /etc/fdfs/  # 拷贝配置文件到/etc/fdfs/目录下

• 1
• 2
• 3
• 4
• 5
• 6
• 7

To modify a configuration file:

vim /etc/fdfs/tracker.conf

• 1

Start Tracker:

# 启动Tracker服务/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart # 重启Tracker服务

• 1
• 2

5. Installation Storage Service

If you are installing on a different server, you need to compile the fastdfs_v5.05.tar.gz source package in a new machine, and you do not need to configure the tracker service.
To modify a configuration file:

vim /etc/fdfs/storage.conf

• 1

To start the storage service:

# 启动Storage服务/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart # 重启Storage服务

• 1
• 2

Test Service 1, modify the configuration file/etc/fdfs/client.conf

2. Start test

/usr/bin/fdfs_test /etc/fdfs/client.conf upload anti-steal.jpg

• 1

Build Nginx to provide HTTP service

2. Unzip the plugin compression pack fastdfs-nginx-module_v1.16.tar.gz
4. Modify the/root/fastdfs-nginx-module/src/config file to remove the local.
   

6. For Nginx re-config, add Fastdfs-nginx-module:

   # 指定自己的路径makemake install

1 Copy the/root/fastdfs-nginx-module/src/mod_fastdfs.conf file to the/etc/fdfs directory for editing:
8. 
   
   
   

10. Configuration of Nginx

    server {        listen       80;        server_name  192.168.74.129;        location /group1/M00/{
    

12. Copy the libfdfsclient.so to the/usr/lib:

    cp /usr/lib64/libfdfsclient.so www.mcyllpt.com /usr/lib/

    • 1
14. Start Nginx.
    

Test if the HTTP service is successful

1. Upload Image:

   /usr/bin/fdfs_test /etc/fdfs/client.conf upload www.078881.cn /root/anti-steal.jpg

   • 1

2. Access command line output URL: http://192.168.74.129/group1/M00/00/00/wKhKgVslH5GAKcLdAAB1_3EXRGc833_big.png
   

3. If not, check whether the 22122 and 23000 port firewalls are down or temporarily shut down the firewall:

   service iptables stop  # 临时关闭防火墙

   • 1

Java uses Fastdfs

The official supply of a jar package: Fastdfs_client_v1.20.jar. If you use Maven management, you can add:

< !--Https://mvnrepository.com/artifact/net.oschina.zcx7878/fastdfs-client-java-->< dependency> <groupId> Net.oschina.zcx7878</groupid> < artifactid>fastdfs-client-java</ artifactid> <version>1.27.0.0 </version></ DEPENDENCY>            

• 1
• 2
• 3
• 4
• 5
• 6

How to use:
1. Add the jar package provided by Fastdfs to the project
2. Initialize the global configuration. Loads a configuration file.
3. Create a Trackerclient object.
4. Obtain a Trackerserver object through Trackerclient.
5. Declare a Storageserver object, null.
6. Obtain a Storageclient object from the Trackerserver object and the Storageserver object.
7. Directly call the Storageclient object method to upload the file.
Create configuration file client.conf:

tracker_server=192.168.74.129:22122

• 1

To test the Java code:

PublicClassfastdfstest {@Testpublic void testUpload () throws IOException, MyException {clientglobal.init ( "E:\\ Intelljidea\\taotao\\taotao-manager\\taotao-manager-web\\src\\main\\resources\\properties\\client.conf "); Trackerclient trackerclient = new trackerclient (www.leyouzaixian2.com); Trackerserver trackerserver = Trackerclient.getconnection (); Storageserver storageserver = null; Storageclient storageclient = new storageclient (Trackerserver, storageServer); String[] strings = Storageclient.upload_file ( "c:\\users\\os\\pictures\\ very sister. jpg",  "jpg", null); for (String string:strings) {System.out.println (string);     

Test results:

Fastdfs Tool Class: Fastdfsclient.java

Import Org.csource.common.NameValuePair;Import Org.csource.fastdfs.ClientGlobal;Import Org.csource.fastdfs.StorageClient1;Import Org.csource.fastdfs.StorageServer;Import org.csource.fastdfs.TrackerClient;Import Org.csource.fastdfs.TrackerServer;PublicClassfastdfsclient {Private Trackerclient trackerclient =NullPrivate Trackerserver Trackerserver =NullPrivate Storageserver Storageserver =NullPrivate StorageClient1 storageclient =NullPublicFastdfsclient (String conf)Throws Exception {if (Conf.contains ("Classpath:")) {conf = Conf.replace ("Classpath:",This.getclass (). GetResource ("/"). GetPath ()); } clientglobal.init (conf); Trackerclient =New Trackerclient (); Trackerserver = Trackerclient.getconnection (); Storageserver =Null Storageclient =New StorageClient1 (www.leyouzaixian2.com trackerserver, storageserver); }/** * Upload File Method * <p>Title:uploadFile</p> * <p>description: </p> *@param filename File Full path *@param extname file extension, not including (.) *@param metas File extension information *@return *@throws Exception * *Public StringUploadFile (String fileName, String extname, namevaluepair[] metas)Throws Exception {String result = Storageclient.upload_file1 (FileName, Extname, Metas);return result; }Public StringUploadFile (String fileName)Throws Exception {Return UploadFile (FileName,NullNULL); }Public StringUploadFile (String fileName, String extname)Throws Exception {Return UploadFile (FileName, Extname,NULL); }/** * Upload File Method * <p>Title:uploadFile</p> * <p>description: </p> *@param the contents of the Filecontent file, byte array *@param extname File extension *@param metas File extension information *@return *@throws Exception * *Public Stringuploadfile (byte[] filecontent, String extname, NameValuePair[] Metas) throws Exception {String result = Storageclient.upload_file1 (Filecontent, Extname, Metas); return result; public String uploadfile (byte[] filecontent) throws Exception {return uploadfile (fileContent , null, null);} public String uploadfile (byte[] Filecontent, String extname) throws Exception {return UploadFile (Filecontent, Extname, null);}        

Java EE Advanced--fastdfs implement Distributed File system