Cluster deployment of FASTDFS

Fastdfs is an open source lightweight Distributed File system consisting of a tracking server (tracker server), a storage server (storage server), and clients (client), which mainly solves the problem of massive data storage, Especially suitable for medium and small documents (recommended range: 4KB < file_size <500MB) for the carrier's online service. In the generation environment Fastdfs is generally configured with a cluster to improve FASTDFS availability and concurrency capabilities.

Deployment Schema:

Environment IP address (firewall that shuts down all environments):

Tracker 192.168.18.178

Group 1:

s1:192.168.110.71

s2:192.168.110.91

Group 2:

s3:192.168.100.90

s4:192.168.100.194

Note: Tracker can deploy multiple units, provide the load, where resources are limited, deploy one.

Because of the need to install Nginx, each machine is installed with dependencies:

Yum-y install zlib pcre pcre-devel zlib-devel

Download the installation software used: http://download.csdn.net/detail/tianwei7518/8745279

I. Installation of tracker

1. Installation Dependent Libfastcommon

Unzip Libfastcommon-master.zip

CD Libfastcommon

./make.sh
./make.sh Install

2. Install Fastdfs

Unzip Fastdfs.zip

CD Fastdfs

./make.sh

./make.sh Install

Default installation directory:/usr/bin

Copy the configuration files under the original installation folder to/ETC/FDFS: CP./conf/*/etc/fdfs/

3. Configure

Edit the tracker.conf in the configuration file directory
Generally only need to change the following several parameters can be:
Disabled=false #启用配置文件
port=22122 #设置tracker的端口号
Base_path=/home/fastdfs #设置tracker的数据文件和日志目录 (need to be created in advance)
http.server_port=8080 #设置http端口号
4. Start
/usr/bin/fdfs_trackerd/etc/fdfs/tracker.conf start

second, the installation tracker agent Nginx

The Nginx installed on tracker is intended primarily to provide a reverse proxy for HTTP access, load balancing, and caching services.

1. Install Nginx

TAR-ZXVF nginx-1.8.0.tar.gz
TAR-ZXVF ngx_cache_purge-2.3.tar.gz
CD nginx-1.8.0
./configure--prefix=/usr/local/nginx--add-module=/root/ngx_cache_purge-2.3

Make
Make install

If you are prompted for an error, you may be missing a dependent package, and you will need to install the dependency pack before running again./configure
Nginx and Nginx cache purge plug-in module installation completed, installation directory/usr/local/nginx

2. Configure Nginx

User root;

Worker_processes 1;
#error_log Logs/error.log;
#error_log Logs/error.log Notice;

Error_log Logs/error.log Info;


#pid Logs/nginx.pid;


Events {worker_connections 1024;}
    HTTP {include mime.types;
    Default_type Application/octet-stream;
    #设置缓存参数 server_names_hash_bucket_size 128;
    Client_header_buffer_size 32k;
    Large_client_header_buffers 4 32k;
    Client_max_body_size 300m;
    Sendfile on;
    Tcp_nopush on;
    Proxy_redirect off;
    Proxy_set_header Host $http _host;
    Proxy_set_header X-real-ip $remote _addr;
    Proxy_set_header x-forwarded-for $proxy _add_x_forwarded_for;
    Proxy_connect_timeout 90;
    Proxy_send_timeout 90;
    Proxy_read_timeout 90;
    Proxy_buffer_size 16k;
    Proxy_buffers 4 64k;
    Proxy_busy_buffers_size 128k;
    Proxy_temp_file_write_size 128k; #设置缓存存储路径, storage mode, allocated memory size, disk maximum space, cache duration Proxy_cache_path/var/cache/nginx/proxy_cache Levels=1:2 Keys_zone=http-cache : 500m Max_size=10g inactive=30d;

    proxy_temp_path/var/cache/nginx/proxy_cache/tmp;

    Keepalive_timeout 65;
        #设置group服务器 upstream Fdfs_group1 {server 192.168.110.71:8090 weight=1 max_fails=2 fail_timeout=30s;
    Server 192.168.110.91:8090 weight=1 max_fails=2 fail_timeout=30s;
        } upstream fdfs_group2 {server 192.168.100.90:8090 weight=1 max_fails=2 fail_timeout=30s;
    Server 192.168.100.194:8090 weight=1 max_fails=2 fail_timeout=30s;
        } server {Listen 80;
        server_name localhost;
        CharSet Utf-8;

        #access_log/usr/local/nginx/logs/localhost.access.log Main;
                location/group1/m00 {proxy_next_upstream http_502 http_504 error timeout invalid_header;
                Proxy_cache Http-cache;
                Proxy_cache_valid 304 12h;
                Proxy_cache_key $uri $is_args$args;
                Proxy_pass Http://fdfs_group1;
        Expires 30d;
       } location/group2/m00 {proxy_next_upstream http_502 http_504 error timeout invalid_header;
                Proxy_cache Http-cache;
                Proxy_cache_valid 304 12h;
                Proxy_cache_key $uri $is_args$args;
                Proxy_pass http://fdfs_group2;
        Expires 30d;
                } #设置清除缓存的访问权限 Location ~/purge (/.*) {allow 127.0.0.1;
                Allow 172.16.1.0/24;
                Deny all;
        Proxy_cache_purge Http-cache $1$is_args$args; }
    }

}

Create cache directory:/var/cache/nginx/proxy_cache/tmp 3. Start

/usr/local/nginx/sbin/nginx

III. Installation of storage
1. Installation

Refer to the first 2 steps of installing Tracker.

2. Configure

Edit the storage.conf in the configuration file directory

Simply change the following several parameters:

Disabled=false #启用配置文件

Group_name=group1 #组名, modified according to the actual situation

port=23000 #设置storage的端口号

Base_path=/home/fastdfs #设置storage的日志目录 (need to be created in advance)

Store_path_count=1 #存储路径个数, need to match the number of Store_path

store_path0=/home/fastdfs# Storage Path

Tracker_server=192.168.17.43:22122#tracker the IP address and port number of the server http.server_port=8080 #设置http端口号

Create a directory Mkdir/home/fastdfs

3. Run

/usr/bin/fdfs_storaged/etc/fdfs/storage.conf start

In addition:

Install storage on the other machines and verify that they are working properly. Note that the group name parameter in the configuration file needs to be adjusted to the actual situation:
group1:192.168.110.71,192.168.110.91
group2:192.168.100.90,192.168.100.194
In addition, all storage in each group must have the same port number.

Iv. installation of Nginx on the storage
The Nginx installed on storage is primarily designed to provide HTTP access services while resolving synchronization latency problems for storage servers in the group.
1. Extract Fastdfs-nginx-module Plugin
Unzip Fastdfs-nginx-module.zip
2. Install Nginx
TAR-ZXVF nginx-1.8.0.tar.gz
CD nginx-1.8.0
./configure--prefix=/usr/local/nginx--add-module=. /fastdfs-nginx-module/src
Make
Make install
Installation directory:/usr/local/nginx

If install error: [Emerg] 13513#0:eventfd () failed (38:function not implemented)
The reason is: the compile-time with the--with-file-aio module, the Linux 2.6.22 after the kernel to support. The server is 2.6.18. You can also download a lower version of the Nginx version
3. Configure
1) Configure the Fastdfs Nginx plugin
Copy the configuration file of the Fastdfs Nginx plug-in module to the Fastdfs configuration file directory
fastdfs-nginx-module/src/mod_fastdfs.conf/etc/fdfs/
Edit the mod_fastdfs.conf in the/etc/fdfs configuration file directory to set storage information and save it.
Generally only need to change the following several parameters can be:
Base_path=/home/fastdfs #保存日志目录
tracker_server=192.168.18.43:22122 #tracker服务器的IP地址以及端口号
Storage_server_port=23000#storage the port number of the server
group_name=group1# the current server's group name
Url_have_group_name = True #文件url中是否有group名
Store_path_count=1 #存储路径个数, need to match the number of Store_path
Store_path0=/home/fastdfs #存储路径
http.need_find_content_type=true# finds a file type from a file name extension (True when nginx)
Group_count = 2 #设置组的个数
Add specific information for 2 groups at the end:
[Group1]
Group_name=group1
storage_server_port=23000
Store_path_count=1
Store_path0=/home/fastdfs

[Group2]
Group_name=group2
storage_server_port=23000
Store_path_count=1
Store_path0=/home/fastdfs
Establish a symbolic connection M00 to the storage directory:
Ln-s/home/fastdfs/data/home/fastdfs/data/m00
2) Configure Nginx
VI nginx.conf
User root;


Location ~/GROUP[1-3]/M00 {
Root/home/fastdfs/data;
Ngx_fastdfs_module;
}
4. Start
/usr/local/nginx/sbin/nginx

Other than that:
Install Nginx on the other machine storage and verify that it is working properly. Note that the group name parameter in the configuration file needs to be adjusted to the actual situation:
group1:192.168.110.71,192.168.110.91
group2:192.168.100.90,192.168.100.194
In addition the port number of the Nginx 8090.

All configurations have been completed thus far.

Four, test

Configure/etc/fdfs/client.conf

Base_path=/home/fastdfs #日志存放路径

tracker_server=192.168.18.43:22122 #tracker服务器IP地址和端口号

http.tracker_server_port=8080 #tracker服务器的http端口号

Upload a file through Fdfs_upload_file to Fastdfs, the program will automatically return the file URL,
#fdfs_upload_file/etc/fdfs/client.conf 40-15052pzk5.jpg
Group1/m00/00/00/wkhur1vmh_2admdfaaf1dmvtk4w934.jpg
Then use browser access to access the normal
Http://192.168.18.43/group1/M00/00/00/wKhuR1Vmh_2ADmdfAAF1dmVtk4w934.jpg

Note: You can use Fdfs_monitor to view the operation of tracker and all group

# fdfs_monitor/etc/fdfs/client.conf [2015-05-27 20:19:59] Debug-base_path=/home/fastdfs, connect_timeout=30, network _timeout=60, Tracker_server_count=1, anti_steal_token=0, Anti_steal_secret_key length=0, use_connection_pool=0, g_ connection_pool_max_idle_time=3600s, use_storage_id=0, storage server ID count:0 server_count=1, server_index=0 Tracke R Server is 192.168.18.43:22122 group count:2 group 1:group name = group1 disk Total spaces = 45438 MB disk free spaces = 33920 MB trunk free spaces = 0 MB Storage Server Count = 2 Active Server count = 2 Storage Server port = 23000 Storage HT

        TP port = 8080 Store Path count = 1 subdir count per path = 256-Write Server index = 1 current trunk file ID = 0
                Storage 1:id = 192.168.110.71 ip_addr = 192.168.110.71 (localhost) ACTIVE  HTTP domain = Version = 5.06 join time = 2015-05-27 01:37:04 up Time = Total StorAge = 95217 MB free storage = 47563 MB Upload priority = Ten Store_path_cou NT = 1 Subdir_count_per_path = 256 Storage_port = 23000 Storage_http_port
                = 8080 Current_write_path = 0 Source Storage id = if_trunk_server = 0 Connection.alloc_count = 256 Connection.current_count = 1 Connection.max_coun  t = 2 Total_upload_count = 1 Success_upload_count = 1 Total_append_count = 0 Success_append_count = 0 Total_modify_count = 0 Success_modify_count =  0 Total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 Success_set_meta_count = 0 total_delete_count = 0 Success_delete_coun t = 0 ToTal_download_count = 0 Success_download_count = 0 Total_get_meta_count = 0
                Success_get_meta_count = 0 Total_create_link_count = 0 Success_create_link_count = 0  Total_delete_link_count = 0 Success_delete_link_count = 0 total_upload_bytes = 95606 success_upload_bytes = 95606 total_append_bytes = 0 Success_append_ bytes = 0 Total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_ bytes = 0 Success_download_bytes = 0 total_sync_in_bytes = 0 success_sync_ In_bytes = 0 Total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_fil E_open_count = 1 Success_file_open_count = 1 Total_file_read_count = 0 suc Cess_file_read_count = 0
                Total_file_write_count = 1 Success_file_write_count = 1 last_heart_beat_t IME = 2015-05-27 20:19:49 last_source_update = 2015-05-27 20:14:04 last_sync_update = 1969 -12-31 16:00:00 last_synced_timestamp = 1969-12-31 16:00:00 Storage = 192.1 68.110.91 ip_addr = 192.168.110.91 (localhost) ACTIVE http domain = vers Ion = 5.06 Join time = 2015-05-27 19:35:36 up time = 2015-05-27 19:35:36 t Otal storage = 45438 MB free storage = 33920 MB Upload priority = Ten Store _path_count = 1 Subdir_count_per_path = 256 Storage_port = 23000 Storage_h
                Ttp_port = 8080 Current_write_path = 0 Source Storage id = 192.168.110.71
    If_trunk_server = 0            Connection.alloc_count = 256 Connection.current_count = 1 Connection.max_count = 1 Total_upload_count = 0 Success_upload_count = 0 Total_append_count = 
                0 Success_append_count = 0 Total_modify_count = 0 Success_modify_count = 0 Total_truncate_count = 0 Success_truncate_count = 0 total_set_meta_count  = 0 Success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 Total_download_count = 0 success_download_count = 0 total_get_meta_co UNT = 0 Success_get_meta_count = 0 total_create_link_count = 0 success_cre
                Ate_link_count = 0 Total_delete_link_count = 0 Success_delete_link_count = 0 Total_upload_bytes =0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 
                Total_modify_bytes = 0 Success_modify_bytes = 0 stotal_download_bytes = 0 Success_download_bytes = 0 Total_sync_in_bytes = 95606 success_sync_in_by TES = 95606 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file _open_count = 1 Success_file_open_count = 1 Total_file_read_count = 0 succ
                Ess_file_read_count = 0 Total_file_write_count = 1 success_file_write_count = 1
                Last_heart_beat_time = 2015-05-27 20:19:51 last_source_update = 1969-12-31 16:00:00 Last_sync_update = 2015-05-27 20:14:07 last_synced_timestamp = 2015-05-27 20:14:05 ( -1s delay) Group 2:g Roup name = group2 DiskTotal spaces = 9916 MB disk free spaces = 7434 MB trunk free spaces = 0 MB Storage Server Count = 2 Active Server count = 2 s Torage Server port = 23000 Storage HTTP port = 8080 Store Path count = 1 subdir count per path = 256 Current write server Index = 0 current trunk File ID = 0 Storage 1:id = 192.168.100.194 ip_addr = 192  .168.100.194 (localhost) ACTIVE http domain = Version = 5.06 Join time =
                2015-05-27 20:03:37 up/2015-05-27 20:03:37 Total storage = 47368 MB Free storage = 37371 MB Upload priority = Ten Store_path_count = 1 subdir _count_per_path = 256 Storage_port = 23000 Storage_http_port = 8080 Curren T_write_path = 0 Source Storage id = 192.168.100.90 If_trunk_server = 0 Co Nnection.alloc_count = 2Connection.current_count = 1 Connection.max_count = 1 Total_upload_coun  t = 0 Success_upload_count = 0 total_append_count = 0 success_append_count = 0 Total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 Success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_c Ount = 0 Total_delete_count = 0 success_delete_count = 0 total_download_co UNT = 0 Success_download_count = 0 total_get_meta_count = 0 Success_get_me Ta_count = 0 Total_create_link_count = 0 success_create_link_count = 0 tot
                Al_delete_link_count = 0 Success_delete_link_count = 0 total_upload_bytes = 0 Success_upload_bytes = 0 Total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_byte s = 0 Total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_byt Es = 0 success_sync_out_bytes = 0 total_file_open_count = 0 success_file_o Pen_count = 0 Total_file_read_count = 0 success_file_read_count = 0 Total_ File_write_count = 0 Success_file_write_count = 0 last_heart_beat_time = 2015-05-27 20:19:
                Last_source_update = 1969-12-31 16:00:00 last_sync_update = 1969-12-31 16:00:00
                Last_synced_timestamp = 1969-12-31 16:00:00 Storage 2:id = 192.168.100.90 IP_ADDR = 192.168.100. localhost ACTIVE http domain = Version = 5.06 Join time = 2015-05- 19:50:27 up/2015-05-27 19:50:27 Total storage = 9916 MB free sto Rage = 7434 MB Upload priority = Ten Store_path_count = 1 subdir_count_per_ Path = 256 Storage_port = 23000 Storage_http_port = 8080 Current_write_pat H = 0 Source Storage id = If_trunk_server = 0 Connection.alloc_count = 25  6 Connection.current_count = 1 Connection.max_count = 1 Total_upload_count = 0 Success_upload_count = 0 total_append_count = 0 success_append_count  = 0 Total_modify_count = 0 Success_modify_count = 0 Total_truncate_count = 0 SucceSs_truncate_count = 0 Total_set_meta_count = 0 Success_set_meta_count = 0 Total_delete_count = 0 Success_delete_count = 0 Total_download_count = 0 S
                Uccess_download_count = 0 Total_get_meta_count = 0 Success_get_meta_count = 0
                Total_create_link_count = 0 Success_create_link_count = 0 Total_delete_link_count = 0 Success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes 
                = 0 Total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0  Success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 Total_sync_in_bytes = 0 success_sync_in_bytes = 0 total_sync_out_byte s = 0 suCcess_sync_out_bytes = 0 Total_file_open_count = 0 Success_file_open_count = 0
                Total_file_read_count = 0 Success_file_read_count = 0 Total_file_write_count = 0 Success_file_write_count = 0 Last_heart_beat_time = 2015-05-27 20:19:48 last_s Ource_update = 1969-12-31 16:00:00 last_sync_update = 1969-12-31 16:00:00 last_synced_time  Stamp = 1969-12-31 16:00:00

Reference:

1.fastdfs 5.01 + nginx + cache cluster Installation configuration manual
2. Distributed File system Fastdfs architecture analysis and configuration optimization
3.FastDFS Installation Required Package