Fastdfs combined with FASTDHT for File de-re-storage

When storing files, in order to save storage space, you need to implement file de-weight, that is, the same file only on the server to store one copy. One implementation is to upload the file to the application server, calculate the MD5 and store it in the database, and then decide whether to upload the storage server. The disadvantage of this is that the application server side needs to do concurrency control, the implementation is relatively more complex.

Fastdfs itself supports file-handling mechanisms, but requires FASTDHT to be stored as an index of file hashes. FASTDHT is an open source Key-value database Fastdfs the same author. The principle of its discharge is:

Fastdfs Storage Server Each upload calculates the hash value of the file, and then from the FASTDHT server to find the comparison, if not returned, then write the hash, and save the file, if there is a return, then create a new file link (soft chain), do not save the file.

Construction of experimental Environment:

1. Use VMware Player to create a new virtual machine and install the CentOS system.

I'm using an image of Tohoku University http://mirror.neu.edu.cn/centos/6.8/isos/i386/, get the file for Centos-6.8-i386-livecd.iso

2, Installation Fastdfs

Online installation information is very much, recommended reference HTTPS://MY.OSCHINA.NET/HARLANBLOG/BLOG/466487?FROMERR=CQE6BTU2, we mainly want to test the function of FASTDHT, so complete to the 5th step.

3, Installation FASTDHT

Refer to the installation of FASTDHT in https://my.oschina.net/u/999023/blog/796238

Problems encountered in the construction process:

1, after installing FASTDHT, start Fastdfs storage error Fdfs_storaged:symbol lookup error:fdfs_storaged:undefined symbol:g_current_time

Workaround: Reinstall the Libfastcommon (refer to the second step of installing Fastdfs above)

2, the test program on the host of the virtual machine, that is, the host program needs to access the virtual machine Fastdfs service, set up the environment to find the virtual machine can be online, can ping detox host.

But the host's program accesses the virtual machine's Fastdfs service times wrong connection time out

Workaround: Install the Telnet service and turn off the firewall. At this point, the Fastdfs service is started, assuming that the virtual machine IP is 192.168.0.1,FASTDFS service port 22122, the command window on the host (Windows system)

A successful connection to Telnet 192.168.0.1 22122 indicates that the FASTDFS service can be accessed normally.

Test:

1, the same file divided into two times successively uploaded (serial)

2, the same file upload simultaneously (parallel, three threads same file upload simultaneously)

Conclusion:

After testing the integrated FASTDHT, Fastdfs can implement file de-weight, so that the FASTDFS client does not need to do additional concurrency control, can reduce a large part of the workload. But the current Fastdfs function is tied with FASTDHT, temporarily do not support other K-v library, and fastdht online information is less, if hold its source (C language Implementation), use up or there is a great risk.

Https://my.oschina.net/u/999023/blog/796287#comment-list

Fastdfs combined with FASTDHT for File de-re-storage