Fastdfs combined with FASTDHT for File de-re-storage

Source: Internet
Author: User

When storing files, in order to save storage space, you need to implement file de-weight, that is, the same file only on the server to store one copy. One implementation is to upload the file to the application server, calculate the MD5 and store it in the database, and then decide whether to upload the storage server. The disadvantage of this is that the application server side needs to do concurrency control, the implementation is relatively more complex.

Fastdfs itself supports file-handling mechanisms, but requires FASTDHT to be stored as an index of file hashes. FASTDHT is an open source Key-value database Fastdfs the same author. The principle of its discharge is:

Fastdfs Storage Server Each upload calculates the hash value of the file, and then from the FASTDHT server to find the comparison, if not returned, then write the hash, and save the file, if there is a return, then create a new file link (soft chain), do not save the file.

Construction of experimental Environment:

1. Use VMware Player to create a new virtual machine and install the CentOS system.

I'm using an image of Tohoku University, get the file for Centos-6.8-i386-livecd.iso

2, Installation Fastdfs

Online installation information is very much, recommended reference HTTPS://MY.OSCHINA.NET/HARLANBLOG/BLOG/466487?FROMERR=CQE6BTU2, we mainly want to test the function of FASTDHT, so complete to the 5th step.

3, Installation FASTDHT

Refer to the installation of FASTDHT in

Problems encountered in the construction process:

1, after installing FASTDHT, start Fastdfs storage error Fdfs_storaged:symbol lookup error:fdfs_storaged:undefined symbol:g_current_time

Workaround: Reinstall the Libfastcommon (refer to the second step of installing Fastdfs above)

2, the test program on the host of the virtual machine, that is, the host program needs to access the virtual machine Fastdfs service, set up the environment to find the virtual machine can be online, can ping detox host.

But the host's program accesses the virtual machine's Fastdfs service times wrong connection time out

Workaround: Install the Telnet service and turn off the firewall. At this point, the Fastdfs service is started, assuming that the virtual machine IP is,FASTDFS service port 22122, the command window on the host (Windows system)

A successful connection to Telnet 22122 indicates that the FASTDFS service can be accessed normally.


1, the same file divided into two times successively uploaded (serial)

2, the same file upload simultaneously (parallel, three threads same file upload simultaneously)


After testing the integrated FASTDHT, Fastdfs can implement file de-weight, so that the FASTDFS client does not need to do additional concurrency control, can reduce a large part of the workload. But the current Fastdfs function is tied with FASTDHT, temporarily do not support other K-v library, and fastdht online information is less, if hold its source (C language Implementation), use up or there is a great risk.


Fastdfs combined with FASTDHT for File de-re-storage

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.