Data Synchronization solution (with Java source code), data synchronization with java source code

Data Synchronization solution (with Java source code), data synchronization with java source code

I. background
Data Synchronization is often encountered, and the system's built-in replication function cannot achieve incremental synchronization. Full replication is required each time. In case of an exception, you can only re-start the operation, which is very troublesome, it is time-consuming to process large files.
Ii. Solutions
1. Calculate the data fingerprint of the source directory
2. Calculate the data fingerprint of the target directory
3. Compare the fingerprint data, find out the differences, get the list of files to be added, deleted, or updated, and calculate the data size to be updated.
4. One-to-one synchronization difference. If a large file is encountered, the fingerprint data will be cached in the target folder for the next data synchronization.
Iii. Data fingerprint description
A Data fingerprint is a unique identifier of a folder or file. Its format is as follows:
File relative path + (separator) + modification date +: + Data Length +: + content fingerprint
The content fingerprint is composed of multiple Content blocks md5.
Content blocks are used to separate and process large files. Each time you compare data, the smallest synchronization object is the content block, which avoids processing the entire file and is also a key point for incremental synchronization.
4. Download source code
Source code: https://github.com/xxonehjh/file-sync
Executable File: fileSync. jar
Usage: target directory of the java-jar fileSync. jar source directory