Perl multithreaded rsync backup file to remote host

Demand:

The host has hundreds of backup files to Rsync to the remote host, we will cut the large file for dozens of small files for multi-threaded transmission.

Here is a demo using 14 1G files:

[[email protected] test] # pwd
/ root / test
[[email protected] test] # ll
Total dosage 13631540
-rw-r--r--. 1 root root 1073741824 June 11 18:29 test10.data
-rw-r--r--. 1 root root 1073741824 June 11 18:30 test11.data
-rw-r--r--. 1 root root 1073741824 June 11 18:31 test12.data
-rw-r--r--. 1 root root 1073741824 June 11 18:31 test13.data
-rw-r--r--. 1 root root 1073741824 June 11 18:32 test14.data
-rw-r--r--. 1 root root 1073741824 June 11 18:23 test2.data
-rw-r--r--. 1 root root 1073741824 June 11 18:24 test3.data
-rw-r--r--. 1 root root 1073741824 June 11 18:25 test4.data
-rw-r--r--. 1 root root 1073741824 June 11 18:26 test5.data
-rw-r--r--. 1 root root 1073741824 June 11 18:26 test6.data
-rw-r--r--. 1 root root 1073741824 June 11 18:27 test7.data
-rw-r--r--. 1 root root 1073741824 June 11 18:28 test8.data
-rw-r--r--. 1 root root 1073741824 June 11 18:29 test9.data
[[email protected] test] #

Script Name: tq.pl

#! / usr / bin / env perl

use strict;

use threads;

use Thread :: Queue;

use File :: Find;

use File :: Rsync;

use POSIX qw (strftime);

#Localhost file directory

my $ srcFilePath = ‘/ root / test /’;

#Using the queue, insert the files to be backed up into the queue one by one

my $ fileQueue = Thread :: Queue-> new ();

#Remote host backup directory

my $ remotedir = ‘[email protected] :: lansggtest’;

#Maximum number of threads

my $ thread_max = 5;

my $ backupTime = strftime ("% Y% m% d% H% M% S", localtime (time));

print "begin: $ backupTime \ n";

#Retrieve all files in the directory to be backed up, except for. .linux in the current directory

sub findAllFile {

        unless ($ _ eq ‘.‘) {

        print "corrent file: $ File :: Find :: name \ n";

        $ fileQueue-> enqueue ($ _);

        }

}

find (\ & findAllFile, $ srcFilePath);

#Transfer using rsync

sub rsync {

    my $ file = shift;

    print "rsync-$ file \ n";

    my $ obj = File :: Rsync-> new (

    {

    archive => 1,

    compress => 1,

    checksum => 1,

    recursive => 1,

    times => 1,

# verbose => 1,

    timeout => 300,

    progress => 1,

    stats => 1,

    ‘Ignore-times’ => 1,

    ‘Password-file’ => ‘./rsync.pass’,

    }

);

$ obj-> exec ({src => "$ srcFilePath $ file", dest => $ remotedir}) or warn "rsync Failed! \ n";

#print $ obj-> out;

}

#Checking untransmitted files in the queue

while ($ fileQueue-> pending ()) {

    while (scalar (threads-> list ()) <$ thread_max) {

# my $ readQueue = $ fileQueue-> dequeue (); #This is used for debugging, to view the currently transferred file

# print "current file Queue is $ readQueue \ n";

#Generating threads

        threads-> create (\ & rsync, $ readQueue);

#View the current total number of threads

        my $ thread_count = threads-> list ();

# print "thread_count is $ thread_count \ n";

    }

#Determine if the current thread has completed the job and recycle it

    foreach my $ thread (threads-> list (threads :: all)) {

        if ($ thread-> is_joinable ()) {

                $ thread-> join ();

            }

        }

}

#joindrop the remaining threads (because when the queue in the while is empty, there may still be threads executing, but the program will exit the while loop at this time, so an additional program is needed to join the remaining threads)

foreach my $ thread (threads-> list (threads :: all)) {

    $ thread-> join ();

    }

$ backupTime = strftime ("% Y% m% d% H% M% S", localtime (time));

print "end: $ backupTime \ n";

This script is a core feature that can be later added with logging, mail sending and other functions.

When we execute the script, look at the host thread condition

[[email protected] pl]# ps -ef |grep tq

root       6377   2152 88 19:05 pts/3    00:00:12 perl ./tq.pl

[[email protected] pl]# pstree -p 6377

perl(6377)─┬─rsync(6379)

          ├─rsync(6381)

          ├─rsync(6383)

          ├─rsync(6385)

          ├─rsync(6387)

          ├─{perl}(6378)

          ├─{perl}(6380)

          ├─{perl}(6382)

          ├─{perl}(6384)

          └─{perl}(6386)

         

[[email protected] pl]# ps -ef |grep rsync

root       6379   6377 14 19:05 pts/3    00:00:14 rsync --archive --checksum --compress --ignore-times --progress --recursive --stats --times --password-file=./rsync.pass --timeout=300 /root/test//test13.data [email protected]::lansggtest

root       6381   6377 14 19:05 pts/3    00:00:14 rsync --archive --checksum --compress --ignore-times --progress --recursive --stats --times --password-file=./rsync.pass --timeout=300 /root/test//test12.data [email protected]::lansggtest

root       6383   6377 14 19:05 pts/3    00:00:14 rsync --archive --checksum --compress --ignore-times --progress --recursive --stats --times --password-file=./rsync.pass --timeout=300 /root/test//test1.data [email protected]::lansggtest

root       6385   6377 14 19:05 pts/3    00:00:14 rsync --archive --checksum --compress --ignore-times --progress --recursive --stats --times --password-file=./rsync.pass --timeout=300 /root/test//test8.data [email protected]::lansggtest

root       6387   6377 12 19:05 pts/3    00:00:12 rsync --archive --checksum --compress --ignore-times --progress --recursive --stats --times --password-file=./rsync.pass --timeout=300 /root/test//test3.data [email protected]::lansggtest

root       6399   2193  0 19:06 pts/2    00:00:00 grep rsync

This article is from the "Big Wind" blog, please be sure to keep this source http://lansgg.blog.51cto.com/5675165/1787988

Perl multithreaded rsync backup file to remote host