Details Python implementation of P2P file transfer efficiency instances between servers

Source: Internet
Author: User
Tags flush scp file sleep socket

Application scenarios

The number of servers is growing. You need to deploy the same file on a large scale. For example, you need to install the kernel-debuginfo package on hundreds of servers at the same time because it is convenient to use the systemtap tool to locate problems, the original method uses a source server, and The rsync or scp file transmission method can only distribute this file at one point. At this time, the delivery speed will be relatively slow, for the above reasons, I wrote a tool for transferring files based on the bt protocol. In actual tests, the tool was transferred to 10 data centers, and more than 70 machines were transferred to a M kernel file, to all machines, the source uses a 2 Mbit/s speed limit for uploading. The test results can be transmitted as long as 140 seconds. This efficiency is very high, if the speed is not limited, the speed will be faster.

In a distributed deployment environment, such as puppet and saltstack, the file distribution efficiency is greatly improved.

The code is as follows: Copy code
#! /Usr/bin/env python

Import libtorrent as lt
Import sys
Import OS
Import time
From optparse import OptionParser
Import socket
Import struct
Import fcntl

Def get_interface_ip (ifname ):
S = socket. socket (socket. AF_INET, socket. SOCK_DGRAM)
Return socket. inet_ntoa (fcntl. ioctl (s. fileno (), 0x8915, struct. pack ('256s ',
Ifname [: 15]) [20: 24])
Def ip2long (ip ):
Return reduce (lambda a, B :( a <8) + B, [int (I) for I in ip. split ('.')])

 
Def get_wan_ip_address ():
Interfaces = set (['eth0', 'eth1', 'eth2', 'eth3', 'em1', 'em2', 'em3', 'em4'])
Ip =''
For I in interfaces:
Try:
Ip = get_interface_ip (I)
If (ip2long (ip) <ip2long ('10. 0.0.0 ') or ip2long (ip)> ip2long ('10. 255.255.255 '))
And (ip2long (ip) <ip2long ('192. 16.0.0 ') or ip2long (ip)> ip2long ('192. 33.255.255 '))
And (ip2long (ip) <ip2long ('192. 168.0.0 ') or ip2long (ip)> ip2long ('192. 168.255.255 ')):
Return ip
Except t:
Pass

Return ip

Def make_torrent (path, save ):
Fs = lt. file_storage ()
Lt. add_files (fs, path)
If fs. num_files () = 0:
Print 'No files added'
Sys. exit (1)

Input = OS. path. abspath (path)
Basename = OS. path. basename (path)
T = lt. create_torrent (fs, 0, 4*1024*1024)

T. add_tracker ("http: // 10.0.1.5: 8760/announce ")
T. set_creator ('libtorrent % s' % lt. version)

Lt. set_piece_hashes (t, OS. path. split (input) [0], lambda x: sys. stderr. write ('.'))
Sys. stderr. write ('N ')

Save = OS. path. dirname (input)
Save = "% s/% s. torrent" % (save, basename)
F = open (save, "wb ")
F. write (lt. bencode (t. generate ()))
F. close ()
Print "the bt torrent file is store at % s" % save

 
Def dl_status (handle ):
While not (handle. is_seed ()):
S = handle. status ()

State_str = ['queued', 'checking', 'downloading metadata ',
'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume']
Print 'ractive _ time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' %
(S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000,
S. num_peers, s. num_seeds, state_str [s. state]),
Sys. stdout. flush ()

Time. sleep (1)
Def seed_status (handle, seedtime = 100 ):
Seedtime = int (seedtime)
If seedtime: <100:
Seedtime= 100
While seedtime> 0:
Seedtime-= 1
S = handle. status ()

State_str = ['queued', 'checking', 'downloading metadata ',
'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume']
Print 'rseed _ time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' %
(S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000,
S. num_peers, s. num_seeds, state_str [s. state]),
Sys. stdout. flush ()

Time. sleep (1)

Def remove_torrents (torrent, session ):
Session. remove_torrent (torrent)

Def read_alerts (session ):
Alert = session. pop_alert ()
While alert:
# Print alert, alert. message ()
Alert = session. pop_alert ()

Def download (torrent, path, upload_rate_limit = 0, seedtime = 100 ):
Try:
Session = lt. session ()
Session. set_alert_queue_size_limit (1024*1024)

Sts = lt. session_settings ()
Sts. ssl_listen = False
Sts. user_agent = "Thunder deploy system"
Sts. tracker_completion_timeout = 5
Sts. tracker_receive_timeout = 5
Sts. stop_tracker_timeout = 5
Sts. active_downloads =-1
Sts. active_seeds =-1
Sts. active_limit =-1
Sts. auto_scrape_min_interval = 5
Sts. udp_tracker_token_expiry = 120
Sts. min_announce_interval = 1
Sts. inactivity_timeout = 60
Sts. connection_speed = 10
Sts. allow_multiple_connections_per_ip = True
Sts. max_out_request_queue = 128
Sts. request_queue_size = 3

Sts. use_read_cache = False
Session. set_settings (sts)

Session. set_alert_mask (lt. alert. category_t.tracker_notification | lt. alert. category_t.status_notification)
Session. set_alert_mask (lt. alert. category_t.status_notification)

Ipaddr = get_wan_ip_address ()
# Print ipaddr
If ipaddr = "":
Session. listen_on (6881,688 1)
Else:
Session. listen_on (6881,688 1, ipaddr)

Limit = int (upload_rate_limit)
If limit >=100:
Session. set_upload_rate_limit (limit * 1024)
Session. set_local_upload_rate_limit (limit * 1024)
Print session. upload_rate_limit ()
Torrent_info = lt. torrent_info (torrent)
Add_params = {
'SAVE _ path': path,
'Storage _ mode': lt. storage_mode_t.storage_mode_sparse,
'Paused': False,
'Auto _ managed': True,
'Ti': torrent_info,
        }

Handle = session. add_torrent (add_params)

Read_alerts (session)
St = time. time ()
Dl_status (handle)
Et = time. time ()-st
Print 'all file download in %. 2 fnstart to seedingn' % et
Sys. stdout. write ('N ')
Handle. super_seeding ()
Seed_status (handle, seedtime)

Remove_torrents (handle, session)
Assert len (session. get_torrents () = 0

Finally:
Print 'download finished'

If _ name _ = '_ main __':
Usage = "usage: % prog [options] n
% Prog-d-f <torrent file = "">-s <file save = "" path = ""> n
Or n
% Prog-m-p <file or = "" dir = "">-s <torrent save = "" path = ""> n"

Parser = OptionParser (usage = usage)
Parser. add_option ("-d", "-- download", dest = "download ",
Help = "start to download file", action = "store_false", default = True)
Parser. add_option ("-f", "-- file", dest = "file ",
Help = "torrent file ")
Parser. add_option ("-u", "-- upload", dest = "upload ",
Help = "set upload rate limit, default is not limit", default = 0)
Parser. add_option ("-t", "-- time", dest = "time ",
Help = "set seed time, default is 100 s", default = 100)
Parser. add_option ("-p", "-- path", dest = "path ",
Help = "to make torrent with this path ")
Parser. add_option ("-m", "-- make", dest = "make ",
Help = "make torrent", action = "store_false", default = True)
Parser. add_option ("-s", "-- save", dest = "save ",
Help = "file save path, default is store to./", default = "./")
(Options, args) = parser. parse_args ()
# Download (sys. argv [1])
If len (sys. argv )! = 6 and len (sys. argv )! = 4 and len (sys. argv )! = 8 and len (sys. argv )! = 10:
Parser. print_help ()
Sys. exit ()
If options. download = False and options. file! = "":
Download (options. file, options. save, options. upload, options. time)
Elif options. make = False and options. path! = "":
Make_torrent (options. path, options. save)
</Torrent> </file> </torrent>
Prepare the environment

You need to install a libtorrent Library on all operating systems. Download address:

Http://code.google.com/p/libtorrent/downloads/list

Remember to include./configure-enable-python-binding during compilation, and then mak, make install, enter the binding Directory, make, and make install to run this small tool. Of course, for large-scale deployment, it is impossible to compile and install each server, as long as the compiled libtorrent. so libtorrent-rasterbar.so.7 files with bt. put the file py in the same directory and write a shell script.
Lib = 'dirname $0'
Export LD_LIBRARY_PATH = $ LD_LIBRARY_PATH: $ lib
Python bt. py-d-f <seed file>-s <file storage path>-t <preparation time>-u <limit upload speed>
Usage

First, generate a seed file on the source server, as shown below:
Python bt. py-m-p <file or folder to be published>-s <seed storage address>

Publish a file

Run

Python bt. py-d-f <seed file>-s <file storage path>-t <preparation time>-u <limit upload speed>

The default setting is 100 s, and the upload speed is unlimited by default. The unit of speed limit is kb.

As long as one machine is complete, it will be automatically used as a seed and uploaded during the download process. Any machine can be used as the source server. Of course, there is also a central tracker server, in the script, I set up a tracker source server and put it on port 10.0.1.5, which is 8760. Of course, you can also use the opentracker software to build a tracker server and modify the source code, in addition, considering that the release is a private file, the code has disabled the dht. If you want to be safer, you can build a private tracker server by yourself. You can use the search engine to build the tracker server, find the build method!

At present, it is basically usable. It will be simpler to consider in the future and adopt the magnetic link method, so that you do not need to copy a seed file for each server, you can use a single command line to publish the entire file.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.