Details Python implementation of P2P file transfer efficiency instances between servers

Application scenarios

The number of servers is growing. You need to deploy the same file on a large scale. For example, you need to install the kernel-debuginfo package on hundreds of servers at the same time because it is convenient to use the systemtap tool to locate problems, the original method uses a source server, and The rsync or scp file transmission method can only distribute this file at one point. At this time, the delivery speed will be relatively slow, for the above reasons, I wrote a tool for transferring files based on the bt protocol. In actual tests, the tool was transferred to 10 data centers, and more than 70 machines were transferred to a M kernel file, to all machines, the source uses a 2 Mbit/s speed limit for uploading. The test results can be transmitted as long as 140 seconds. This efficiency is very high, if the speed is not limited, the speed will be faster.

In a distributed deployment environment, such as puppet and saltstack, the file distribution efficiency is greatly improved.

The code is as follows:

Copy code

#! /Usr/bin/env python Import libtorrent as lt Import sys Import OS Import time From optparse import OptionParser Import socket Import struct Import fcntl Def get_interface_ip (ifname ): S = socket. socket (socket. AF_INET, socket. SOCK_DGRAM) Return socket. inet_ntoa (fcntl. ioctl (s. fileno (), 0x8915, struct. pack ('256s ', Ifname [: 15]) [20: 24]) Def ip2long (ip ): Return reduce (lambda a, B :( a <8) + B, [int (I) for I in ip. split ('.')])   Def get_wan_ip_address (): Interfaces = set (['eth0', 'eth1', 'eth2', 'eth3', 'em1', 'em2', 'em3', 'em4']) Ip ='' For I in interfaces: Try: Ip = get_interface_ip (I) If (ip2long (ip) <ip2long ('10. 0.0.0 ') or ip2long (ip)> ip2long ('10. 255.255.255 ')) And (ip2long (ip) <ip2long ('192. 16.0.0 ') or ip2long (ip)> ip2long ('192. 33.255.255 ')) And (ip2long (ip) <ip2long ('192. 168.0.0 ') or ip2long (ip)> ip2long ('192. 168.255.255 ')): Return ip Except t: Pass Return ip Def make_torrent (path, save ): Fs = lt. file_storage () Lt. add_files (fs, path) If fs. num_files () = 0: Print 'No files added' Sys. exit (1) Input = OS. path. abspath (path) Basename = OS. path. basename (path) T = lt. create_torrent (fs, 0, 4*1024*1024) T. add_tracker ("http: // 10.0.1.5: 8760/announce ") T. set_creator ('libtorrent % s' % lt. version) Lt. set_piece_hashes (t, OS. path. split (input) [0], lambda x: sys. stderr. write ('.')) Sys. stderr. write ('N ') Save = OS. path. dirname (input) Save = "% s/% s. torrent" % (save, basename) F = open (save, "wb ") F. write (lt. bencode (t. generate ())) F. close () Print "the bt torrent file is store at % s" % save   Def dl_status (handle ): While not (handle. is_seed ()): S = handle. status () State_str = ['queued', 'checking', 'downloading metadata ', 'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume'] Print 'ractive _ time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' % (S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000, S. num_peers, s. num_seeds, state_str [s. state]), Sys. stdout. flush () Time. sleep (1) Def seed_status (handle, seedtime = 100 ): Seedtime = int (seedtime) If seedtime: <100: Seedtime= 100 While seedtime> 0: Seedtime-= 1 S = handle. status () State_str = ['queued', 'checking', 'downloading metadata ', 'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume'] Print 'rseed _ time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' % (S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000, S. num_peers, s. num_seeds, state_str [s. state]), Sys. stdout. flush () Time. sleep (1) Def remove_torrents (torrent, session ): Session. remove_torrent (torrent) Def read_alerts (session ): Alert = session. pop_alert () While alert: # Print alert, alert. message () Alert = session. pop_alert () Def download (torrent, path, upload_rate_limit = 0, seedtime = 100 ): Try: Session = lt. session () Session. set_alert_queue_size_limit (1024*1024) Sts = lt. session_settings () Sts. ssl_listen = False Sts. user_agent = "Thunder deploy system" Sts. tracker_completion_timeout = 5 Sts. tracker_receive_timeout = 5 Sts. stop_tracker_timeout = 5 Sts. active_downloads =-1 Sts. active_seeds =-1 Sts. active_limit =-1 Sts. auto_scrape_min_interval = 5 Sts. udp_tracker_token_expiry = 120 Sts. min_announce_interval = 1 Sts. inactivity_timeout = 60 Sts. connection_speed = 10 Sts. allow_multiple_connections_per_ip = True Sts. max_out_request_queue = 128 Sts. request_queue_size = 3 Sts. use_read_cache = False Session. set_settings (sts) Session. set_alert_mask (lt. alert. category_t.tracker_notification | lt. alert. category_t.status_notification) Session. set_alert_mask (lt. alert. category_t.status_notification) Ipaddr = get_wan_ip_address () # Print ipaddr If ipaddr = "": Session. listen_on (6881,688 1) Else: Session. listen_on (6881,688 1, ipaddr) Limit = int (upload_rate_limit) If limit >=100: Session. set_upload_rate_limit (limit * 1024) Session. set_local_upload_rate_limit (limit * 1024) Print session. upload_rate_limit () Torrent_info = lt. torrent_info (torrent) Add_params = { 'SAVE _ path': path, 'Storage _ mode': lt. storage_mode_t.storage_mode_sparse, 'Paused': False, 'Auto _ managed': True, 'Ti': torrent_info,         } Handle = session. add_torrent (add_params) Read_alerts (session) St = time. time () Dl_status (handle) Et = time. time ()-st Print 'all file download in %. 2 fnstart to seedingn' % et Sys. stdout. write ('N ') Handle. super_seeding () Seed_status (handle, seedtime) Remove_torrents (handle, session) Assert len (session. get_torrents () = 0 Finally: Print 'download finished' If _ name _ = '_ main __': Usage = "usage: % prog [options] n % Prog-d-f <torrent file = "">-s <file save = "" path = ""> n Or n % Prog-m-p <file or = "" dir = "">-s <torrent save = "" path = ""> n" Parser = OptionParser (usage = usage) Parser. add_option ("-d", "-- download", dest = "download ", Help = "start to download file", action = "store_false", default = True) Parser. add_option ("-f", "-- file", dest = "file ", Help = "torrent file ") Parser. add_option ("-u", "-- upload", dest = "upload ", Help = "set upload rate limit, default is not limit", default = 0) Parser. add_option ("-t", "-- time", dest = "time ", Help = "set seed time, default is 100 s", default = 100) Parser. add_option ("-p", "-- path", dest = "path ", Help = "to make torrent with this path ") Parser. add_option ("-m", "-- make", dest = "make ", Help = "make torrent", action = "store_false", default = True) Parser. add_option ("-s", "-- save", dest = "save ", Help = "file save path, default is store to./", default = "./") (Options, args) = parser. parse_args () # Download (sys. argv [1]) If len (sys. argv )! = 6 and len (sys. argv )! = 4 and len (sys. argv )! = 8 and len (sys. argv )! = 10: Parser. print_help () Sys. exit () If options. download = False and options. file! = "": Download (options. file, options. save, options. upload, options. time) Elif options. make = False and options. path! = "": Make_torrent (options. path, options. save) </Torrent> </file> </torrent>

Prepare the environment

You need to install a libtorrent Library on all operating systems. Download address:

Http://code.google.com/p/libtorrent/downloads/list

Remember to include./configure-enable-python-binding during compilation, and then mak, make install, enter the binding Directory, make, and make install to run this small tool. Of course, for large-scale deployment, it is impossible to compile and install each server, as long as the compiled libtorrent. so libtorrent-rasterbar.so.7 files with bt. put the file py in the same directory and write a shell script.
Lib = 'dirname $0'
Export LD_LIBRARY_PATH = $ LD_LIBRARY_PATH: $ lib
Python bt. py-d-f <seed file>-s <file storage path>-t <preparation time>-u <limit upload speed>
Usage

First, generate a seed file on the source server, as shown below:
Python bt. py-m-p <file or folder to be published>-s <seed storage address>

Publish a file

Run

Python bt. py-d-f <seed file>-s <file storage path>-t <preparation time>-u <limit upload speed>

The default setting is 100 s, and the upload speed is unlimited by default. The unit of speed limit is kb.

As long as one machine is complete, it will be automatically used as a seed and uploaded during the download process. Any machine can be used as the source server. Of course, there is also a central tracker server, in the script, I set up a tracker source server and put it on port 10.0.1.5, which is 8760. Of course, you can also use the opentracker software to build a tracker server and modify the source code, in addition, considering that the release is a private file, the code has disabled the dht. If you want to be safer, you can build a private tracker server by yourself. You can use the search engine to build the tracker server, find the build method!

At present, it is basically usable. It will be simpler to consider in the future and adopt the magnetic link method, so that you do not need to copy a seed file for each server, you can use a single command line to publish the entire file.