A Peer-to-peer file transfer instance implemented by Python _python

Given the increasing number of servers in my hands, and sometimes the need to deploy the same file on a large scale, for example, because of the convenience of using the Systemtap tool to locate the problem, you need to have hundreds of servers installed at the same time Kernel-debuginfo this package, the original way to use a source server , using rsync or SCP file transfer mode can only do a point down to distribute this file, this time will be compared to the speed of slow, based on the above reasons, I wrote a BT protocol transfer files of the small tool, the actual test, transmission to 10 rooms, More than 70 machines transmit a 240M of this kernel file, to all machines, the source uses speed limit 2m/s upload speed, the test results probably as long as 140s, you can complete the transmission, this efficiency is very high, if the speed will be faster, below the program open source.

#!/usr/bin/env python import libtorrent as LT import sys import OS import time from optparse import optionparser Import Socket import struct Import fcntl def get_interface_ip (ifname): s = socket.socket (socket.af_inet, socket. SOCK_DGRAM) return Socket.inet_ntoa (Fcntl.ioctl (S.fileno (), 0x8915, Struct.pack (' 256s ', ifname[:15]) [20:2
 
 
4] def ip2long (IP): Return to reduce (lambda a,b: (a<<8) +b,[int (i) for I in Ip.split ('. '))  Def get_wan_ip_address (): interfaces = set ([' eth0 ', ' eth1 ', ' eth2 ', ' eth3 ', ' em1 ', ' em2 ', ' em3 ', ' em4 ']) IP = ' for I in Interfaces:try:ip = Get_interface_ip (i) if (Ip2long (IP) < Ip2long (' 10.0.0.0 ') or Ip2long (IP) &G T Ip2long (' 10.255.255.255 ')) \ and (Ip2long (IP) < Ip2long (' 172.16.0.0 ') or Ip2long (IP) > Ip2long (' 172.33.255.2 ] \ and (Ip2long (IP) < Ip2long (' 192.168.0.0 ') or Ip2long (IP) > Ip2long (' 192.168.255.255 ')): Retu RN IP except:pass return IP def make_tOrrent (Path, save): FS = Lt.file_storage () lt.add_files (FS, path) if fs.num_files () = = 0:print ' No files added ' Sys.exit (1) input = Os.path.abspath (path) basename = os.path.basename (path) t = lt.create_torrent (FS, 0, 4 * 1024 * 1024) t.add_tracker ("Http://10.0.1.5:8760/announce") t.set_creator (' libtorrent%s '% lt.version) lt.set_p
  Iece_hashes (t, os.path.split (input) [0], Lambda x:sys.stderr.write ('. '))
  Sys.stderr.write (' \ n ') Save = os.path.dirname (input) Save = "%s/%s.torrent"% (save, basename) F=open (Save, "WB") F.write (Lt.bencode (t.generate)) F.close () print "The BT torrent file is store at%s"% Save Def dl_status (hand Le): While Not (Handle.is_seed ()): s = handle.status () state_str = [' Queued ', ' checking ', ' downloading Metadat A ', \ ' downloading ', ' finished ', ' seeding ', ' allocating ', ' checking fastresume '] print ' \ractive_time:%d,%.2 f%% complete (down:%.1f kb/s up:%.1f kb/s peers:%d, Seeds:%d)%s '% \ (S.active_time, s.progress *, s.download_rate/1000, s.upload_rate/1000, S.num_peers, S.num_se EDS, State_str[s.state]), Sys.stdout.flush () time.sleep (1) def seed_status (handle, seedtime=100): Seedtime = i 
 
    NT (Seedtime) if seedtime < 100:seedtime = + while Seedtime > 0:seedtime-= 1 s = handle.status () STATE_STR = [' Queued ', ' checking ', ' downloading metadata ', ' downloading ', ' finished ', ' seeding ', ' Allocat ' ing ', ' checking fastresume '] print ' \rseed_time:%d,%.2f%% complete (down:%.1f kb/s-up:%.1f kb/s:%d, Peers:  %d)%s '% \ (s.active_time, s.progress *, s.download_rate/1000, s.upload_rate/1000, S.num_peers, S.num_seeds, State_str[s.state]), Sys.stdout.flush () time.sleep (1) def remove_torrents (torrent, session): s Ession.remove_torrent (torrent) def read_alerts (session): Alert = Session.pop_alert () While alert: #print alert, Alert.message () Alert = Session.pop_alert () def download (torrent, path, upload_rate_limit=0, seedtime=100): Try:session = Lt.session ( ) Session.set_alert_queue_size_limit (1024 * 1024) sts = lt.session_settings () Sts.ssl_listen = False sts . user_agent = "Thunder Deploy system" Sts.tracker_completion_timeout = 5 Sts.tracker_receive_timeout = 5 STS.S Top_tracker_timeout = 5 Sts.active_downloads =-1 sts.active_seeds =-1 Sts.active_limit =-1 STS.AUTO_SCR  Ape_min_interval = 5 Sts.udp_tracker_token_expiry = Sts.min_announce_interval = 1 Sts.inactivity_timeout =
    Sts.connection_speed = Ten Sts.allow_multiple_connections_per_ip = True sts.max_out_request_queue = 128 Sts.request_queue_size = 3 Sts.use_read_cache = False session.set_settings (STS) Session.set_alert_mask (LT . alert.category_t.tracker_notification | lt.alert.category_t.status_notification) Session.set_alert_mask (Lt.alert.category_t.status_notificatiON) ipaddr = Get_wan_ip_address () #print ipaddr if ipaddr = = "": session.listen_on (6881, 6881) Els E:session.listen_on (6881, 6881, ipaddr) limit = Int (upload_rate_limit) if limit>=100:session.se T_upload_rate_limit (limit*1024) session.set_local_upload_rate_limit (limit*1024) Print session.upload_rate_limit ( ) torrent_info = lt.torrent_info (torrent) add_params = {' Save_path ': path, ' Storage_mode ': lt.storage _mode_t.storage_mode_sparse, ' paused ': False, ' auto_managed ': True, ' ti ': torrent_info,} hand Le = session.add_torrent (add_params) Read_alerts (session) St = Time.time () dl_status (handle) et = time.t IME ()-St print ' \nall file download in%.2f\nstart to seeding\n '% et sys.stdout.write (' \ n ') handle.super_se  Eding () seed_status (handle, Seedtime) remove_torrents (handle, session) Assert Len (session.get_torrents ()) = = 0 Finally:prinT ' download finished ' if __name__ = ' __main__ ': Usage = ' usage:%prog [options] \ n \%prog-d-F <torrent file= "" >-S <file save= "" path= "" >\n \ or \ n \%prog-m-P <file or= "" dir= "" >-S <torrent save= "" Path = "" >\n "parser = Optionparser (usage=usage) parser.add_option ("-D ","--download ", dest=" Download ", help=" STA " RT to download file ", action=" Store_false ", Default=true) parser.add_option (" F ","--file ", dest=" file ", help=" Tor  Rent file ") parser.add_option ("-U ","--upload ", dest=" upload ", help=" set upload rate limit, default is not limit ", default=0) parser.add_option ("T", "--time", dest= "Time", help= "set seed time, default is 100s", default=100) p Arser.add_option ("-P", "--path", dest= "path", help= "to do torrent with this path") parser.add_option ("-M", "--ma Ke ", dest=" make ", help=" make torrent ", action=" Store_false ", Default=true) parser.add_option (" s ","--save ", dest= "Save", help= "file" Save path, default is store to./", default="./") (options, args) = Parser.parse_args () #download (sys.argv[1)) if L En (SYS.ARGV)!= 6 and Len (SYS.ARGV)!= 4 and Len (SYS.ARGV)!= 8 and Len (SYS.ARGV)!= 10:parser.print_help () sys.e XIT () if options.download = = False and Options.file!= "": Download (Options.file, Options.save, Options.upload, Optio Ns.time) elif Options.make = = False and Options.path!= "": Make_torrent (Options.path, Options.save) </torrent&gt ;</file></file></torrent>

Preparation environment:
you need to install a libtorrent library on all the OS and download the address:

Http://code.google.com/p/libtorrent/downloads/list

Remember to compile the time with the./configure–enable-python-binding, then Mak,make install, into the binding directory, Make,make install on
Can run this little tool
Of course, large-scale deployment can not use each to compile the installation of the way, as long as the compiled libtorrent.so libtorrent-rasterbar.so.7 file with bt.py this file into the same directory, and write a shell script

Copy Code code as follows:

Lib= ' DirName $ '
Export ld_library_path= $LD _library_path: $lib
Python bt.py-d-F < seed files >-s < file save path >-T < do kind of time >-u < limit upload speed >

How to use:
First generate the seed file on the source server

Copy Code code as follows:

Python bt.py-m-P < files or folders to publish >-S < seed save Address >

Publish File
On the source server, execute

Copy Code code as follows:

Python bt.py-d-F < seed files >-s < file save path >-T < do kind of time >-u < limit upload speed >

The default setting of the time is 100s, upload speed by default does not limit the speed of the unit is KB

Below the machine, directly can

Copy Code code as follows:

Python bt.py-d-F < seed files >-s < file save path >-T < do kind of time >

As long as there is a machine completed, automatically as a seed, in the process of downloading will be uploaded, any machine can be used as a source server, of course, there is also the center of the tracker server, in the script, I built a tracker source server, Put on the 10.0.1.5 port is 8760, of course, you can also use OpenTracker this software to build a tracker server, modify the corresponding part of the source code, in addition to the release are private files, code as has been banned DHT, if you want to be more secure, you build a private TR Acker Server, concrete building method to use the search engine, find out how to build!

At present basically do can use, the follow-up consideration is simpler, the use of magnetic link way, so you can do without each to copy a seed file, using a separate command line can publish the entire file