A p2p File Transfer instance implemented by python

Considering the increasing number of servers in my hands, it is sometimes necessary to deploy the same file on a large scale. For example, it is convenient to use the systemtap tool to locate the problem, you need to install the kernel-debuginfo package on hundreds of servers at the same time. The original method is to use one source server, and the file transmission method such as rsync or scp can only be used to distribute the file at one point, at this time, the delivery speed will be relatively slow. Based on the above reasons, I wrote a tool for transferring files based on the bt protocol. In actual tests, the tool was transferred to 10 data centers, more than 70 machines transmit a 140 m Kernel File to all machines. The source uses a 2 m/s speed limit for uploading. The test result is about s, the entire transmission can be completed, and the efficiency is very high. If the speed is not limited, the speed will be faster. The program will be open-source below.

Copy codeThe Code is as follows:
#! /Usr/bin/env python

Import libtorrent as lt
Import sys
Import OS
Import time
From optparse import OptionParser
Import socket
Import struct
Import fcntl

Def get_interface_ip (ifname ):
S = socket. socket (socket. AF_INET, socket. SOCK_DGRAM)
Return socket. inet_ntoa (fcntl. ioctl (s. fileno (), 0x8915, struct. pack ('256s ',
Ifname [: 15]) [20: 24])
Def ip2long (ip ):
Return reduce (lambda a, B :( a <8) + B, [int (I) for I in ip. split ('.')])

 
Def get_wan_ip_address ():
Interfaces = set (['eth0', 'eth1', 'eth2', 'eth3', 'em1', 'em2', 'em3', 'em4'])
Ip =''
For I in interfaces:
Try:
Ip = get_interface_ip (I)
If (ip2long (ip) <ip2long ('10. 0.0.0 ') or ip2long (ip)> ip2long ('10. 255.255.255 '))\
And (ip2long (ip) <ip2long ('192. 16.0.0 ') or ip2long (ip)> ip2long ('192. 33.0000255 '))\
And (ip2long (ip) <ip2long ('192. 168.0.0 ') or ip2long (ip)> ip2long ('192. 168.255.255 ')):
Return ip
Except t:
Pass

Return ip

Def make_torrent (path, save ):
Fs = lt. file_storage ()
Lt. add_files (fs, path)
If fs. num_files () = 0:
Print 'no files added'
Sys. exit (1)

Input = OS. path. abspath (path)
Basename = OS. path. basename (path)
T = lt. create_torrent (fs, 0, 4*1024*1024)

T. add_tracker ("http: // 10.0.1.5: 8760/announce ")
T. set_creator ('libtorrent % s' % lt. version)

Lt. set_piece_hashes (t, OS. path. split (input) [0], lambda x: sys. stderr. write ('.'))
Sys. stderr. write ('\ n ')

Save = OS. path. dirname (input)
Save = "% s/% s. torrent" % (save, basename)
F = open (save, "wb ")
F. write (lt. bencode (t. generate ()))
F. close ()
Print "the bt torrent file is store at % s" % save

 
Def dl_status (handle ):
While not (handle. is_seed ()):
S = handle. status ()

State_str = ['queued', 'checking', 'downloading metadata ',\
'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume']
Print '\ ractive_time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' % \
(S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000 ,\
S. num_peers, s. num_seeds, state_str [s. state]),
Sys. stdout. flush ()

Time. sleep (1)
Def seed_status (handle, seedtime = 100 ):
Seedtime = int (seedtime)
If seedtime: <100:
Seedtime= 100
While seedtime> 0:
Seedtime-= 1
S = handle. status ()

State_str = ['queued', 'checking', 'downloading metadata ',\
'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume']
Print '\ rseed_time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' % \
(S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000 ,\
S. num_peers, s. num_seeds, state_str [s. state]),
Sys. stdout. flush ()

Time. sleep (1)

Def remove_torrents (torrent, session ):
Session. remove_torrent (torrent)

Def read_alerts (session ):
Alert = session. pop_alert ()
While alert:
# Print alert, alert. message ()
Alert = session. pop_alert ()

Def download (torrent, path, upload_rate_limit = 0, seedtime = 100 ):
Try:
Session = lt. session ()
Session. set_alert_queue_size_limit (1024*1024)

Sts = lt. session_settings ()
Sts. ssl_listen = False
Sts. user_agent = "Thunder deploy system"
Sts. tracker_completion_timeout = 5
Sts. tracker_receive_timeout = 5
Sts. stop_tracker_timeout = 5
Sts. active_downloads =-1
Sts. active_seeds =-1
Sts. active_limit =-1
Sts. auto_scrape_min_interval = 5
Sts. udp_tracker_token_expiry = 120
Sts. min_announce_interval = 1
Sts. inactivity_timeout = 60
Sts. connection_speed = 10
Sts. allow_multiple_connections_per_ip = True
Sts. max_out_request_queue = 128
Sts. request_queue_size = 3

Sts. use_read_cache = False
Session. set_settings (sts)

Session. set_alert_mask (lt. alert. category_t.tracker_notification | lt. alert. category_t.status_notification)
Session. set_alert_mask (lt. alert. category_t.status_notification)

Ipaddr = get_wan_ip_address ()
# Print ipaddr
If ipaddr = "":
Session. listen_on (6881,688 1)
Else:
Session. listen_on (6881,688 1, ipaddr)

Limit = int (upload_rate_limit)
If limit >=100:
Session. set_upload_rate_limit (limit * 1024)
Session. set_local_upload_rate_limit (limit * 1024)
Print session. upload_rate_limit ()
Torrent_info = lt. torrent_info (torrent)
Add_params = {
'Save _ path': path,
'Storage _ mode': lt. storage_mode_t.storage_mode_sparse,
'Paused': False,
'Auto _ managed': True,
'Ti': torrent_info,
}

Handle = session. add_torrent (add_params)

Read_alerts (session)
St = time. time ()
Dl_status (handle)
Et = time. time ()-st
Print '\ nall file download in %. 2f \ nstart to seeding \ n' % et
Sys. stdout. write ('\ n ')
Handle. super_seeding ()
Seed_status (handle, seedtime)

Remove_torrents (handle, session)
Assert len (session. get_torrents () = 0

Finally:
Print 'Download finished'

If _ name _ = '_ main __':
Usage = "usage: % prog [options] \ n \
% Prog-d-f <torrent file = "">-s <file save = "" path = ""> \ n \
Or \ n \
% Prog-m-p <file or = "" dir = "">-s <torrent save = "" path = ""> \ n"

Parser = OptionParser (usage = usage)
Parser. add_option ("-d", "-- download", dest = "download ",
Help = "start to download file", action = "store_false", default = True)
Parser. add_option ("-f", "-- file", dest = "file ",
Help = "torrent file ")
Parser. add_option ("-u", "-- upload", dest = "upload ",
Help = "set upload rate limit, default is not limit", default = 0)
Parser. add_option ("-t", "-- time", dest = "time ",
Help = "set seed time, default is 100 s", default = 100)
Parser. add_option ("-p", "-- path", dest = "path ",
Help = "to make torrent with this path ")
Parser. add_option ("-m", "-- make", dest = "make ",
Help = "make torrent", action = "store_false", default = True)
Parser. add_option ("-s", "-- save", dest = "save ",
Help = "file save path, default is store to./", default = "./")
(Options, args) = parser. parse_args ()
# Download (sys. argv [1])
If len (sys. argv )! = 6 and len (sys. argv )! = 4 and len (sys. argv )! = 8 and len (sys. argv )! = 10:
Parser. print_help ()
Sys. exit ()
If options. download = False and options. file! = "":
Download (options. file, options. save, options. upload, options. time)
Elif options. make = False and options. path! = "":
Make_torrent (options. path, options. save)
</Torrent> </file> </torrent>

Prepare the environment:
Install a libtorrent library on all operating systems ,:

Http://code.google.com/p/libtorrent/downloads/list

Remember to include./configure-enable-python-binding during compilation, then mak, make install, and enter the binding directory. make and make install
You can run this small tool.
Of course, for large-scale deployment, it is impossible to compile and install each server, as long as the compiled libtorrent. so libtorrent-rasterbar.so.7 files with bt. put the file py in the same directory and write another shell script.
Copy codeThe Code is as follows: lib = 'dirname $0'
Export LD_LIBRARY_PATH = $ LD_LIBRARY_PATH: $ lib
Python bt. py-d-f <seed File>-s <file storage path>-t <preparation time>-u <limit upload speed>

Usage:
First, generate a seed file on the source server.
Copy codeThe Code is as follows: python bt. py-m-p <file or folder to be released>-s <seed storage address>
Publish a file
On the source server, run
Copy codeThe Code is as follows: python bt. py-d-f <seed File>-s <file storage path>-t <start time>-u <speed limit>
The default setting time is 100 s, and the upload speed is unlimited by default. The maximum speed is KB.

The following machine can be used directly
Copy codeThe Code is as follows: python bt. py-d-f <seed File>-s <file storage path>-t <start time>

As long as one machine is complete, it will be automatically used as a seed and uploaded during the download process. Any machine can be used as the source server. Of course, there is also a central tracker server, in the script, I set up a tracker source server and put it on port 10.0.1.5, Which is 8760. Of course, you can also use the opentracker software to build a tracker server and modify the source code, in addition, considering that the release is a private file, the Code has disabled the dht. If you want to be safer, you can build a private tracker server by yourself. You can use the search engine to build the tracker server, find the build method!

At present, it is basically usable. It will be simpler to consider in the future and adopt the magnetic link method, so that you do not need to copy a seed file for each server, you can use a single command line to publish the entire file.