A p2p file transfer instance implemented by python

This article mainly introduces a p2p file transfer instance implemented by python. This article is used to solve the problem of file synchronization maintained by multiple servers. if you need a friend, please refer to the section below considering the increasing number of servers in my hands, sometimes you need to deploy the same file on a large scale. for example, you need to install the kernel-debuginfo package on hundreds of servers at the same time because it is convenient to use the systemtap tool to locate the problem, the original method uses a source server, and The rsync or scp file transmission method can only distribute this file at one point. at this time, the delivery speed will be relatively slow, for the above reasons, I wrote a tool for transferring files based on the bt protocol. in actual tests, the tool was transferred to 10 data centers, and more than 70 machines were transferred to a M kernel file, to all machines, the source uses a 2 Mbit/s speed limit for uploading. the test results can be transmitted as long as 140 seconds. This efficiency is very high, if the speed is not limited, the speed will be faster. The program is open-source below.

The code is as follows:

#! /Usr/bin/env python

Import libtorrent as lt
Import sys
Import OS
Import time
From optparse import OptionParser
Import socket
Import struct
Import fcntl

Def get_interface_ip (ifname ):
S = socket. socket (socket. AF_INET, socket. SOCK_DGRAM)
Return socket. inet_ntoa (fcntl. ioctl (s. fileno (), 0x8915, struct. pack ('256s ',
Ifname [: 15]) [20: 24])
Def ip2long (ip ):
Return reduce (lambda a, B :( a <8) + B, [int (I) for I in ip. split ('.')])


Def get_wan_ip_address ():
Interfaces = set (['eth0', 'eth1', 'eth2', 'eth3', 'em1', 'em2', 'em3', 'em4'])
Ip =''
For I in interfaces:
Try:
Ip = get_interface_ip (I)
If (ip2long (ip) <ip2long ('10. 0.0.0 ') or ip2long (ip)> ip2long ('10. 255.255.255 '))\
And (ip2long (ip) <ip2long ('192. 16.0.0 ') or ip2long (ip)> ip2long ('192. 33.0000255 '))\
And (ip2long (ip) <ip2long ('192. 168.0.0 ') or ip2long (ip)> ip2long ('192. 168.255.255 ')):
Return ip
Except t:
Pass

Return ip

Def make_torrent (path, save ):
Fs = lt. file_storage ()
Lt. add_files (fs, path)
If fs. num_files () = 0:
Print 'No files added'
Sys. exit (1)

Input = OS. path. abspath (path)
Basename = OS. path. basename (path)
T = lt. create_torrent (fs, 0, 4*1024*1024)

T. add_tracker ("http: // 10.0.1.5: 8760/announce ")
T. set_creator ('libtorrent % s' % lt. version)

Lt. set_piece_hashes (t, OS. path. split (input) [0], lambda x: sys. stderr. write ('.'))
Sys. stderr. write ('\ n ')

Save = OS. path. dirname (input)
Save = "% s/% s. torrent" % (save, basename)
F = open (save, "wb ")
F. write (lt. bencode (t. generate ()))
F. close ()
Print "the bt torrent file is store at % s" % save


Def dl_status (handle ):
While not (handle. is_seed ()):
S = handle. status ()

State_str = ['queued', 'checking', 'downloading metadata ',\
'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume']
Print '\ ractive_time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' % \
(S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000 ,\
S. num_peers, s. num_seeds, state_str [s. state]),
Sys. stdout. flush ()

Time. sleep (1)
Def seed_status (handle, seedtime = 100 ):
Seedtime = int (seedtime)
If seedtime: <100:
Seedtime= 100
While seedtime> 0:
Seedtime-= 1
S = handle. status ()

State_str = ['queued', 'checking', 'downloading metadata ',\
'Downloading', 'finished', 'secret', 'allocating', 'checking fastresume']
Print '\ rseed_time: % d, %. 2f % complete (down: %. 1f kb/s up: %. 1f kB/s peers: % d, seeds: % d) % s' % \
(S. active_time, s. progress * 100, s. download_rate/1000, s. upload_rate/1000 ,\
S. num_peers, s. num_seeds, state_str [s. state]),
Sys. stdout. flush ()

Time. sleep (1)

Def remove_torrents (torrent, session ):
Session. remove_torrent (torrent)

Def read_alerts (session ):
Alert = session. pop_alert ()
While alert:
# Print alert, alert. message ()
Alert = session. pop_alert ()

Def download (torrent, path, upload_rate_limit = 0, seedtime = 100 ):
Try:
Session = lt. session ()
Session. set_alert_queue_size_limit (1024*1024)

Sts = lt. session_settings ()
Sts. ssl_listen = False
Sts. user_agent = "Thunder deploy system"
Sts. tracker_completion_timeout = 5
Sts. tracker_receive_timeout = 5
Sts. stop_tracker_timeout = 5
Sts. active_downloads =-1
Sts. active_seeds =-1
Sts. active_limit =-1
Sts. auto_scrape_min_interval = 5
Sts. udp_tracker_token_expiry = 120
Sts. min_announce_interval = 1
Sts. inactivity_timeout = 60
Sts. connection_speed = 10
Sts. allow_multiple_connections_per_ip = True
Sts. max_out_request_queue = 128
Sts. request_queue_size = 3

Sts. use_read_cache = False
Session. set_settings (sts)

Session. set_alert_mask (lt. alert. category_t.tracker_notification | lt. alert. category_t.status_notification)
Session. set_alert_mask (lt. alert. category_t.status_notification)

Ipaddr = get_wan_ip_address ()
# Print ipaddr
If ipaddr = "":
Session. listen_on (6881,688 1)
Else:
Session. listen_on (6881,688 1, ipaddr)

Limit = int (upload_rate_limit)
If limit >=100:
Session. set_upload_rate_limit (limit * 1024)
Session. set_local_upload_rate_limit (limit * 1024)
Print session. upload_rate_limit ()
Torrent_info = lt. torrent_info (torrent)
Add_params = {
'SAVE _ path': path,
'Storage _ mode': lt. storage_mode_t.storage_mode_sparse,
'Paused': False,
'Auto _ managed': True,
'Ti': torrent_info,
}

Handle = session. add_torrent (add_params)

Read_alerts (session)
St = time. time ()
Dl_status (handle)
Et = time. time ()-st
Print '\ nall file download in %. 2f \ nstart to seeding \ n' % et
Sys. stdout. write ('\ n ')
Handle. super_seeding ()
Seed_status (handle, seedtime)

Remove_torrents (handle, session)
Assert len (session. get_torrents () = 0

Finally:
Print 'download finished'

If _ name _ = '_ main __':
Usage = "usage: % prog [options] \ n \
% Prog-d-f -S \ N \
Or \ n \
% Prog-m-p -S \ N"

Parser = OptionParser (usage = usage)
Parser. add_option ("-d", "-- download", dest = "download ",
Help = "start to download file", action = "store_false", default = True)
Parser. add_option ("-f", "-- file", dest = "file ",
Help = "torrent file ")
Parser. add_option ("-u", "-- upload", dest = "upload ",
Help = "set upload rate limit, default is not limit", default = 0)
Parser. add_option ("-t", "-- time", dest = "time ",
Help = "set seed time, default is 100 s", default = 100)
Parser. add_option ("-p", "-- path", dest = "path ",
Help = "to make torrent with this path ")
Parser. add_option ("-m", "-- make", dest = "make ",
Help = "make torrent", action = "store_false", default = True)
Parser. add_option ("-s", "-- save", dest = "save ",
Help = "file save path, default is store to./", default = "./")
(Options, args) = parser. parse_args ()
# Download (sys. argv [1])
If len (sys. argv )! = 6 and len (sys. argv )! = 4 and len (sys. argv )! = 8 and len (sys. argv )! = 10:
Parser. print_help ()
Sys. exit ()
If options. download = False and options. file! = "":
Download (options. file, options. save, options. upload, options. time)
Elif options. make = False and options. path! = "":
Make_torrent (options. path, options. save)

Prepare the environment:
Install a libtorrent Library on all operating systems ,:

Http://code.google.com/p/libtorrent/downloads/list

Remember to include./configure-enable-python-binding during compilation, then mak, make install, and enter the binding Directory. make and make install
You can run this small tool.
Of course, for large-scale deployment, it is impossible to compile and install each server, as long as the compiled libtorrent. so libtorrent-rasterbar.so.7 files with bt. put the file py in the same directory and write another shell script.

The code is as follows:

Lib = 'dirname $0'
Export LD_LIBRARY_PATH = $ LD_LIBRARY_PATH: $ lib
Python bt. py-d-f <种子文件> -S <文件保存路径> -T <做种时间> -U <限制上传速度>

Usage:
First, generate a seed file on the source server.

The code is as follows:

Python bt. py-m-p <要发布的文件或者文件夹> -S <种子保存地址>

Publish a file
On the source server, run

The code is as follows:

Python bt. py-d-f <种子文件> -S <文件保存路径> -T <做种时间> -U <限制上传速度>

The default setting time is 100 s, and the upload speed is unlimited by default. The maximum speed is KB.

The following machine can be used directly

The code is as follows:

Python bt. py-d-f <种子文件> -S <文件保存路径> -T <做种时间>

As long as one machine is complete, it will be automatically used as a seed and uploaded during the download process. any machine can be used as the source server. of course, there is also a central tracker server, in the script, I set up a tracker source server and put it on port 10.0.1.5, which is 8760. of course, you can also use the opentracker software to build a tracker server and modify the source code, in addition, considering that the release is a private file, the code has disabled the dht. if you want to be safer, you can build a private tracker server by yourself. you can use the search engine to build the tracker server, find the build method!

At present, it is basically usable. it will be simpler to consider in the future and adopt the magnetic link method, so that you do not need to copy a seed file for each server, you can use a single command line to publish the entire file.