Setting up a private peer-to transmission network

June went to Shanghai to participate in the Dockone organization of the container assembly, reported a small project, accidentally won the first prize, here to share with you the core of the network of the building, is also the company has put into use the large file transfer mode .

project Design :

Original intention:

1, as a provider of global cloud computing business, whether it is the expansion of its own overseas node business, platform expansion or customer custom template business needs, will involve the transmission of large files. Some common ISO system image files, some custom-made system templates, less more than 10 g, more than dozens of G. Such a large number of files in the global transmission between multiple nodes, how to ensure its stable, fast implementation, the most important is the integrity of the requirements of the document is our objective face of the problem.

2, traditional FTP, such as rsync file transfer can not meet the transmission requirements of large files. We often encounter a hard day to transfer the template to a data center node, and the results tell the file is corrupted and cannot be deployed. Reliability is not guaranteed.

3, the customer urgently needs to realize the large files between the various nodes of the world fast, reliable distribution, synchronization

Implementation ideas:

1. Stable and fast: the stability of large file distribution is limited by the transmission network, which is limited by the transmission protocol. Our distribution network is based on the company's dedicated GPN, the stability of the transmission network is guaranteed, the transmission protocol used by the peer transmission, is obviously better than the FTP and other related protocols.

In fact, we can achieve this through public network and private networks.

2, Development: Peer (BT) related software, the tool community has a lot of molded products, so we do not need to develop their own. Just choose the right open source tool for two development. In the era of cloud computing, development efficiency is the key.

3. Deployment delivery: Distribution network Docker containerized deployment, mirroring delivery. Docker-based lightweight, cross-platform features, standard, unified packaging solutions to generate deliverables for mirrored products. With a good REST API, it is ideal for automated testing and continuous integration, and for the dynamic expansion of distribution networks.

Project implementation:

1, Network: File distribution network built in our company's proprietary GPN global Internet (blog does not play ads, the actual public network and its own intranet can also be achieved, but pay attention to speed limit!) ).

2, Tracker: The use of centralized peer-to transmission mode, tracker using Peertracker. Peertracker is a simple, efficient and fast BT Tracker. Responsible for recording the information of each peer, can build multiple tracker to do redundancy, prevent a tracker outage cause the cluster not to be available.

3, Peer: the use of transmission, is a cross-platform free software, with a web interface, support API calls, programmability is good.

4, Agent: Based on flask + DOCKERAPI development, encapsulated a set of DOCKERAPI related operations, provide authentication mechanism, receive central control node scheduling, create peer Docker container, perform related tasks. The deployment level is consistent with the transmission peer side.

5, Control node: based on Django + celery development. can monitor the entire distribution network operation, distribution task execution, dynamic expansion of distribution nodes, issued tasks and other operations.

The entire schematic and flowchart are as follows:

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M00/82/DA/wKiom1dihabg1VTZAAG18Dl1t7c702.jpg-wh_500x0-wm_3 -wmp_4-s_189961255.jpg "title=" flowchart. jpg "alt=" wkiom1dihabg1vtzaag18dl1t7c702.jpg-wh_50 "/>

650) this.width=650; "Src=" Http://s2.51cto.com/wyfs02/M00/82/D9/wKioL1dihuGAPjDkAAEhhRMUucY630.png-wh_500x0-wm_3 -wmp_4-s_3850393846.png "title=" peer network map 1.png "alt=" Wkiol1dihugapjdkaaehhrmuucy630.png-wh_50 "/>

Here we share a playbook, which is used to build up the peer network, and then share it with Docker and the implementation code section.

GitHub's address:

Https://github.com/xujpxm/ansible-peertracker

Due to the relatively hasty writing, there is a post-improvement, tested this script on the UBUNTU14 to work properly.

Readme:

This is a script that uses ansible to build a network of peers. Tracker adoption of Google 2010 Open source peertracker,p2p seed generation and transmission using transmission.

installation

Install Peertracker:

Ansible-playbook install_peertracker.yml

Database created by default: Peertracker, user name: Peertracker, Password: Yais6yab

Once the script is complete, create a database from the Web page, as described in the Peertracker installation below.

Install Transmission:

Ansible-playbook install_transmission.yml

Default Rpc_username and Password: transmission/transmission

You can modify the/etc/transmission-daemon/settings.json file customization yourself.

Create a download directory

Ansible-playbook mk_download_dir.yml

The default download path is:/data/transmission-daemon/downloads can change itself

Peertracker

P Eertracker is a simple, efficient and rapid bittorent Tracker.
Installation Requirements:
1. HTTP Web server.apache,nginx,lighttpd can be used as long as PHP is supported.
2. php5+ recommended php5.3 or later
3. Database. The Web server that supports MySQL, SQLite3, PostgreSQL8.0, and txtSQL scripts is apache2, which uses MySQL.

Important Links:

Development Website:http://code.google.com/p/peertracker/
Issue Tracker:http://code.google.com/p/peertracker/issues/list
Source Code Repository:http://peertracker.googlecode.com/svn/trunk/
GitHub respository:Https://github.com/JonnyJD/peertracker

Installation of Peertracker

Script a installation environment: Ubuntu
1. After confirming that the code is copied to the Web site root, the browser accesses: 2. Edit the configuration file tracker.mysql.php, which has been edited well in advance, mainly changed the database Db_user, Db_pass and db_name, the default database and user is peertracker, password: Yais6yab, You can also modify the files in the templates yourself.

2. Install tracker Database via the utilities provided on the help page. After the installation is successful, the database structure is simple, with only two tables: Pt_peers and Pt_tasks, recording peer and task information separately.

4. after the database is successfully created, it is actually ready to use, and the tracker URL is: http://serverip/peertracker/mysql/announce.php If you are using an extranet, you can move the corresponding announce and scrape.php files to the top-level directory of the site, and the resulting tracker URL is: http://serverip/announce.php

6. Examples of help.php pages:

You can also view the status of peer:

Http://serverip/peertracker/mysql/scrape.php?stats

Transmission

Transmission is a powerful BitTorrent open source client that implements most of the features described in the BT protocol. Currently, several versions are available for download on its official website, including: Mac, GTK +, qt version, and daemon version.
Transmission supports features such as DHT, Magnet Link, UTP, and PEX. In particular, support magnet link download is very important, because many resources on the Internet are used in this way to share, rather than traditional. torrent files, such as TPB, Btdigg.

installation/configuration of the transmission

Installation
Installation is simple, directly apt-get installation, run transmission role, the script will be installed automatically.
Installation process will be TRANSMISSION-CLI also installed, after the installation of the system will be more than the following command line tools.

• TRANSMISSION-CLI: Stand-alone command-line client.
  

• Transmission-create: The command-line tool used to build the. torrent seed file.
  

• Transmission-daemon: Daemon daemon.
  

• Transmission-edit: The announce URL used to modify the. torrent seed file.
  

• Transmission-remote: The program that controls daemon.
  

• Transmission-show: View information about the. torrent file.
  Configuration
  Settings.json is the primary configuration file that sets daemon parameters, including the RPC user name password configuration. It is actually a symbolic link that points to the original file that is/etc/transmission-daemon/settings.json.
  The script provides a template file, mainly modifies the download path, RPC authentication and whitelist, the default RPC authentication username/password is: transmission/transmission
  Several configuration parameters for RPC:
  

• RPC-AUTHENTICATION-REQUIRED:RPC certification, recommended to open
  

• Rpc-bind-address:string (default = "0.0.0.0") Where to listen for RPC connections
  

• Rpc-enabled:boolean (default = True)

• Rpc-password:string
  

• Rpc-port: Default port 9091Number (defaults = 9091)
  

• Rpc-url:string (default =/transmission/. Added in v2.2)
  

• Rpc-username:string
  

• Rpc-whitelist: Sets the whitelist String (comma-delimited list of IP addresses. Wildcards allowed using '. Example: "127.0.0.,192.168..", Default: "127.0.0.1")
  

• Rpc-whitelist-enabled:boolean (default = True)
  

transmission using Web Interface controls

browser access: seed file can be generated using the transmission transmission-remote command, for example:
transmission-create-c "Standard Template "-T Http://192.168.1.1/announce.php-o template.torrent standardtemplate

It's my own test. Transmission real-time speed, speed limit 100M, up to five peer basic full bandwidth:

Note that the transmission download path is/data/transmission-daemon/downloads, note that the file directory exists and permissions, can be created with mk_download_dir.yml script

Transmission-create the permission to create a seed file is 600 and needs to be changed to 644 to make the other readable

This article is from "Jason's blog" blog, please be sure to keep this source http://xujunxian.blog.51cto.com/8614409/1790037

Setting up a private peer-to transmission network