Java-based multi-thread download source code analysis (1)

This article implements a Java-based multi-thread download device, which provides the following functions:

1. Use multiple threads to download files and display the download speed at each time point.
2. manage multiple downloads, including Thread Scheduling and memory management.

The structure of this article is as follows: first, we will discuss how to use Java multithreading to download a single file. Then, we will discuss how to manage multiple file downloads in the system. Including Thread Scheduling and memory management.

I. Management of single file downloads 1. Single file download class level

First, we will briefly introduce the class levels of a single file download management:

To a graph.

 

1. Create a download class for the file to be downloaded. Download is responsible for managing the thread management, file management, and current speed calculation operations when the file is downloaded.
2. Based on the number of threads, this file is divided into tnum segments, each of which is a downloadblock. In the actual download process, instead of downloading all the items at a time, you can download a fixed size di each time. Therefore, each downloadblock is divided into N segments.
3. Apply for a thread downloadthread for each downloadblock. Its main function is to download a piece of di each time and write it into the file.

2. Single file download

For a single download, follow these steps:

1. Connect to the resource server, obtain resource information, and create a file
2. Split resources and assign a Fixed DOWNLOAD area to each thread.

1) encapsulate download attributes

Each download is abstracted and encapsulated before the download is established.

First, encapsulate the URL and target file in a downloadconfig class.

It contains four attributes:

Private URL; // file private file; // download the file to save the target file private int nthread; // Number of threads required to download the file private int priority; // download priority

As shown in the following figure:

2) connect to the resource server, obtain resource information, create a file, and specify the file size

length = config.getUrl().openConnection().getContentLength();RandomAccessFile file = new RandomAccessFile(config.getFile(), "rw");file.setLength(length);file.close();

3) Split resources, allocate a fixed download area for each thread, and add the current download to the queue.

Int size = length/config. getnthread (); For (INT I = 0; I <config. getnthread (); I ++) {int start = I * size; int Len; if (I = config. getnthread ()-1) Len = length-start; else Len = size; // Add the current download to the download queue adddownloadblock (getdownloadblock (START, Len ));}

3) Start the thread to download

The download procedure is as follows:

1. Create a cache and create a connection. Set the range for obtaining resource data, create a file, and set the write location

// Create the cache byte [] B; If (Block. getlength () <constants. bytes_read) B = new byte [(INT) block. getlength ()]; elseb = new byte [constants. bytes_read]; // create a connection. Sets the range for obtaining resource data, from startpos to endposurlconnection con = NULL; con. setrequestproperty ("range", "bytes =" + block. getstart () + "-" + block. getstart () + block. getlength ()-1); randomaccessfile file = new randomaccessfile (Block. getdownload (). getconfig (). getFile (), "RW"); // create randomaccessfilefile. seek (Block. getstart (); // write data from startpos

2. if the length of the current block is greater than 0, get a fixed size resource from the URL resource and write it into the file.

3. Update the start and length of the block. If the length is greater than 0, continue with Step 2. Otherwise, the current block has been downloaded and exited the thread.

Inputstream in = block. getdownload (). getconfig (). geturl (). openstream (); int N; // download the files in the block, while (count <block. getlength () {If (needsplit () {// check whether the block still needs to be segmented (that is, the remaining size of the current block is greater than the number of downloads at a time) Long newlength = (Block. getlength ()-count)/2; long Newstart = block. getstart () + block. getlength ()-newlength; downloadblock newblock = block. getdownload (). getdownloadblock (Newstart, newlength); block. setlength ( Block. getlength ()-newlength); block. getdownload (). adddownloadblock (newblock);} // Write File n = in. read (B); If (n <0) {break;} else if (count + n> block. getlength () {file. write (B, 0, (INT) (Block. getlength ()-count); Count = block. getlength () ;}else {count + = N; file. write (B, 0, n) ;}// set block count in downloadif (n> 0) {// count the number of downloaded segments in each block, used to calculate the current download speed. Block. getdownload (). setblockcount (Block. getstart (), count) ;}} in. Close (); file. Close ();

Ii. Calculation of the current file download speed and progress

As shown in the first figure, each block is divided into many segments D1, D2 ,... Therefore, to calculate the current download speed, you need to count the number of downloaded segments D. A concurrenthashmap <long, long> to save the number of segments D that have been downloaded for each block. The key is the start value of each block, and the value is the downloaded section D of the block.
At the current time point, we need to count the number of segments D that have been downloaded by the current download, and then compare it with the previous time point to get the current download speed. For specific code, see:

Class checkspeedtask extends timertask {Private Static final log = logfactory. getlog (checkspeedtask. class); Private download; private concurrenthashmap <long, long> blockcounts; private long speed = 0; // byte/sprivate Long Count = 0; // total downloaded byte countprivate long lastcount = 0; private long time = 0; // check timeprivate long lasttime = 0; Public checkspeedtask (download downloa D, long starttime, concurrenthashmap <long, long> blockcounts) {This. download = download; this. lasttime = starttime; this. blockcounts = blockcounts;} @ overridepublic void run () {try {time = system. currenttimemillis (); Count = 0; // count the number of D segments that have been downloaded. For (long C: blockcounts. values () {count + = C;} speed = (count-lastcount)/(time-lasttime)/1000); log. debug (blockcounts. size () + "threads are downloading" + download + ", cuttent is" + speed + "Byte/s," + (count * 1.0)/download. getlength () * 100 + "% downloaded"); download. setcount (count); download. setspeed (speed); lasttime = time; lastcount = count;} catch (exception e) {// todo: handle takes tione. printstacktrace ();}}}

In this way, we can calculate the current download speed in the run () function of the thread class.

while(activeThreads.size() > 0 || blockQueue.size() > 0){Thread.sleep(1000);checkSpeed();}

The code above demonstrates how to use Java multithreading to download a single file. Next we will continue to discuss how to schedule and manage multiple downloads.