Java multithreaded Breakpoint Download file

Last Update:2017-01-13 Source: Internet

Author: User

Developer on Alibaba Coud: Build your first app with APIs, SDKs, and tutorials on the Alibaba Cloud. Read more ＞

First, download the file information class, entity

Encapsulates information about the upcoming download of resources

The code is as follows

Copy Code

Package com.hoo.entity;

/**
* function: Download file information class
* @author Hoojo
* @createDate 2011-9-21 05:14:58
* @file Downloadinfo.java
* @package com.hoo.entity
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Downloadinfo {
Download file URL
Private String URL;
Download file name
Private String FileName;
Download file path
Private String FilePath;
How many downloads are divided into sections, each with one thread to complete the download
private int splitter;

Download file default save path
Private final static String File_path = "C:/temp";
Default chunking number, number of threads
Private final static int splitter_num = 5;

Public Downloadinfo () {
Super ();
}

/**
* @param URL Download address
*/
Public downloadinfo (String URL) {
This (URL, null, NULL, splitter_num);
}

/**
* @param url Download address URL
* How many segments or how many threads @param splitter to download
*/
Public downloadinfo (String URL, int splitter) {
This (URL, null, NULL, splitter);
}

/***
* @param URL Download address
* @param filename File name
* @param filePath File Save path
* How many segments or how many threads @param splitter to download
*/
Public downloadinfo (string URL, String fileName, string filePath, int splitter) {
Super ();
if (url = null | | "". Equals (URL)) {
throw new RuntimeException ("URL is not null!");
}
This.url = URL;
This.filename = (FileName = = NULL | | ". Equals (FileName))? GetFileName (URL): fileName;
This.filepath = (FilePath = = NULL | | ". Equals (FilePath)"? File_path:filepath;
This.splitter = (Splitter < 1)? Splitter_num:splitter;
}

/**
* function: get file name via URL
* @author Hoojo
* @createDate 2011-9-30 05:00:00
* @param URL
* @return
*/
private string GetFileName (string url) {
Return url.substring (Url.lastindexof ("/") + 1, url.length ());
}

Public String GetUrl () {
return URL;
}

public void SetUrl (String URL) {
if (url = null | | "". Equals (URL)) {
throw new RuntimeException ("URL is not null!");
}
This.url = URL;
}

Public String GetFileName () {
return fileName;
}

public void Setfilename (String fileName) {
This.filename = (FileName = = NULL | | ". Equals (FileName))? GetFileName (URL): fileName;
}

Public String GetFilePath () {
return filePath;
}

public void SetFilePath (String filePath) {
This.filepath = (FilePath = = NULL | | ". Equals (FilePath)"? File_path:filepath;
}

public int Getsplitter () {
return splitter;
}

public void Setsplitter (int splitter) {
This.splitter = (Splitter < 1)? Splitter_num:splitter;
}

@Override
Public String toString () {
return This.url + "#" + This.filename + "#" + This.filepath + "#" + this.splitter;
}
}

Second, randomly write a document

Package com.hoo.download;

Import java.io.IOException;
Import Java.io.RandomAccessFile;

/**
* function: write files, save files
* @author Hoojo
* @createDate 2011-9-21 05:44:02
* @file Saveitemfile.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Saveitemfile {
Storing files
Private Randomaccessfile Itemfile;

Public Saveitemfile () throws IOException {
This ("", 0);
}

/**
* @param name file paths, names
* @param POS Write point location position
* @throws IOException
*/
Public Saveitemfile (String name, long Pos) throws IOException {
Itemfile = new Randomaccessfile (name, rw);
Start writing data at the specified POS location
Itemfile.seek (POS);
}

/**
* function: Sync Method Write file
* @author Hoojo
* @createDate 2011-9-26 12:21:22
* @param buff Buffer array
* @param start position
* @param length
* @return
*/
public synchronized int write (byte[] buff, int start, int length) {
int i =-1;
try {
Itemfile.write (buff, start, length);
i = length;
catch (IOException e) {
E.printstacktrace ();
}
return i;
}

public void Close () throws IOException {
if (itemfile!= null) {
Itemfile.close ();
}
}
}

This class basically completes the write file to the local specified file pointer, and returns the length of the current write file (the file pointer). This class will be called by the thread, and the file will be called by the thread after it is divided into the corresponding block. Each thread will call this class to complete the random write of the file.

Three, a single thread download files

The code is as follows

Copy Code

Package com.hoo.download;

Import java.io.IOException;
Import Java.io.InputStream;
Import java.net.HttpURLConnection;
Import java.net.MalformedURLException;
Import Java.net.URL;
Import java.net.URLConnection;
Import Com.hoo.util.LogUtils;

/**
* function: single-threaded download files
* @author Hoojo
* @createDate 2011-9-22 02:55:10
* @file Downloadfile.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class DownloadFile extends Thread {

Download file URL
Private String URL;
Download file start location
Private long startpos;
Download file End Location
Private long Endpos;
Thread ID
private int threadId;

Whether the download is complete
Private Boolean isdownloadover = false;

Private Saveitemfile Itemfile;

private static final int buff_length = 1024 * 8;

/**
* @param url Download file URL
* @param name File
* @param startpos download file starting point
* @param endpos download file end point
* @param threadId Thread ID
* @throws IOException
*/
Public DownloadFile (string url, string name, long startpos, long endpos, int threadId) throws IOException {
Super ();
This.url = URL;
This.startpos = startpos;
This.endpos = Endpos;
This.threadid = threadId;
Block Download Write file contents
This.itemfile = new Saveitemfile (name, startpos);
}

@Override
public void Run () {
while (Endpos > Startpos &&!isdownloadover) {
try {
URL url = new URL (this.url);
HttpURLConnection conn = (httpurlconnection) url.openconnection ();

Set the connection timeout to 10000ms
Conn.setconnecttimeout (10000);
Sets the read data timeout of 10000ms
Conn.setreadtimeout (10000);

SetHeader (conn);

String property = "bytes=" + startpos + "-";
Conn.setrequestproperty ("RANGE", property);

Output log Information
LogUtils.log ("Start" + threadId + ":" + property + Endpos);
Printheader (conn);

Get file input stream, read file contents
InputStream is = Conn.getinputstream ();

byte[] buff = new Byte[buff_length];
int length =-1;
LogUtils.log ("#start #thread:" + threadId + ", startpos:" + startpos + ", Endpos:" + endpos);
while (length = Is.read (buff)) > 0 && startpos < endpos &&!isdownloadover) {
Writes the contents of the file, returns the length of the last write
Startpos + + itemfile.write (buff, 0, length);
}
LogUtils.log ("#over #thread:" + threadId + ", startpos:" + startpos + ", Endpos:" + endpos);
LogUtils.log ("Thread" + ThreadId + "is execute over!");
This.isdownloadover = true;
catch (Malformedurlexception e) {
E.printstacktrace ();
catch (IOException e) {
E.printstacktrace ();
finally {
try {
if (itemfile!= null) {
Itemfile.close ();
}
catch (IOException e) {
E.printstacktrace ();
}
}
}
if (Endpos < startpos &&!isdownloadover) {
LogUtils.log ("Thread" + threadId + "Startpos > Endpos, not need download file!");
This.isdownloadover = true;
}
if (Endpos = = Startpos &&!isdownloadover) {
LogUtils.log ("Thread" + threadId + "startpos = endpos, not need download file!");
This.isdownloadover = true;
}
}

/**
* function: Print Download file header information
* @author Hoojo
* @createDate 2011-9-22 05:44:35
* @param Conn HttpURLConnection
*/
public static void Printheader (URLConnection conn) {
int i = 1;
while (true) {
String Header = Conn.getheaderfieldkey (i);
i++;
if (header!= null) {
Logutils.info (header + ":" + Conn.getheaderfield (i));
} else {
Break
}
}
}

/**
* function: Set URLConnection header information, disguise request information
* @author Hoojo
* @createDate 2011-9-28 05:29:43
* @param con
*/
public static void SetHeader (URLConnection conn) {
Conn.setrequestproperty ("User-agent", "mozilla/5.0" (X11; U Linux i686; En-us; rv:1.9.0.3) gecko/2008092510 ubuntu/8.04 (Hardy) firefox/3.0.3 ");
Conn.setrequestproperty ("Accept-language", "en-us,en;q=0.7,zh-cn;q=0.3");
Conn.setrequestproperty ("accept-encoding", "utf-8");
Conn.setrequestproperty ("Accept-charset", "iso-8859-1,utf-8;q=0.7,*;q=0.7");
Conn.setrequestproperty ("Keep-alive", "300");
Conn.setrequestproperty ("Connnection", "keep-alive");
Conn.setrequestproperty ("If-modified-since", "Fri, 2009 17:00:05 GMT");
Conn.setrequestproperty ("If-none-match", "" 1261d8-4290-df64d224 ");
Conn.setrequestproperty ("Cache-conntrol", "max-age=0");
Conn.setrequestproperty ("Referer", "http://www.baidu.com");
}

public Boolean isdownloadover () {
return isdownloadover;
}

Public long Getstartpos () {
return startpos;
}

Public long Getendpos () {
return endpos;
}
}

This class is primarily complete with a single thread file download, and will read the resource information for the specified URL through URLConnection. It then reads the contents of the file with InputStream, and then invokes the call to the Saveitemfile class to write locally the contents of the block currently being read.

Four, segmented multithreaded write file content

The code is as follows

Copy Code

Package com.hoo.download;

Import Java.io.DataInputStream;
Import Java.io.DataOutputStream;
Import Java.io.File;
Import Java.io.FileInputStream;
Import Java.io.FileOutputStream;
Import java.io.IOException;
Import java.net.HttpURLConnection;
Import java.net.MalformedURLException;
Import Java.net.URL;
Import Com.hoo.entity.DownloadInfo;
Import Com.hoo.util.LogUtils;

/**
* function: Bulk Download files
* @author Hoojo
* @createDate 2011-9-22 05:51:54
* @file Batchdownloadfile.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Batchdownloadfile implements Runnable {
Download file information
Private Downloadinfo Downloadinfo;
A set of start download locations
Private long[] startpos;
A set of end download Locations
Private long[] Endpos;
Sleep time
private static final int sleep_seconds = 500;
Child thread Download
Private downloadfile[] Fileitem;
File length
private int length;
is the first file
Private Boolean-i = true;
Whether to stop downloading
Private Boolean stop = false;
Temporary file information
Private File tempfile;

Public Batchdownloadfile (Downloadinfo downloadinfo) {
This.downloadinfo = Downloadinfo;
String TempPath = This.downloadInfo.getFilePath () + File.separator + downloadinfo.getfilename () + ". Position";
Tempfile = new File (TempPath);
If a file with a read in point location exists
if (tempfile.exists ()) {
i = false;
Just read the contents directly
try {
Readposinfo ();
catch (IOException e) {
E.printstacktrace ();
}
} else {
The length of the array is divided into the number of segments
startpos = new Long[downloadinfo.getsplitter ()];
Endpos = new Long[downloadinfo.getsplitter ()];
}
}

@Override
public void Run () {
First download, get download file length
if (a) {
Length = This.getfilesize ();//Get file Lengths
if (length = = 1) {
LogUtils.log ("File length is know!");
Stop = true;
else if (length = = 2) {
LogUtils.log ("Read file length is error!");
Stop = true;
else if (length > 0) {
/**
* EG
* Start:1, 3, 5, 7, 9
* End:3, 5, 7, 9, length
*/
for (int i = 0, len = startpos.length i < len; i++) {
int size = i * (Length/len);
Startpos[i] = size;

Set the position of the last ending point
if (i = = len-1) {
Endpos[i] = length;
} else {
Size = (i + 1) * (Length/len);
Endpos[i] = size;
}
LogUtils.log ("Start-end position[" + i + "]:" + startpos[i] + "-" + endpos[i]);
}
} else {
LogUtils.log ("Get file length was error, download is stop!");
Stop = true;
}
}

Child Threads Start downloading
if (!stop) {
Create a single-threaded Download object array
Fileitem = new Downloadfile[startpos.length];//startpos.length = Downloadinfo.getsplitter ()
for (int i = 0; i < startpos.length; i++) {
try {
Creates a specified number of single-threaded download objects, each of which completes the download of the specified block content independently
Fileitem[i] = new DownloadFile (
Downloadinfo.geturl (),
This.downloadInfo.getFilePath () + File.separator + downloadinfo.getfilename (),
Startpos[i], endpos[i], I
);
Fileitem[i].start ()//Start thread, start download
LogUtils.log ("Thread: + i +", startpos: "+ startpos[i] +", Endpos: "+ endpos[i]";
catch (IOException e) {
E.printstacktrace ();
}
}

Looping write download file length information
while (!stop) {
try {
Writeposinfo ();
LogUtils.log ("Downloading ...");
Thread.Sleep (Sleep_seconds);
Stop = true;
catch (IOException e) {
E.printstacktrace ();
catch (Interruptedexception e) {
E.printstacktrace ();
}
for (int i = 0; i < startpos.length; i++) {
if (!fileitem[i].isdownloadover ()) {
stop = false;
Break
}
}
}
Logutils.info ("Download task is finished!");
}
}

/**
* Save the write point data in a temporary file
* @author Hoojo
* @createDate 2011-9-23 05:25:37
* @throws IOException
*/
private void Writeposinfo () throws IOException {
DataOutputStream dos = new DataOutputStream (new FileOutputStream (tempfile));
Dos.writeint (startpos.length);
for (int i = 0; i < startpos.length; i++) {
Dos.writelong (Fileitem[i].getstartpos ());
Dos.writelong (Fileitem[i].getendpos ());
Logutils.info ("[" + fileitem[i].getstartpos () + "#" + fileitem[i].getendpos () + "]");
}
Dos.close ();
}

/**
* function: read location information for write point
* @author Hoojo
* @createDate 2011-9-23 05:30:29
* @throws IOException
*/
private void Readposinfo () throws IOException {
DataInputStream dis = new DataInputStream (new FileInputStream (tempfile));
int startposlength = Dis.readint ();
startpos = new Long[startposlength];
Endpos = new Long[startposlength];
for (int i = 0; i < startposlength; i++) {
Startpos[i] = Dis.readlong ();
Endpos[i] = Dis.readlong ();
}
Dis.close ();
}

/**
* function: Get the length of the download file
* @author Hoojo
* @createDate 2011-9-26 12:15:08
* @return
*/
private int GetFileSize () {
int filelength =-1;
try {
URL url = new URL (this.downloadInfo.getUrl ());
HttpURLConnection conn = (httpurlconnection) url.openconnection ();

Downloadfile.setheader (conn);

int statecode = Conn.getresponsecode ();
Determine if HTTP status is http/1.1 206 Partial content or OK
if (statecode!= httpurlconnection.http_ok && statecode!= httpurlconnection.http_partial) {
LogUtils.log ("Error Code:" + statecode);
Return-2;
else if (Statecode >= 400) {
LogUtils.log ("Error Code:" + statecode);
Return-2;
} else {
Get length
Filelength = Conn.getcontentlength ();
LogUtils.log ("filelength:" + filelength);
}

Read file length
/*for (int i = 1;; i++) {
String Header = Conn.getheaderfieldkey (i);
if (header!= null) {
if ("Content-length". Equals (header)) {
Filelength = Integer.parseint (Conn.getheaderfield (i));
Break
}
} else {
Break
}
}
*/

Downloadfile.printheader (conn);
catch (Malformedurlexception e) {
E.printstacktrace ();
catch (IOException e) {
E.printstacktrace ();
}
return filelength;
}
}

This class basically completes reading the contents of the specified URL resource and gets the length of the resource. The resource is then divided into the specified number of blocks, and the starting download location and the ending download location of each block are stored in an array. Each piece opens up a separate thread to start the download. Before starting the download, you need to create a temporary file that writes to the current download thread's start download pointer position and the end download pointer location.

V. Tools, test Class

Log Tool class

The code is as follows

Copy Code

Package com.hoo.util;

/**
* function: Logging Tool class
* @author Hoojo
* @createDate 2011-9-21 05:21:27
* @file Logutils.java
* @package Com.hoo.util
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
Public abstract class Logutils {

public static void log (Object message) {
SYSTEM.ERR.PRINTLN (message);
}

public static void log (String message) {
SYSTEM.ERR.PRINTLN (message);
}

public static void log (int message) {
SYSTEM.ERR.PRINTLN (message);
}

public static void info (Object message) {
SYSTEM.OUT.PRINTLN (message);
}

public static void info (String message) {
SYSTEM.OUT.PRINTLN (message);
}

public static void info (int message) {
SYSTEM.OUT.PRINTLN (message);
}
}

Download Tool class
Package com.hoo.util;

Import Com.hoo.download.BatchDownloadFile;
Import Com.hoo.entity.DownloadInfo;

/**
* function: block Multi-threaded Download tool class
* @author Hoojo
* @createDate 2011-9-28 05:22:18
* @file Downloadutils.java
* @package Com.hoo.util
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
Public abstract class Downloadutils {

public static void Download (String URL) {
Downloadinfo bean = new Downloadinfo (URL);
Logutils.info (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
}

public static void Download (String url, int threadnum) {
Downloadinfo bean = new Downloadinfo (URL, threadnum);
Logutils.info (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
}

public static void Download (string url, String fileName, string filePath, int threadnum) {
Downloadinfo bean = new Downloadinfo (URL, fileName, FilePath, threadnum);
Logutils.info (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
}
}

Download Test class

The code is as follows

Copy Code

Package com.hoo.test;

Import Com.hoo.util.DownloadUtils;

/**
* function: Download Test
* @author Hoojo
* @createDate 2011-9-23 05:49:46
* @file Testdownloadmain.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.111cn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Testdownloadmain {

public static void Main (string[] args) {
/*downloadinfo bean = new Downloadinfo ("yun_qi_img/2011092116224363.jpg");
System.out.println (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();

Downloadutils.download ("yun_qi_img/2011092116224363.jpg");
Downloadutils.download ("http://mp3.baidu.com/j?j=2&url=http%3A%2F%2Fzhangmenshiting2.baidu.com%2Fdata% 2fmusic%2f1669425%2f%25e9%2599%25b7%25e5%2585%25a5%25e7%2588%25b1%25e9%2587%258c%25e9%259d%25a2.mp3%3fxcode% 3d2ff36fb70737c816553396c56deab3f1 "," Aa.mp3 "," C:/temp ", 5);
}
}

Multi-threaded downloads are mainly in the third and fourth parts, and others are well understood. The source code to provide the corresponding comments

This article is an English version of an article which is originally in the Chinese language on aliyun.com and is provided for information purposes only. This website makes no representation or warranty of any kind, either expressed or implied, as to the accuracy, completeness ownership or reliability of the article or any translations thereof. If you have any concerns or complaints relating to the article, please send an email, providing a detailed description of the concern or complaint, to info-contact@alibabacloud.com. A staff member will contact you within 5 working days. Once verified, infringing content will be removed immediately.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

Get Started for Free

Sales Support

1 on 1 presale consultation

Chat Contact Sales
After-Sales Support

24/7 Technical Support 6 Free Tickets per Quarter Faster Response

Open a Ticket
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.

Learn More

Java multithreaded Breakpoint Download file

Contact Us

What's Trending

Top 10 Tags

Top 10 Keywords

A Free Trial That Lets You Build Big!

Sales Support

After-Sales Support

Java multithreaded Breakpoint Download file

Contact Us

What's Trending

Top 10 Tags

Top 10 Keywords

Trending Topic

A Free Trial That Lets You Build Big!

Sales Support

After-Sales Support