Using Java to implement HTTP multithreaded breakpoints download file (i) _java

Source: Internet
Author: User

Rationale: Use URLConnection to obtain information about the length, headers, etc. of the file to be downloaded, and to set the header information for the response. And through the urlconnection to get the input stream, the file into the specified block, each piece of separate open a thread to complete the data read, write. Reads the downloaded file information through the input stream, and then writes the read information to the local file Randomaccessfile randomly. At the same time, each thread writes data to the file pointer, which is the length of the data being written, and needs to be saved in a temporary file. So when the download is not completed, the next time you download the file from this file to read the last download length, and then continue to the previous location to start the download. and write the length of this download to this file.

First, download the file information class, entity
Encapsulates information about the upcoming download of resources

Copy Code code as follows:

Package com.hoo.entity;
/**
* <b>function:</b> Download file information class
* @author Hoojo
* @createDate 2011-9-21 05:14:58
* @file Downloadinfo.java
* @package com.hoo.entity
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Downloadinfo {
Download file URL
Private String URL;
Download file name
Private String FileName;
Download file path
Private String FilePath;
How many downloads are divided into sections, each with one thread to complete the download
private int splitter;
Download file default save path
Private final static String File_path = "C:/temp";
Default chunking number, number of threads
Private final static int splitter_num = 5;
Public Downloadinfo () {
Super ();
}
/**
* @param URL Download address
*/
Public downloadinfo (String URL) {
This (URL, null, NULL, splitter_num);
}
/**
* @param url Download address URL
* How many segments or how many threads @param splitter to download
*/
Public downloadinfo (String URL, int splitter) {
This (URL, null, NULL, splitter);
}
/***
* @param URL Download address
* @param filename File name
* @param filePath File Save path
* How many segments or how many threads @param splitter to download
*/
Public downloadinfo (string URL, String fileName, string filePath, int splitter) {
Super ();
if (url = null | | "". Equals (URL)) {
throw new RuntimeException ("URL is not null!");
}
This.url = URL;
This.filename = (FileName = = NULL | | ". Equals (FileName))? GetFileName (URL): fileName;
This.filepath = (FilePath = = NULL | | ". Equals (FilePath)"? File_path:filepath;
This.splitter = (Splitter < 1)? Splitter_num:splitter;
}
/**
* <b>function:</b> get file name via URL
* @author Hoojo
* @createDate 2011-9-30 05:00:00
* @param URL
* @return
*/
private string GetFileName (string url) {
Return url.substring (Url.lastindexof ("/") + 1, url.length ());
}
Public String GetUrl () {
return URL;
}
public void SetUrl (String URL) {
if (url = null | | "". Equals (URL)) {
throw new RuntimeException ("URL is not null!");
}
This.url = URL;
}
Public String GetFileName () {
return fileName;
}
public void Setfilename (String fileName) {
This.filename = (FileName = = NULL | | ". Equals (FileName))? GetFileName (URL): fileName;
}
Public String GetFilePath () {
return filePath;
}
public void SetFilePath (String filePath) {
This.filepath = (FilePath = = NULL | | ". Equals (FilePath)"? File_path:filepath;
}
public int Getsplitter () {
return splitter;
}
public void Setsplitter (int splitter) {
This.splitter = (Splitter < 1)? Splitter_num:splitter;
}
@Override
Public String toString () {
return This.url + "#" + This.filename + "#" + This.filepath + "#" + this.splitter;
}
}

second, randomly write a document
Copy Code code as follows:

Package com.hoo.download;
Import java.io.IOException;
Import Java.io.RandomAccessFile;
/**
* <b>function:</b> write files, save files
* @author Hoojo
* @createDate 2011-9-21 05:44:02
* @file Saveitemfile.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Saveitemfile {
Storing files
Private Randomaccessfile Itemfile;
Public Saveitemfile () throws IOException {
This ("", 0);
}
/**
* @param name file paths, names
* @param POS Write point location position
* @throws IOException
*/
Public Saveitemfile (String name, long Pos) throws IOException {
Itemfile = new Randomaccessfile (name, rw);
Start writing data at the specified POS location
Itemfile.seek (POS);
}
/**
* <b>function:</b> Sync Method Write file
* @author Hoojo
* @createDate 2011-9-26 12:21:22
* @param buff Buffer array
* @param start position
* @param length
* @return
*/
public synchronized int write (byte[] buff, int start, int length) {
int i =-1;
try {
Itemfile.write (buff, start, length);
i = length;
catch (IOException e) {
E.printstacktrace ();
}
return i;
}
public void Close () throws IOException {
if (itemfile!= null) {
Itemfile.close ();
}
}
}

This class basically completes the write file to the local specified file pointer, and returns the length of the current write file (the file pointer). This class will be called by the thread, and the file will be called by the thread after it is divided into the corresponding block. Each thread will call this class to complete the random write of the file.

three, a single thread download files
Copy Code code as follows:

Package com.hoo.download;
Import java.io.IOException;
Import Java.io.InputStream;
Import java.net.HttpURLConnection;
Import java.net.MalformedURLException;
Import Java.net.URL;
Import java.net.URLConnection;
Import Com.hoo.util.LogUtils;
/**
* <b>function:</b> single-threaded download files
* @author Hoojo
* @createDate 2011-9-22 02:55:10
* @file Downloadfile.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class DownloadFile extends Thread {
Download file URL
Private String URL;
Download file start location
Private long startpos;
Download file End Location
Private long Endpos;
Thread ID
private int threadId;
Whether the download is complete
Private Boolean isdownloadover = false;
Private Saveitemfile Itemfile;
private static final int buff_length = 1024 * 8;
/**
* @param url Download file URL
* @param name File
* @param startpos download file starting point
* @param endpos download file end point
* @param threadId Thread ID
* @throws IOException
*/
Public DownloadFile (string url, string name, long startpos, long endpos, int threadId) throws IOException {
Super ();
This.url = URL;
This.startpos = startpos;
This.endpos = Endpos;
This.threadid = threadId;
Block Download Write file contents
This.itemfile = new Saveitemfile (name, startpos);
}
@Override
public void Run () {
while (Endpos > Startpos &&!isdownloadover) {
try {
URL url = new URL (this.url);
HttpURLConnection conn = (httpurlconnection) url.openconnection ();
Set the connection timeout to 10000ms
Conn.setconnecttimeout (10000);
Sets the read data timeout of 10000ms
Conn.setreadtimeout (10000);
SetHeader (conn);
String property = "bytes=" + startpos + "-";
Conn.setrequestproperty ("RANGE", property);
Output log Information
LogUtils.log ("Start" + threadId + ":" + property + Endpos);
Printheader (conn);
Get file input stream, read file contents
InputStream is = Conn.getinputstream ();
byte[] buff = new Byte[buff_length];
int length =-1;
LogUtils.log ("#start #thread:" + threadId + ", startpos:" + startpos + ", Endpos:" + endpos);
while (length = Is.read (buff)) > 0 && startpos < endpos &&!isdownloadover) {
Writes the contents of the file, returns the length of the last write
Startpos + + itemfile.write (buff, 0, length);
}
LogUtils.log ("#over #thread:" + threadId + ", startpos:" + startpos + ", Endpos:" + endpos);
LogUtils.log ("Thread" + ThreadId + "is execute over!");
This.isdownloadover = true;
catch (Malformedurlexception e) {
E.printstacktrace ();
catch (IOException e) {
E.printstacktrace ();
finally {
try {
if (itemfile!= null) {
Itemfile.close ();
}
catch (IOException e) {
E.printstacktrace ();
}
}
}
if (Endpos < startpos &&!isdownloadover) {
LogUtils.log ("Thread" + threadId + "Startpos > Endpos, not need download file!");
This.isdownloadover = true;
}
if (Endpos = = Startpos &&!isdownloadover) {
LogUtils.log ("Thread" + threadId + "startpos = endpos, not need download file!");
This.isdownloadover = true;
}
}
/**
* <b>function:</b> Print Download file header information
* @author Hoojo
* @createDate 2011-9-22 05:44:35
* @param Conn HttpURLConnection
*/
public static void Printheader (URLConnection conn) {
int i = 1;
while (true) {
String Header = Conn.getheaderfieldkey (i);
i++;
if (header!= null) {
Logutils.info (header + ":" + Conn.getheaderfield (i));
} else {
Break
}
}
}
/**
* <b>function:</b> Set URLConnection header information, disguise request information
* @author Hoojo
* @createDate 2011-9-28 05:29:43
* @param con
*/
public static void SetHeader (URLConnection conn) {
Conn.setrequestproperty ("User-agent", "mozilla/5.0" (X11; U Linux i686; En-us; rv:1.9.0.3) gecko/2008092510 ubuntu/8.04 (Hardy) firefox/3.0.3 ");
Conn.setrequestproperty ("Accept-language", "en-us,en;q=0.7,zh-cn;q=0.3");
Conn.setrequestproperty ("accept-encoding", "utf-8");
Conn.setrequestproperty ("Accept-charset", "iso-8859-1,utf-8;q=0.7,*;q=0.7");
Conn.setrequestproperty ("Keep-alive", "300");
Conn.setrequestproperty ("Connnection", "keep-alive");
Conn.setrequestproperty ("If-modified-since", "Fri, 2009 17:00:05 GMT");
Conn.setrequestproperty ("If-none-match", "1261d8-4290-df64d224\");
Conn.setrequestproperty ("Cache-conntrol", "max-age=0");
Conn.setrequestproperty ("Referer", "http://www.baidu.com");
}
public Boolean isdownloadover () {
return isdownloadover;
}
Public long Getstartpos () {
return startpos;
}
Public long Getendpos () {
return endpos;
}
}

This class is primarily complete with a single thread file download, and will read the resource information for the specified URL through URLConnection. It then reads the contents of the file with InputStream, and then invokes the call to the Saveitemfile class to write locally the contents of the block currently being read.

Four, segmented multithreaded write file content
Copy Code code as follows:

Package com.hoo.download;
Import Java.io.DataInputStream;
Import Java.io.DataOutputStream;
Import Java.io.File;
Import Java.io.FileInputStream;
Import Java.io.FileOutputStream;
Import java.io.IOException;
Import java.net.HttpURLConnection;
Import java.net.MalformedURLException;
Import Java.net.URL;
Import Com.hoo.entity.DownloadInfo;
Import Com.hoo.util.LogUtils;
/**
* <b>function:</b> Bulk Download files
* @author Hoojo
* @createDate 2011-9-22 05:51:54
* @file Batchdownloadfile.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Batchdownloadfile implements Runnable {
Download file information
Private Downloadinfo Downloadinfo;
A set of start download locations
Private long[] startpos;
A set of end download Locations
Private long[] Endpos;
Sleep time
private static final int sleep_seconds = 500;
Child thread Download
Private downloadfile[] Fileitem;
File length
private int length;
is the first file
Private Boolean-i = true;
Whether to stop downloading
Private Boolean stop = false;
Temporary file information
Private File tempfile;
Public Batchdownloadfile (Downloadinfo downloadinfo) {
This.downloadinfo = Downloadinfo;
String TempPath = This.downloadInfo.getFilePath () + File.separator + downloadinfo.getfilename () + ". Position";
Tempfile = new File (TempPath);
If a file with a read in point location exists
if (tempfile.exists ()) {
i = false;
Just read the contents directly
try {
Readposinfo ();
catch (IOException e) {
E.printstacktrace ();
}
} else {
The length of the array is divided into the number of segments
startpos = new Long[downloadinfo.getsplitter ()];
Endpos = new Long[downloadinfo.getsplitter ()];
}
}
@Override
public void Run () {
First download, get download file length
if (a) {
Length = This.getfilesize ();//Get file Lengths
if (length = = 1) {
LogUtils.log ("File length is know!");
Stop = true;
else if (length = = 2) {
LogUtils.log ("Read file length is error!");
Stop = true;
else if (length > 0) {
/**
* EG
* Start:1, 3, 5, 7, 9
* End:3, 5, 7, 9, length
*/
for (int i = 0, len = startpos.length i < len; i++) {
int size = i * (Length/len);
Startpos[i] = size;
Set the position of the last ending point
if (i = = len-1) {
Endpos[i] = length;
} else {
Size = (i + 1) * (Length/len);
Endpos[i] = size;
}
LogUtils.log ("Start-end position[" + i + "]:" + startpos[i] + "-" + endpos[i]);
}
} else {
LogUtils.log ("Get file length was error, download is stop!");
Stop = true;
}
}
Child Threads Start downloading
if (!stop) {
Create a single-threaded Download object array
Fileitem = new Downloadfile[startpos.length];//startpos.length = Downloadinfo.getsplitter ()
for (int i = 0; i < startpos.length; i++) {
try {
Creates a specified number of single-threaded download objects, each of which completes the download of the specified block content independently
Fileitem[i] = new DownloadFile (
Downloadinfo.geturl (),
This.downloadInfo.getFilePath () + File.separator + downloadinfo.getfilename (),
Startpos[i], endpos[i], I
);
Fileitem[i].start ()//Start thread, start download
LogUtils.log ("Thread: + i +", startpos: "+ startpos[i] +", Endpos: "+ endpos[i]";
catch (IOException e) {
E.printstacktrace ();
}
}
Looping write download file length information
while (!stop) {
try {
Writeposinfo ();
LogUtils.log ("Downloading ...");
Thread.Sleep (Sleep_seconds);
Stop = true;
catch (IOException e) {
E.printstacktrace ();
catch (Interruptedexception e) {
E.printstacktrace ();
}
for (int i = 0; i < startpos.length; i++) {
if (!fileitem[i].isdownloadover ()) {
stop = false;
Break
}
}
}
Logutils.info ("Download task is finished!");
}
}
/**
* Save the write point data in a temporary file
* @author Hoojo
* @createDate 2011-9-23 05:25:37
* @throws IOException
*/
private void Writeposinfo () throws IOException {
DataOutputStream dos = new DataOutputStream (new FileOutputStream (tempfile));
Dos.writeint (startpos.length);
for (int i = 0; i < startpos.length; i++) {
Dos.writelong (Fileitem[i].getstartpos ());
Dos.writelong (Fileitem[i].getendpos ());
Logutils.info ("[" + fileitem[i].getstartpos () + "#" + fileitem[i].getendpos () + "]");
}
Dos.close ();
}
/**
* <b>function:</b> read location information for write point
* @author Hoojo
* @createDate 2011-9-23 05:30:29
* @throws IOException
*/
private void Readposinfo () throws IOException {
DataInputStream dis = new DataInputStream (new FileInputStream (tempfile));
int startposlength = Dis.readint ();
startpos = new Long[startposlength];
Endpos = new Long[startposlength];
for (int i = 0; i < startposlength; i++) {
Startpos[i] = Dis.readlong ();
Endpos[i] = Dis.readlong ();
}
Dis.close ();
}
/**
* <b>function:</b> Get the length of the download file
* @author Hoojo
* @createDate 2011-9-26 12:15:08
* @return
*/
private int GetFileSize () {
int filelength =-1;
try {
URL url = new URL (this.downloadInfo.getUrl ());
HttpURLConnection conn = (httpurlconnection) url.openconnection ();
Downloadfile.setheader (conn);
int statecode = Conn.getresponsecode ();
Determine if HTTP status is http/1.1 206 Partial content or OK
if (statecode!= httpurlconnection.http_ok && statecode!= httpurlconnection.http_partial) {
LogUtils.log ("Error Code:" + statecode);
Return-2;
else if (Statecode >= 400) {
LogUtils.log ("Error Code:" + statecode);
Return-2;
} else {
Get length
Filelength = Conn.getcontentlength ();
LogUtils.log ("filelength:" + filelength);
}
Read file length
/*for (int i = 1;; i++) {
String Header = Conn.getheaderfieldkey (i);
if (header!= null) {
if ("Content-length". Equals (header)) {
Filelength = Integer.parseint (Conn.getheaderfield (i));
Break
}
} else {
Break
}
}
*/
Downloadfile.printheader (conn);
catch (Malformedurlexception e) {
E.printstacktrace ();
catch (IOException e) {
E.printstacktrace ();
}
return filelength;
}
}

This class basically completes reading the contents of the specified URL resource and gets the length of the resource. The resource is then divided into the specified number of blocks, and the starting download location and the ending download location of each block are stored in an array. Each piece opens up a separate thread to start the download. Before starting the download, you need to create a temporary file that writes to the current download thread's start download pointer position and the end download pointer location.

v. Tools, test class
Log Tool class
Copy Code code as follows:

Package com.hoo.util;
/**
* <b>function:</b> Logging Tool class
* @author Hoojo
* @createDate 2011-9-21 05:21:27
* @file Logutils.java
* @package Com.hoo.util
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
Public abstract class Logutils {
public static void log (Object message) {
SYSTEM.ERR.PRINTLN (message);
}
public static void log (String message) {
SYSTEM.ERR.PRINTLN (message);
}
public static void log (int message) {
SYSTEM.ERR.PRINTLN (message);
}
public static void info (Object message) {
SYSTEM.OUT.PRINTLN (message);
}
public static void info (String message) {
SYSTEM.OUT.PRINTLN (message);
}
public static void info (int message) {
SYSTEM.OUT.PRINTLN (message);
}
}

Download Tool class
Copy Code code as follows:

Package com.hoo.util;
Import Com.hoo.download.BatchDownloadFile;
Import Com.hoo.entity.DownloadInfo;
/**
* <b>function:</b> block Multi-threaded Download tool class
* @author Hoojo
* @createDate 2011-9-28 05:22:18
* @file Downloadutils.java
* @package Com.hoo.util
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
Public abstract class Downloadutils {
public static void Download (String URL) {
Downloadinfo bean = new Downloadinfo (URL);
Logutils.info (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
}
public static void Download (String url, int threadnum) {
Downloadinfo bean = new Downloadinfo (URL, threadnum);
Logutils.info (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
}
public static void Download (string url, String fileName, string filePath, int threadnum) {
Downloadinfo bean = new Downloadinfo (URL, fileName, FilePath, threadnum);
Logutils.info (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
}
}

Download Test class
Copy Code code as follows:

Package com.hoo.test;
Import Com.hoo.util.DownloadUtils;
/**
* <b>function:</b> Download Test
* @author Hoojo
* @createDate 2011-9-23 05:49:46
* @file Testdownloadmain.java
* @package Com.hoo.download
* @project Multithreaddownload
* @blog Http://blog.csdn.net/IBM_hoojo
* @email hoojo_@126.com
* @version 1.0
*/
public class Testdownloadmain {
public static void Main (string[] args) {
/*downloadinfo bean = new Downloadinfo ("yun_qi_img/2011092116224363.jpg");
System.out.println (Bean);
Batchdownloadfile down = new Batchdownloadfile (bean);
New Thread (Down). Start ();
Downloadutils.download ("yun_qi_img/2011092116224363.jpg");
Downloadutils.download ("http://mp3.baidu.com/j?j=2&url=http%3A%2F%2Fzhangmenshiting2.baidu.com%2Fdata% 2fmusic%2f1669425%2f%25e9%2599%25b7%25e5%2585%25a5%25e7%2588%25b1%25e9%2587%258c%25e9%259d%25a2.mp3%3fxcode% 3d2ff36fb70737c816553396c56deab3f1 "," Aa.mp3 "," C:/temp ", 5);
}
}

Multi-threaded downloads are mainly in the third and fourth parts, and others are well understood. The source code provides the corresponding annotation, easy to understand.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

Tags Index: