Java for large file download (HTTP mode)

Java implementation of large file download, based on HTTP, control God horse will not say.

Idea: The download file is nothing but to read the file and then write the file, mainly these two steps, the main difficulty:

1. Read the file, is the hard disk to the memory process, due to the JDK memory limit, can not read too large.

2. Write a file, is to respond to the browser side of the process, the HTTP protocol is a short link, if the file is too slow to write, the time is too long, it will cause the browser to die.

Knowledge Points:

The 1.org.apache.http.impl.client.closeablehttpclient analog httpclient client sends an HTTP request that can control the byte location of the requested file.

2.BufferedInputStream is familiar with using it to accept requests for streaming information caching.

The 3.RandomAccessFile file is a random class that can write stream information to a file at a specified location.

Based on the above information, my implementation of the idea is to first determine the size of the download file, with multi-threaded segmentation custom HTTP request quantity and request content, response to write to Randomaccessfile in the specified location. In the vulgar point is the large HTTP partition into a small HTTP request, equivalent to each request a Web page.

Nonsense not to say, on the code.

Downloadmanagertest class:

package xxxx;import java.io.file;import java.io.ioexception;import  java.io.randomaccessfile;import java.net.httpurlconnection;import java.net.url;import  java.util.concurrent.countdownlatch;import org.apache.commons.lang.exception.exceptionutils;import  Org.apache.http.impl.client.closeablehttpclient;import org.apache.http.impl.client.httpclients;import  org.apache.http.impl.conn.PoolingHttpClientConnectionManager;import org.junit.After;import  org.junit.before;import org.junit.test;import org.junit.runner.runwith;import org.slf4j.logger; Import org.slf4j.loggerfactory;import org.springframework.beans.factory.annotation.autowired;import  org.springframework.core.task.TaskExecutor;import org.springframework.test.context.ActiveProfiles; import org.springframework.test.context.contextconfiguration;import  Org.springframework.test.context.junit4.AbstractTransactionalJUnit4SpringContextTests;import org.springframework.test.context.junit4.springjunit4classrunner;/** *  *  File download Management class  */@RunWith (Springjunit4classrunner.class) @ActiveProfiles ("Test") @ContextConfiguration (locations={ "Classpath:test/applicationcontext.xml"}) public class downloadmanagertest extends  abstracttransactionaljunit4springcontexttests{private static final logger logger =  loggerfactory.getlogger (Downloadmanagertest.class);/** *  *  the number of bytes downloaded per thread  */ private long unitsize = 1000 * 1024; @Autowiredprivate  TaskExecutor  taskexecutor;private closeablehttpclient httpclient;private long starttimes;private  long endtimes;         @Before     public  Void setup ()  throws exception    {    starttimes =  system.currenttimemillis ();Nbsp;       system.out.println ("Test start ....");    }          @After     public void teardown ()   throws exception    {    endtimes =  System.currenttimemillis ();         system.out.println ("End of test!!");         system.out.println ("********************");         system.out.println ("Total download time:" + (Endtimes-starttimes)/1000+ "s");         system.out.println ("********************");    }public  Downloadmanagertest ()  {system.out.println ("Initialize test class ...."); Poolinghttpclientconnectionmanager cm = new poolinghttpclientconnectionmanager (); Cm.setmaxtotal (+); Httpclient = httpclients.custom (). Setconnectionmanager (CM). Build ();}/** *  *  start multiple threads download file  */@Testpublic  void  dodownload ()  throws  ioexception {string remotefileurl= "Http://{host}:{port}/{project}/xx.xml"; String localpath= "e://test//"; String filename = new url (Remotefileurl). GetFile (); SYSTEM.OUT.PRINTLN ("Remote file name:" +filename); Filename = filename.substring (Filename.lastindexof ("/")   + 1,filename.length ()). Replace ("%20",  " "); SYSTEM.OUT.PRINTLN ("Local file name:" +filename); Long filesize = this.getremotefilesize (Remotefileurl); This.createfile (Localpath+system.currenttimemillis () +filename, filesize); long threadcount =  (filesize/unitsize) + (filesize % unitsize!=0?1:0); long offset  = 0; Countdownlatch end = new countdownlatch (Threadcount.intvalue ());if  (fileSize  <= unitsize)  {//  If the remote file size is less than or equal to unitsizedownloadthreadtest downloadthread = new& nbsp;D ownloadthreadtest (remotefileurl,localpath+filename, offset, filesize,end,httpclient); Taskexecutor.execute (Downloadthread);}  else {//  if the remote file size is greater than unitsizefor  (int i = 1; i <  threadcount; i++)  {downloadthreadtest downloadthread = new downloadthreadtest ( remotefileurl, localpath+filename, offset, unitsize,end,httpclient); TaskExecutor.execute ( Downloadthread); offset = offset + unitsize;} if  (filesize % unitsize != 0)  {//  if not divisible, you need to create another thread to download the remaining bytes Downloadthreadtest  downloadthread = new downloadthreadtest (remotefileurl, localpath+filename,  offset, filesize - unitsize *  (threadCount-1), end,httpclient); Taskexecutor.execute ( Downloadthread);}} Try {end.await ();}  catch  (interruptedexception e)  {logger.error ("downloadmanager exception msg:{}" , ExceptionutiLs.getfullstacktrace (e)); E.printstacktrace ();} System.out.println ("111111"); Logger.debug ("Download done! {}  ", localpath+filename);//return localpath+filename;} /** *  *  get remote File size  */private long getremotefilesize (String  Remotefileurl)  throws IOException {long fileSize = 0; httpurlconnection httpconnection =  (httpurlconnection)  new url (REMOTEFILEURL). OpenConnection (); Httpconnection.setrequestmethod ("HEAD");int responsecode =  Httpconnection.getresponsecode ();if  (responsecode >= 400)  {logger.debug ("Web server response Error!"); return 0;} string sheader;for  (int i = 1;;  i++)  {sheader = httpconnection.getheaderfieldkey (i);if  (sHeader != null  && sheader.equals ("Content-length"))  {system.out.println ("File size contentlength:" +  Httpconnection.getcontentlength ()); Filesize = long.parselonG (Httpconnection.getheaderfield (Sheader)); break;}} Return filesize;} /** *  *  creates a file of the specified size  */private void createfile (string filename,  long filesize)  throws ioexception {file newfile = new file (fileName); Randomaccessfile raf = new randomaccessfile (newfile,  "RW"); Raf.setLength (fileSize); Raf.close ();} Public taskexecutor gettaskexecutor ()  {return taskexecutor;} Public void settaskexecutor (Taskexecutor taskexecutor)  {this.taskExecutor =  Taskexecutor;}}

Downloadthreadtest class:

package xxxx;import java.io.bufferedinputstream;import java.io.file;import  java.io.ioexception;import java.io.randomaccessfile;import java.util.concurrent.countdownlatch; import org.apache.commons.lang.exception.exceptionutils;import  org.apache.http.client.clientprotocolexception;import  Org.apache.http.client.methods.closeablehttpresponse;import org.apache.http.client.methods.httpget;import  org.apache.http.impl.client.CloseableHttpClient;import org.apache.http.protocol.BasicHttpContext; import org.apache.http.protocol.httpcontext;import org.slf4j.logger;import  org.slf4j.loggerfactory;/** *  *  is responsible for the file download class  */public class downloadthreadtest  extends Thread {private static final Logger LOGGER =  Loggerfactory.getlogger (Downloadthreadtest.class);/** *  *  files to download  */private  String url = null;/** *  *  Local file name  */private String fileName = null;/** *  *  Offset  */private long offset = 0;/** *  *  number of downloaded bytes assigned to this thread  */ private long length = 0;private countdownlatch end;private  closeablehttpclient httpclient;private httpcontext context;/** *  *  @param  url *             Download file Address  *   *  @param  fileName *              Save file name  *  *  @param  offset *              This thread download offset  *  *  @param  length *              this thread download length  *  *  *  * @ Author angus.wang *  *&nbSp;*/public downloadthreadtest (String url, string file, long offset, long  length,countdownlatch end, closeablehttpclient httpclient)  {this.url = url ; this.filename = file;this.offset = offset;this.length = length;this.end =  end;this.httpclient = httpclient;this.context = new basichttpcontext (); Logger.debug ("offset ="  + offset +  "byte number ="  + length);} Public void run ()  {try {httpget httpget = new httpget (This.url); Httpget.addheader ("Range",  "bytes="  + this.offset +  "-" +  (this.offset +  this.length - 1)); Closeablehttpresponse response = httpclient.execute (Httpget,context); Bufferedinputstream bis = new bufferedinputstream (Response.getentity (). GetContent ()); byte[ ] buff = new byte[1024];int  bytesread; File newfile = new file (FileName); Randomaccessfile raf = new randomaccessfile (newfile,  "RW");while  ((bytesRead  = bis.read (buff, 0, buff.length))  != -1)  {raf.seek (This.offset); Raf.write ( Buff, 0, bytesread); this.offset = this.offset + bytesread;} Raf.close (); Bis.close ();}  catch  (clientprotocolexception e)  {logger.error ("downloadthread exception msg:{} ", Exceptionutils.getfullstacktrace (e));}  catch  (ioexception e)  {logger.error ("downloadthread exception msg:{}", Exceptionutils.getfullstacktrace (e));}  finally {end.countdown (); Logger.info (End.getcount ()  +  " is go on!"); System.out.println (End.getcount ()  +  " is go on!");}}}

Application.xml

<bean id= "Taskexecutor" class= "Org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor" ><!-- Thread pool Active threads--><property name= "Corepoolsize" value= "5"/><!--thread Pools maximum active threads--><property name= " Maxpoolsize "value="/><!--the maximum capacity of the queue--><property name= "queuecapacity" value= "ten"/></bean> <bean id= "Downloadmanager" class= "xx. Downloadmanagertest "><property name=" Taskexecutor "ref=" Taskexecutor "/></bean>

Test run, 500M, I have a speed of about half an hour. To download larger files, you can change the maximum queue size indefinitely, as long as the JDK memory is large enough.

If you disagree, you are welcome to correct the great God.

Java for large file download (HTTP mode)