Non-C # articles, but you can learn from the idea--using Java to implement breakpoint continuation (HTTP)

Using Java to implement breakpoint continuation (HTTP)



Content:

(a) The principle of the continuation of the breakpoint
Two Key points in Java implementation of breakpoint continuation
(iii) The realization of the core of the breakpoint continuous transmission
About the author




At COSL (zhong_hua@263.net)
May 2001

(a) The principle of the continuation of the breakpoint
In fact, the principle of the continuation of the breakpoint is very simple, is the HTTP request and the general download is different.
For example, when a browser requests a text on the server, the request is made as follows:
Suppose the server domain name is wwww.sjtu.edu.cn and the file name is Down.zip.
Get/down.zip http/1.1
Accept:image/gif, Image/x-xbitmap, Image/jpeg, Image/pjpeg, application/vnd.ms-
Excel, Application/msword, Application/vnd.ms-powerpoint, */*
Accept-language:zh-cn
Accept-encoding:gzip, deflate
user-agent:mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Connection:keep-alive


After the server receives the request, it looks for the requested file, extracts the file's information, and returns it to the browser, returning the following information:


200
content-length=106786028
Accept-ranges=bytes
Date=mon, APR 2001 12:56:11 GMT
etag=w/"02ca57e173c11:95b"
Content-type=application/octet-stream
server=microsoft-iis/5.0
Last-modified=mon, APR 2001 12:56:11 GMT




The so-called breakpoint continuation, that is, from the file has been downloaded from the place to continue to download. So in the client browser, pass
Add a message to the Web server-where to start.
The following is a "browser" of your own to pass the request information to the Web server, which requires starting from 2000070 bytes.
Get/down.zip http/1.0
User-agent:netfox
range:bytes=2000070-
Accept:text/html, Image/gif, Image/jpeg, *; Q=.2, */*; q=.2


Take a closer look and you'll find one more line range:bytes=2000070-
This line is meant to tell the server that the file is down.zip from 2000070 bytes and that the preceding byte is not to be transmitted.
After the server receives this request, the information returned is as follows:
206
content-length=106786028
Content-range=bytes 2000070-106786027/106786028
Date=mon, APR 2001 12:55:20 GMT
etag=w/"02ca57e173c11:95b"
Content-type=application/octet-stream
server=microsoft-iis/5.0
Last-modified=mon, APR 2001 12:55:20 GMT


Compared to the information returned by the previous server, you will see an added line:
Content-range=bytes 2000070-106786027/106786028
The returned code is also changed to 206, and is no longer 200.


Knowing the above principles, you can proceed to the programming of the continuation of the breakpoint.


Two Key points in Java implementation of breakpoint continuation


(1) What method to implement the submission of range:bytes=2000070-.
Of course, with the most original socket is certainly able to complete, but it is too much trouble, in fact, Java NET package provides this functionality. The code is as follows:

URL url = new URL ("Http://www.sjtu.edu.cn/down.zip");
HttpURLConnection httpconnection = (httpurlconnection) url.openconnection ();


Set User-agent
Httpconnection.setrequestproperty ("User-agent", "Netfox");
Set the starting position of a breakpoint continuation
Httpconnection.setrequestproperty ("RANGE", "bytes=2000070");
Get input stream
InputStream input = Httpconnection.getinputstream ();



The byte stream that is fetched from the input stream is the byte stream that the Down.zip file starts with 2000070.
You see, in fact, the continuation of the breakpoint in Java to achieve a very simple bar.
The next thing to do is how to save the obtained stream into the file.


The method used to save the file.
I am using the Randaccessfile class in the IO package.
The operation is fairly simple, assuming that the file is saved from 2000070 and the code is as follows:
Randomaccess osavedfile = new Randomaccessfile ("Down.zip", "RW");
Long NPOs = 2000070;
Locate file pointer to NPOs location
Osavedfile.seek (NPOs);
Byte[] B = new byte[1024];
int nread;
Reads a byte stream from the input stream and writes it to a file
while ((Nread=input.read (b,0,1024)) > 0)
{
Osavedfile.write (B,0,nread);
}

How, also very simple.
The next thing to do is to integrate into a complete program. Including a series of line program control system and so on.



(iii) The realization of the core of the breakpoint continuous transmission
The main use of 6 classes, including a test class.
Sitefilefetch.java is responsible for crawling the entire file, controlling internal threads (Filesplitterfetch classes).
Filesplitterfetch.java is responsible for the capture of some files.
Fileaccess.java is responsible for the storage of files.
Siteinfobean.java information about the files you want to crawl, such as the directory where the file is saved, the name, the URL of the crawl file, and so on.
Utility.java Tool class, put some simple methods.
Testmethod.java Test class.


The following are the source programs:

/*
**sitefilefetch.java
*/
Package Netfox;
Import java.io.*;
Import java.net.*;


public class Sitefilefetch extends Thread {


Siteinfobean Siteinfobean = null; File Information Bean
Long[] Nstartpos; Start position
Long[] Nendpos; End Position
Filesplitterfetch[] Filesplitterfetch; Child Thread Object
Long nfilelength; File length
Boolean bfirst = true; Whether to fetch files for the first time
Boolean bstop = false; Stop flag
File tmpfile; Temporary information for file downloads
DataOutputStream output; Output stream to File


Public Sitefilefetch (Siteinfobean Bean) throws IOException
{
Siteinfobean = Bean;
Tmpfile = File.createtempfile ("Zhong", "1111", New File (Bean.getsfilepath ());
Tmpfile = new File (Bean.getsfilepath () +file.separator + bean.getsfilename () + ". Info");
if (Tmpfile.exists ())
{
Bfirst = false;
Read_npos ();
}
Else
{
Nstartpos = new Long[bean.getnsplitter ()];
Nendpos = new Long[bean.getnsplitter ()];
}



}


public void Run ()
{
Get file length
Split file
Instance Filesplitterfetch
Start Filesplitterfetch Thread
Wait for child thread to return
try{
if (Bfirst)
{
Nfilelength = GetFileSize ();
if (nfilelength = = 1)
{
System.err.println ("File Length is not known!");
}
else if (nfilelength = 2)
{
System.err.println ("File is not access!");
}
Else
{
for (int i=0;i<nstartpos.length;i++)
{
Nstartpos[i] = (long) (i* (nfilelength/nstartpos.length));
}
for (int i=0;i<nendpos.length-1;i++)
{
Nendpos[i] = nstartpos[i+1];
}
Nendpos[nendpos.length-1] = nfilelength;
}
}


Start Child threads
Filesplitterfetch = new Filesplitterfetch[nstartpos.length];
for (int i=0;i<nstartpos.length;i++)
{
Filesplitterfetch[i] = new Filesplitterfetch (Siteinfobean.getssiteurl (),
Siteinfobean.getsfilepath () + File.separator + siteinfobean.getsfilename (),
Nstartpos[i],nendpos[i],i);
Utility.log ("Thread" + i + ", Nstartpos =" + nstartpos[i] + ", Nendpos =" + nendpos[i]);
Filesplitterfetch[i].start ();
}
FILESPLITTERFETCH[NPOS.LENGTH-1] = new Filesplitterfetch (Siteinfobean.getssiteurl (),
Siteinfobean.getsfilepath () + File.separator + siteinfobean.getsfilename (), Npos[npos.length-1],nfilelength, NPOS.LENGTH-1);
Utility.log ("Thread" + (npos.length-1) + ", Nstartpos =" + npos[npos.length-1] + ",
Nendpos = "+ nfilelength);
Filesplitterfetch[npos.length-1].start ();


Wait for child thread to end
int count = 0;
Whether to end a while loop
Boolean breakwhile = false;


while (!bstop)
{
Write_npos ();
Utility.sleep (500);
Breakwhile = true;


for (int i=0;i<nstartpos.length;i++)
{
if (!filesplitterfetch[i].bdownover)
{
Breakwhile = false;
Break
}
}
if (breakwhile)
Break


count++;
if (count>4)
Sitestop ();
}


System.err.println ("File download end!") ");
}
catch (Exception e) {e.printstacktrace ();}
}


Get file length
Public long GetFileSize ()
{
int nfilelength =-1;
try{
URL url = new URL (siteinfobean.getssiteurl ());
HttpURLConnection httpconnection = (httpurlconnection) url.openconnection ();
Httpconnection.setrequestproperty ("User-agent", "Netfox");


int Responsecode=httpconnection.getresponsecode ();
if (responsecode>=400)
{
Processerrorcode (Responsecode);
Return-2; -2 represent access is error
}


String Sheader;


for (int i=1;;; i++)
{
DataInputStream in = new DataInputStream (Httpconnection.getinputstream ());
Utility.log (In.readline ());
Sheader=httpconnection.getheaderfieldkey (i);
if (sheader!=null)
{
if (Sheader.equals ("Content-length"))
{
Nfilelength = Integer.parseint (Httpconnection.getheaderfield (Sheader));
Break
}
}
Else
Break
}
}
catch (IOException e) {e.printstacktrace ();}
catch (Exception e) {e.printstacktrace ();}


Utility.log (nfilelength);


return nfilelength;
}


Save download information (file pointer location)
private void Write_npos ()
{
try{
Output = new DataOutputStream (new FileOutputStream (tmpfile));
Output.writeint (nstartpos.length);
for (int i=0;i<nstartpos.length;i++)
{
Output.writelong (Npos[i]);
Output.writelong (Filesplitterfetch[i].nstartpos);
Output.writelong (Filesplitterfetch[i].nendpos);
}
Output.close ();
}
catch (IOException e) {e.printstacktrace ();}
catch (Exception e) {e.printstacktrace ();}
}


Read saved download information (file pointer location)
private void Read_npos ()
{
try{
DataInputStream input = new DataInputStream (new FileInputStream (tmpfile));
int ncount = Input.readint ();
Nstartpos = new Long[ncount];
Nendpos = new Long[ncount];
for (int i=0;i<nstartpos.length;i++)
{
Nstartpos[i] = Input.readlong ();
Nendpos[i] = Input.readlong ();
}
Input.close ();
}
catch (IOException e) {e.printstacktrace ();}
catch (Exception e) {e.printstacktrace ();}
}


private void Processerrorcode (int nerrorcode)
{
System.err.println ("Error Code:" + nerrorcode);
}


Stop file Downloads
public void Sitestop ()
{
Bstop = true;
for (int i=0;i<nstartpos.length;i++)
Filesplitterfetch[i].splitterstop ();


}
}
/*
**filesplitterfetch.java
*/
Package Netfox;


Import java.io.*;
Import java.net.*;


public class Filesplitterfetch extends Thread {


String sURL; File URL
Long Nstartpos; File Snippet Start Position
Long Nendpos; File Snippet End Position
int nthreadid; Thread ' s ID
Boolean bdownover = false; Downing is Over
Boolean bstop = false; Stop identical
Fileaccessi Fileaccessi = null; File Access Interface


Public Filesplitterfetch (String surl,string sname,long nstart,long nend,int id) throws IOException
{
This.surl = sURL;
This.nstartpos = Nstart;
This.nendpos = Nend;
Nthreadid = ID;
Fileaccessi = new Fileaccessi (Sname,nstartpos);
}


public void Run ()
{
while (Nstartpos < Nendpos &&!bstop)
{


try{
URL url = new URL (surl);
HttpURLConnection httpconnection = (httpurlconnection) url.openconnection ();
Httpconnection.setrequestproperty ("User-agent", "Netfox");
String sproperty = "bytes=" +nstartpos+ "-";
Httpconnection.setrequestproperty ("RANGE", Sproperty);
Utility.log (Sproperty);


InputStream input = Httpconnection.getinputstream ();
Logresponsehead (httpconnection);


Byte[] B = new byte[1024];
int nread;
while ((Nread=input.read (b,0,1024)) > 0 && nstartpos < nendpos &&!bstop)
{
Nstartpos + + fileaccessi.write (b,0,nread);
if (Nthreadid = 1)
Utility.log ("Nstartpos =" + Nstartpos + ", Nendpos =" + Nendpos);
}


Utility.log ("Thread" + Nthreadid + "is over!");
Bdownover = true;
NPOs = Fileaccessi.write (B,0,nread);
}
catch (Exception e) {e.printstacktrace ();}
}
}


Print header information for a response
public void Logresponsehead (httpurlconnection con)
{
for (int i=1;;; i++)
{
String Header=con.getheaderfieldkey (i);
if (header!=null)
Responseheaders.put (header) (Header,httpconnection.getheaderfield);
Utility.log (header+ ":" +con.getheaderfield (header));
Else
Break
}
}


public void Splitterstop ()
{
Bstop = true;
}


}


/*
**fileaccess.java
*/
Package Netfox;
Import java.io.*;


public class Fileaccessi implements serializable{


Randomaccessfile Osavedfile;
Long NPOs;


Public Fileaccessi () throws IOException
{
This ("", 0);
}


Public Fileaccessi (String Sname,long NPOs) throws IOException
{
Osavedfile = new Randomaccessfile (sname, "RW");
This.npos = NPOs;
Osavedfile.seek (NPOs);
}


public synchronized int write (byte[] b,int nstart,int nlen)
{
int n =-1;
try{
Osavedfile.write (B,nstart,nlen);
n = nlen;
}
catch (IOException E)
{
E.printstacktrace ();
}


return n;
}


}


/*
**siteinfobean.java
*/
Package Netfox;


public class Siteinfobean {


Private String Ssiteurl; Site ' s URL
Private String Sfilepath; Saved File ' s Path
Private String sFileName; Saved File ' s Name
private int nsplitter; Count of splited downloading File


Public Siteinfobean ()
{
Default value of Nsplitter is 5
This ("", "", "", 5);
}


Public Siteinfobean (String surl,string spath,string sname,int nspiltter)
{
Ssiteurl= sURL;
Sfilepath = spath;
sFileName = sname;
This.nsplitter = Nspiltter;


}


Public String Getssiteurl ()
{
return ssiteurl;
}


public void Setssiteurl (String value)
{
Ssiteurl = value;
}


Public String Getsfilepath ()
{
return sfilepath;
}


public void Setsfilepath (String value)
{
Sfilepath = value;
}


Public String Getsfilename ()
{
return sfilename;
}


public void Setsfilename (String value)
{
sFileName = value;
}


public int Getnsplitter ()
{
return nsplitter;
}


public void Setnsplitter (int ncount)
{
Nsplitter = ncount;
}
}


/*
**utility.java
*/
Package Netfox;


public class Utility {


Public Utility ()
{


}


public static void sleep (int nsecond)
{
try{
Thread.Sleep (Nsecond);
}
catch (Exception e)
{
E.printstacktrace ();
}
}


public static void log (String sMsg)
{
System.err.println (SMSG);
}


public static void log (int sMsg)
{
System.err.println (SMSG);
}
}


/*
**testmethod.java
*/
Package Netfox;


public class TestMethod {


Public TestMethod ()
{///xx/weblogic60b2_win.exe
try{
Siteinfobean bean = new Siteinfobean ("Http://localhost/xx/weblogic60b2_win.exe", "L:\\temp", "Weblogic60b2_win.exe", 5);
Siteinfobean bean = new Siteinfobean ("Http://localhost:8080/down.zip", "L:\\temp", "Weblogic60b2_win.exe", 5);
Sitefilefetch Filefetch = new Sitefilefetch (bean);
Filefetch.start ();
}
catch (Exception e) {e.printstacktrace ();}


}


public static void Main (string[] args)
{
New TestMethod ();
}
}