Implement HTTP resumable upload using Java

(1) Principle of resumable Data Transfer
In fact, the principle of resumable upload is very simple, that is, the Http request is different from the general download.
For example, when a browser requests a file on the server, the request is as follows:
Assume that the server domain name is wwww.sjtu.edu.cn, and the file name is down.zip.
GET/down.zip HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd. ms-
Excel, application/msword, application/vnd. ms-powerpoint ,*/*
Accept-Language: zh-cn
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Connection: Keep-Alive
After receiving the request, the server searches for the requested file as required, extracts the file information, and then returns it to the browser. The returned information is as follows:
200
Content-Length = 106786028
Accept-Ranges = bytes
Date = Mon, 30 Apr 2001 12:56:11 GMT
ETag = W/"02ca57e173c11: 95b"
Content-Type = application/octet-stream
Server = Microsoft-Microsoft IIS/5.0
Last-Modified = Mon, 30 Apr 2001 12:56:11 GMT
 
The so-called resumable upload means that the download starts from where the object has been downloaded. Therefore, it is sent
Add one more message to the Web server-where to start.
The following is a self-compiled "Browser" to transmit the request information to the Web server, starting from 2000070 bytes.
GET/down.zip HTTP/1.0
User-Agent: NetFox
Value RANGE: bytes = 2000070-
Accept: text/html, image/gif, image/jpeg, *; q =. 2, */*; q =. 2
 
Take a closer look and you will find that there is an additional row RANGE: bytes = 2000070-
In this line, the file down.zip is uploaded starting from 2000070 bytes, and the previous bytes are not required.
After receiving the request, the server returns the following information:
206
Content-Length = 106786028
Content-Range = bytes 2000070-106786027/106786028
Date = Mon, 30 Apr 2001 12:55:20 GMT
ETag = W/"02ca57e173c11: 95b"
Content-Type = application/octet-stream
Server = Microsoft-Microsoft IIS/5.0
Last-Modified = Mon, 30 Apr 2001 12:55:20 GMT
 
Compared with the information returned by the preceding server, a new row is added:
Content-Range = bytes 2000070-106786027/106786028
The Returned Code is also changed to 206, instead of 200.
 
With the above principles, you can program resumable data transfer.
 
(2) Key Points for implementing resumable data transfer in Java
(1) method used to implement the submit RANGE: bytes = 2000070 -.
Of course, it is certainly possible to use the most primitive Socket, but it is too time-consuming. In fact, this function is provided in the Java net package. The Code is as follows:
URL url = new URL ("http://www.sjtu.edu.cn/down.zip ");
HttpURLConnection httpConnection = (HttpURLConnection) url. openConnection ();
// Set User-Agent
HttpConnection. setRequestProperty ("User-Agent", "NetFox ");
// Set the start position of resumable upload
HttpConnection. setRequestProperty ("RANGE", "bytes = 2000070 ");
// Obtain the input stream
InputStream input = httpConnection. getInputStream ();
 
The byte stream that starts from 2000070 in the down.zip file of the input stream.
As you can see, it is quite easy to implement resumable upload in Java.
The next thing to do is how to save the obtained stream to the file.
 
The method used to save the file.
I use the RandAccessFile class in the IO package.
The operation is quite simple. Assume that the file is saved from 2000070. The Code is as follows:
RandomAccess oSavedFile = new RandomAccessFile ("down.zip", "rw ");
Long nPos = 2000070;
// Locate the file pointer to the nPos position
OSavedFile. seek (nPos );
Byte [] B = new byte [1024];
Int nRead;
// Read the byte stream from the input stream and write it to the file
While (nRead = input. read (B, 0,1024) & gt; 0)
{
OSavedFile. write (B, 0, nRead );
}
 
It's easy.
The next step is to integrate it into a complete program. Including a series of thread control and so on.
 
(3) implementation of the resumable upload Kernel
It mainly uses six classes, including one test class.
SiteFileFetch. java is responsible for capturing the entire file and controlling Internal Threads (FileSplitterFetch class ).
FileSplitterFetch. java is responsible for capturing some files.
FileAccess. java stores files.
SiteInfoBean. java indicates the information of the file to be crawled, such as the Directory, name, and URL of the file to be crawled.
Utility. java tool class, put some simple methods.
TestMethod. java test class.
 
The following is the source program:
/*
** SiteFileFetch. java
*/
Package NetFox;
Import java. io .*;
Import java.net .*;
 
Public class SiteFileFetch extends Thread {
 
SiteInfoBean siteInfoBean = null; // File Information Bean
Long [] nStartPos; // start position
Long [] nEndPos; // end position
FileSplitterFetch [] fileSplitterFetch; // subthread object
Long nFileLength; // file length
Boolean bFirst = true; // whether to obtain the object for the first time
Boolean bStop = false; // stop flag
File tmpFile; // temporary File download information
DataOutputStream output; // output stream to the file
 
Public SiteFileFetch (SiteInfoBean bean) throws IOException
{
SiteInfoBean = bean;
// TmpFile = File. createTempFile ("zhong", "1111", new File (bean. getSFilePath ()));
TmpFile = new File (bean. getSFilePath () + File. separator + bean. getSFileName () + ". info ");
If (tmpFile. exists ())
{
BFirst = false;
Read_nPos ();
}
Else
{
NStartPos = new long [bean. getNSplitter ()];
NEndPos = new long [bean. getNSplitter ()];
}
 
}
 
Public void run ()
{
// Obtain the object Length
// Split the file
// Instance FileSplitterFetch
// Start the FileSplitterFetch thread
// Wait for the subthread to return
Try {
If (bFirst)
{
NFileLength = getFileSize ();
If (nFileLength =-1)
{
System. err. println ("File Length is not known! ");
}
Else if (nFileLength =-2)
{
System. err. println ("File is not access! ");
}
Else
{
For (int I = 0; I & lt; nStartPos. length; I ++)
{
NStartPos [I] = (long) (I * (nFileLength/nStartPos. length ));
}
For (int I = 0; I & lt; nEndPos. length-1; I ++)
{
NEndPos [I] = nStartPos [I + 1];
}
NEndPos [nEndPos. length-1] = nFileLength;
}
}
 
// Promoter thread
FileSplitterFetch = new FileSplitterFetch [nStartPos. length];
For (int I = 0; I & lt; nStartPos. length; I ++)
{
FileSplitterFetch [I] = new FileSplitterFetch (siteInfoBean. getSSiteURL (),
SiteInfoBean. getSFilePath () + File. separator + siteInfoBean. getSFileName (),
NStartPos [I], nEndPos [I], I );
Utility. log ("Thread" + I + ", nStartPos =" + nStartPos [I] + ", nEndPos =" + nEndPos [I]);
FileSplitterFetch [I]. start ();
}
// FileSplitterFetch [nPos. length-1] = new FileSplitterFetch (siteInfoBean. getSSiteURL (),
SiteInfoBean. getSFilePath () + File. separator + siteInfoBean. getSFileName (), nPos [nPos. length-1], nFileLength, nPos. length-1 );
// Utility. log ("Thread" + (nPos. length-1) + ", nStartPos =" + nPos [nPos. length-1] + ",
NEndPos = "+ nFileLength );
// FileSplitterFetch [nPos. length-1]. start ();
 
 
// Wait until the sub-thread ends
// Int count = 0;
// Whether to end the while LOOP
Boolean breakWhile = false;
 
While (! BStop)
{
Write_nPos ();
Utility. sleep (500 );
BreakWhile = true;
 
For (int I = 0; I & lt; nStartPos. length; I ++)
{
If (! FileSplitterFetch [I]. bDownOver)
{
BreakWhile = false;
Break;
}
}
If (breakWhile)
Break;
 
// Count ++;
// If (count & gt; 4)
// SiteStop ();
}
 
System. err. println ("the File Download is complete! ");
}
Catch (Exception e) {e. printStackTrace ();}
}
 
// Obtain the object Length
Public long getFileSize ()
{
Int nFileLength =-1;
Try {
URL url = new URL (siteInfoBean. getSSiteURL ());
HttpURLConnection httpConnection = (HttpURLConnection) url. openConnection ();
HttpConnection. setRequestProperty ("User-Agent", "NetFox ");
 
Int responseCode = httpConnection. getResponseCode ();
If (responseCode & gt ;= 400)
{
ProcessErrorCode (responseCode );
Return-2; //-2 represent access is error
}
 
String sHeader;
 
For (int I = 1; I ++)
{
// DataInputStream in = new DataInputStream (httpConnection. getInputStream ());
// Utility. log (in. readLine ());
SHeader = httpConnection. getHeaderFieldKey (I );
If (sHeader! = Null)
{
If (sHeader. equals ("Content-Length "))
{
NFileLength = Integer. parseInt (httpConnection. getHeaderField (sHeader ));
Break;
}
}
Else
Break;
}
}
Catch (IOException e) {e. printStackTrace ();}
Catch (Exception e) {e. printStackTrace ();}
 
Utility. log (nFileLength );
 
Return nFileLength;
}
 
// Save the download information (File pointer location)
Private void write_nPos ()
{
Try {
Output = new DataOutputStream (new FileOutputStream (tmpFile ));
Output. writeInt (nStartPos. length );
For (int I = 0; I & lt; nStartPos. length; I ++)
{
// Output. writeLong (nPos [I]);
Output. writeLong (fileSplitterFetch [I]. nStartPos );
Output. writeLong (fileSplitterFetch [I]. nEndPos );
}
Output. close ();
}
Catch (IOException e) {e. printStackTrace ();}
Catch (Exception e) {e. printStackTrace ();}
}
 
// Read the saved download information (File pointer location)
Private void read_nPos ()
{
Try {
DataInputStream input = new DataInputStream (new FileInputStream (tmpFile ));
Int nCount = input. readInt ();
NStartPos = new long [nCount];
NEndPos = new long [nCount];
For (int I = 0; I & lt; nStartPos. length; I ++)
{
NStartPos [I] = input. readLong ();
NEndPos [I] = input. readLong ();
}
Input. close ();
}
Catch (IOException e) {e. printStackTrace ();}
Catch (Exception e) {e. printStackTrace ();}
}
 
Private void processErrorCode (int nErrorCode)
{
System. err. println ("Error Code:" + nErrorCode );
}
 
// Stop object download
Public void siteStop ()
{
BStop = true;
For (int I = 0; I & lt; nStartPos. length; I ++)
FileSplitterFetch [I]. splitterStop ();
 
}
}
/*
** FileSplitterFetch. java
*/
Package NetFox;
 
Import java. io .*;
Import java.net .*;
 
Public class FileSplitterFetch extends Thread {
 
String sURL; // File URL
Long nStartPos; // File Snippet Start Position
Long nEndPos; // File Snippet End Position
Int nThreadID; // Thread's ID
Boolean bDownOver = false; // Downing is over
Boolean bStop = false; // Stop identical
FileAccessI fileAccessI = null; // File Access interface
 
Public FileSplitterFetch (String sURL, String sName, long nStart, long nEnd, int id) throws IOException
{
This. sURL = sURL;
This. nStartPos = nStart;
This. nEndPos = nEnd;
NThreadID = id;
FileAccessI = new FileAccessI (sName, nStartPos );
}
 
Public void run ()
{
While (nStartPos & lt; nEndPos &&! BStop)
{
 
Try {
URL url = new URL (sURL );
HttpURLConnection httpConnection = (HttpURLConnection) url. openConnection ();
HttpConnection. setRequestProperty ("User-Agent", "NetFox ");
String sProperty = "bytes =" + nStartPos + "-";
HttpConnection. setRequestProperty ("RANGE", sProperty );
Utility. log (sProperty );
 
InputStream input = httpConnection. getInputStream ();
// LogResponseHead (httpConnection );
 
 
Byte [] B = new byte [1024];
Int nRead;
While (nRead = input. read (B, 0,1024) & gt; 0 & nStartPos & lt; nEndPos &&! BStop)
{
NStartPos + = fileAccessI. write (B, 0, nRead );
// If (nThreadID = 1)
// Utility. log ("nStartPos =" + nStartPos + ", nEndPos =" + nEndPos );
}
 
Utility. log ("Thread" + nThreadID + "is over! ");
BDownOver = true;
// NPos = fileAccessI. write (B, 0, nRead );
}
Catch (Exception e) {e. printStackTrace ();}
}
}
 
// Print the Response Header
Public void logResponseHead (HttpURLConnection con)
{
For (int I = 1; I ++)
{
String header = con. getHeaderFieldKey (I );
If (header! = Null)
// ResponseHeaders. put (header, httpConnection. getHeaderField (header ));
Utility. log (header + ":" + con. getHeaderField (header ));
Else
Break;
}
}
 
Public void splitterStop ()
{
BStop = true;
}
 
}
 
/*
** FileAccess. java
*/
Package NetFox;
Import java. io .*;
 
Public class FileAccessI implements Serializable {
 
RandomAccessFile oSavedFile;
Long nPos;
 
Public FileAccessI () throws IOException
{
This ("", 0 );
}
 
Public FileAccessI (String sName, long nPos) throws IOException
{
OSavedFile = new RandomAccessFile (sName, "rw ");
This. nPos = nPos;
OSavedFile. seek (nPos );
}
 
Public synchronized int write (byte [] B, int nStart, int nLen)
{
Int n =-1;
Try {
OSavedFile. write (B, nStart, nLen );
N = nLen;
}
Catch (IOException e)
{
E. printStackTrace ();
}
 
Return n;
}
 
}
 
/*
** SiteInfoBean. java
*/
Package NetFox;
 
Public class SiteInfoBean {
 
Private String sSiteURL; // Site's URL
Private String sFilePath; // Saved File's Path
Private String sFileName; // Saved File's Name
Private int nSplitter; // Count of Splited Downloading File
 
Public SiteInfoBean ()
{
// Default value of nSplitter is 5
This ("", 5 );
}
 
Public SiteInfoBean (String sURL, String sPath, String sName, int nSpiltter)
{
SSiteURL = sURL;
SFilePath = sPath;
SFileName = sName;
This. nSplitter = nSpiltter;
 
}
 
Public String getSSiteURL ()
{
Return sSiteURL;
}
 
Public void setSSiteURL (String value)
{
SSiteURL = value;
}
 
Public String getSFilePath ()
{
Return sFilePath;
}
 
Public void setSFilePath (String value)
{
SFilePath = value;
}
 
Public String getSFileName ()
{
Return sFileName;
}
 
Public void setSFileName (String value)
{
SFileName = value;
}
 
Public int getNSplitter ()
{
Return nSplitter;
}
 
Public void setNSplitter (int nCount)
{
NSplitter = nCount;
}
}
 
/*
** Utility. java
*/
Package NetFox;
 
Public class Utility {
 
Public Utility ()
{
 
}
 
Public static void sleep (int nSecond)
{
Try {
Thread. sleep (nSecond );
}
Catch (Exception e)
{
E. printStackTrace ();
}
}
 
Public static void log (String sMsg)
{
System. err. println (sMsg );
}
 
Public static void log (int sMsg)
{
System. err. println (sMsg );
}
}
 
/*
** TestMethod. java
*/
Package NetFox;
 
Public class TestMethod {
 
Public TestMethod ()
{// Xx/weblogic60b2_win.exe
Try {
SiteInfoBean bean = new SiteInfoBean ("http: // localhost/xx/weblogic60b2_win.exe", "L: \ temp", "weblogic60b2_win.exe", 5 );
// SiteInfoBean bean = new SiteInfoBean ("http: // localhost: 8080/down.zip", "L: \ temp", "weblogic60b2_win.exe", 5 );
SiteFileFetch fileFetch = new SiteFileFetch (bean );
FileFetch. start ();
}
Catch (Exception e) {e. printStackTrace ();
}
}
 
Public static void main (String [] args)
{
New TestMethod ();
}
}