[Go] How to implement a large file breakpoint upload in C #

Recently done in a project that involves file upload issues. File uploads have also been done before. But they are small files, not more than 2m. This request to upload more than 1g of things. There's no way to get a data study. Web-based file upload can use FTP and HTTP two protocol, with the FTP word although the transmission is stable, but security is a serious problem, so no consideration. Only HTTP is left. In HTTP there are 3 ways, put, WebDAV, rfc1867, the first 2 methods are not suitable for large file upload, here also do not say.

Be sure to start analyzing popular upload components after using rfc1867 format processing. After looking at more than n code, it is found that no component programs and some COM components are using the Request.BinaryRead method. Get uploaded data at once, then analyze and process. This is why uploading large files is very slow, IIS time-out does not say, even if the 1g file up, analysis processing will be a while. Then I focus on foreign business components, the more popular power-web,aspupload,activefile,abcupload,aspsmartupload,sa-fileup. One of the better is aspupload and sa-file, they claim to be able to handle 2g files (sa-file EE version even without file size restrictions), and the efficiency is very good, is the programming language inefficient so much? (My programming environment is VB6) looked up some information and felt that they were all directly manipulating the file stream. This will not be subject to file size constraints. That's a good way.

But the foreigner's things are not absolutely perfect, aspupload processing large files, memory consumption is amazing. 1g or so is commonplace. I'm using the 3.0.0.3 version. As for Sa-file although is good thing but cracked hard to find (depressed die) disappointment, found that 2 upload components, lion.web.uploadmodule and aspnetupload, are. NET, it is estimated that the operation of the file stream. However, the upload speed and CPU usage are not as good as the commercial components of foreigners.

Did a test, LAN incoming 1g files. AspUpload upload speed average is 4.4m/s,cpu occupy 10-15, Memory occupies 700m. Sa-file is almost like that. And aspnetupload the fastest also only 1.5m/s, average is 700k/s,cpu occupy 15-39, test environment: piii800,256m memory, 100m LAN. I think aspnetupload slow is possible because while receiving files, while writing hard disk. The cost of low resource consumption is to reduce transmission speed. But also have to admire the foreigner's program, CPU occupied so low ..... This way, 2. NET components are also pass.

A few 2 questions are upload progress and breakpoint continuation.
Display upload progress is relatively simple, mainly to query the status of users upload, script display to the browser, as for no refresh display is to see the proficiency of scripting language use.
The breakpoint continues to pass, the HTTP method is not possible, because every time the browser uploads the file is from the beginning, no range tag. The implemented method can only be used with ActiveX.


After the study decided to write a CGI to deal with file upload. This allows you to avoid IIS and prevent program errors from affecting site access. Younger brother compared dishes can only use VB6 do, after completion found that win CGI efficiency is simply poor can no longer. Simply write a file server that specifically handles uploading of files. But now there's a 2 question.

One, with the Winsock control received text garbled do not know is the program conversion error or Winsock itself garbage, so changed the POWERTCP Winsock tool, the situation has improved garbled not so much ... Ready to change vb.net, direct operation of the socket, the program has not been done, do not know with. NET receive will not garbled. And then I cried.

Second, this problem is more elementary .... The received file stream cannot be restored to a file. Cold One,


Finally, how to efficiently process the file stream, I want to go there are only 2 ways, one is put in memory, and then processed together, the second is to receive one side to write files. But none of the 2 methods are satisfactory .

Before we understand the principle of HTTP breakpoint continuation, let us first understand the HTTP protocol, the HTTP protocol is a simple TCP-based protocol, divided into two types of request and reply. A request protocol is a protocol that sends a message when a client (browser) submits a request to the server (WEB server). The reply protocol is the Protocol that is provided by the server (Web server) when replying to a message to the client (browser). Both the request and the reply agreement are made up of the headers and bodies. The head and body are separated by a line of empty behavior.
Here is an example of a request message with the corresponding reply message:

Get/image/index_r4_c1.jpg http/1.1 Accept: */* referer:http://192.168.3.120:8080 Accept-language:zh-cn Accept-encoding:gzip, deflate user-agent:mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;. NET CLR 1.0.3705) host:192.168.3.120:8080 Connection:keep-alive http/1.1 OK server:microsoft-iis/5.0 Date:tue, June 2003 05:39:40 GMT Content-type:image/jpeg Accept-ranges:bytes Last-modified:thu, 2002 03:05:40 GMT ETag: "bec48eb862c21:934" content-length:2827 ....

Here we say "breakpoint continuation", as the name implies, the breakpoint continues to be in the last download, the location of the disconnection began to continue to download.
In the HTTP protocol, a range segment can be added to the request header to indicate where the client wants to proceed with the download.

For example, starting with the 1024th byte download, the request message is as follows:

get/image/index_r4_c1.jpg http/1.1 Accept: */* Referer: http://192.168.3.120:8080 ACCEPT-LANGUAGE:ZH-CN Accept-encoding:gzip, deflate user-agent:mozilla/4.0 ( Compatible MSIE 6.0; Windows NT 5.0;. NET CLR 1.0.3705) host:192.168.3.120:8080 range:bytes=1024- connection:keep-alive

. The related classes in net

Understanding the above principles, let's look at what classes are available in the. NET framework to do these things.

Completing the HTTP request

System.Net.HttpWebRequest

The HttpWebRequest class provides support for properties and methods defined in WebRequest, as well as additional properties and methods that enable users to interact directly with servers that use HTTP.

HttpWebRequest public HTTP header values that are sent to an Internet resource are exposed as properties, set by method or system. The following table contains the full list. You can set other headers in the Headers property to name/value pairs. Note, however, that some public headers are considered restricted and are either exposed directly by the API or protected by the system and cannot be changed. Range also belongs to the protected column, however. NET provides developers with a more convenient way to do this is to add a AddRange method to the request, adding a specific range of byte-range headers from the beginning or end of the request data

Complete file access

System.IO.FileStream

The FileStream object supports random access to a file using the Seek method, and seek allows the read/write location to be moved to any location in the file. This is done through the byte offset reference point parameter. The byte offset is relative to the lookup reference point, which can be the beginning, current position, or end of the underlying file, represented by three properties of the SeekOrigin class, respectively.

Code implementation

Understand. NET provides the related classes, then, we can easily implement.

The code is as follows:

static void Main (string[] args) { String Strfilename= "C://aa.zip"; Set according to the actual situation String strurl= "Http://www.xxxx.cn/xxxxx.zip"; Set according to the actual situation Open the last downloaded file or create a new file Long Lstartpos = 0; System.IO.FileStream FS; if (System.IO.File.Exists (strFileName)) { fs= System.IO.File.OpenWrite (strFileName); Lstartpos=fs. Length; Fs. Seek (lstartpos,system.io.seekorigin.current); Move the current pointer in a file stream } Else { FS = new System.IO.FileStream (strfilename,system.io.filemode.create); Lstartpos = 0; } Open Network Connection Try { System.Net.HttpWebRequest request = (System.Net.HttpWebRequest) System.Net.HttpWebRequest.Create (strURL); if (lstartpos>0) Request. AddRange ((int) lstartpos); Set Range value Request to server for server response data flow System.IO.Stream ns= request. GetResponse (). GetResponseStream (); byte[] nbytes = new byte[512]; int nreadsize=0; Nreadsize=ns. Read (nbytes,0,512); while (Nreadsize >0) { Fs. Write (nbytes,0,nreadsize); Nreadsize=ns. Read (nbytes,0,512); } Fs. Close (); Ns. Close (); Console.WriteLine ("Download Complete"); } catch (Exception ex) { Fs. Close (); Console.WriteLine ("Error occurred during download:" +ex. ToString ()); } }

[Go] How to implement a large file breakpoint upload in C #