ASP. NET large file upload Problems

1. solution:

. NET large File Upload knowledge

Recently, I am working on the ePartner project, which involves file upload. I have also uploaded files before, but they are all small files, which cannot exceed 2 MB. Upload more than MB. I couldn't find any information to study it. For WEB-based file uploads, FTP and HTTP protocols can be used. Although FTP is stable in transmission, security is a serious problem. In addition, the FTP server reads the user database to obtain permissions, this is not convenient for users. Only HTTP is left. There are three methods in HTTP: PUT, WEBDAV, and RFC1867. The first two methods are not suitable for uploading large files, currently, we use form-based file uploads Based on RFC1867 standard HTML.

I. Briefly introduce RFC1867 (Form-based File Upload in HTML) standards:
1. HTML form with file submission Function
The existing HTML Specification defines eight possible values for the TYPE attribute of the INPUT element: CHECKBOX, HIDDEN, IMAGE, PASSWORD, RADIO, RESET, SUBMIT, TEXT. in addition, when the form uses the POST method, the form has the "application/x-www-form-urlencoded" ENCTYPE attribute by default.

RFC1867 made two changes to HTML:
1) added a FILE option for the TYPE attribute of the INPUT element.
2) The INPUT tag can have the ACCEPT attribute, which can specify the file type or file format list that can be uploaded.

In addition, this standard defines a new MIME type: multipart/form-data, and when processingActions to be taken when marking the form.

For example, when the author of an HTML form wants the user to upload one or more files, he can write as follows:

In the "name" field, enter "Joe Blow". What files are you sending? ', Select
A text file "file1.txt ".
The customer segment may send back the following data:
Content-type: multipart/form-data, boundary = AaB03x

-- AaB03x
Content-disposition: form-data; name = "field1"

Joe Blow
-- AaB03x
Content-disposition: form-data; name = "pics"; filename = "file1.txt"
Content-Type: text/plain

Contents of... file1.txt...
-- AaB03x --
If you select another image file "file2.gif", the client may send the following data:
Content-type: multipart/form-data, boundary = AaB03x

-- AaB03x
Content-disposition: form-data; name = "field1"

Joe Blow
-- AaB03x
Content-disposition: form-data; name = "pics"
Content-type: multipart/mixed, boundary = BbC04y

-- BbC04y
Content-disposition: attachment; filename = "file1.txt"

Content-Type: text/plain

Contents of... file1.txt...
-- BbC04y
Content-disposition: attachment; filename = "file2.gif"
Content-type: image/gif
Content-Transfer-Encoding: binary

... File2.gif content...
-- BbC04y --
-- AaB03x --

Ii. Two methods for processing file uploads using RFC1867:
1. Get the uploaded data at one time and analyze and process it.
After reading N multiple codes, I found that currently no component program and some COM components use the Request. BinaryRead method. Obtain the uploaded data at a time, and then analyze and process it. This is why the upload of large files is slow. If IIS times out, even if hundreds of MB of files are uploaded, the analysis takes a while.
2. Write to the hard disk while receiving files.

I have learned about commercial components outside China. Some popular components include Power-Web, AspUpload, ActiveFile, ABCUpload, aspSmartUpload, and SA-FileUp. Among them, the better is ASPUPLOAD and SA-FILE, they claim to be able to process 2 GB of files (SA-FILE EE edition or even no file size restrictions), and the efficiency is also very good, is the programming language so much less efficient? I checked some information and thought they were all directly operating the file stream. In this way, the file size is not restricted. However, it is not absolutely perfect for foreigners. After ASPUPLOAD processes large files, the memory usage is astonishing. Around 1 GB is common. As for SA-FILE although it is good but difficult to find the crack. Then we found two. NET Upload components, Lion. Web. UpLoadModule and AspnetUpload, which are also operation file streams. However, the upload speed and CPU usage are not as good as the commercial components of foreigners.
A test was conducted to upload 1 GB files in the LAN. The average upload speed of ASPUPLOAD is 4.4 Mb/s, the CPU usage is 10-15, and the memory usage is 700 mb. SA-FILE is almost like this. The AspnetUpload is only 1.5 M/s at the fastest, with an average of 700 K/s and CPU usage of 15-39. test environment: PIII800, 100 M memory, and m lan. I think the slow speed of AspnetUpload may be caused by hard disk writing while receiving files. The low cost of resource occupation is to reduce the transmission speed. But I also have to admire the foreign program. The CPU usage is so low .....

Iii. ASP. NET File Upload Problems
We have encountered one or more problems when uploading large files using ASP. NET. Setting a large value of maxRequestLength does not completely solve the problem, because ASP. NET blocks the entire file until it is loaded into the memory and then processes it. In fact, if The file is large, we often see that Internet Explorer displays "The page cannot be displayed-Cannot find server or DNS Error". It seems that this Error cannot be caught. Why? Because this is a client side error, Application_Error on the server side cannot be handled.
Iv. ASP. NET large file upload Solution
The solution is to use the implicit HttpWorkerRequest and its GetPreloadedEntityBody and ReadEntityBody methods to read data from the pipe created by IIS for ASP. NET in blocks. Chris Hynes provides us with such a solution (using HttpModule) that allows you to upload large files and display the upload progress in real time.
Lion. Web. UpLoadModule and AspnetUpload both use this solution.

Solution Principle:
HttpHandler is used to implement functions similar to ISAPI Extention, process Request information and send Response ).

Solution highlights:
1. httpHandler or HttpModule
A. The request object is intercepted before the asp.net process processes the request.
B. Read and Write Data in Parts
C. Track the upload progress in real time and update the meta information.
2. Use the implicit HttpWorkerRequest to process the file stream using its GetPreloadedEntityBody and ReadEntityBody methods.
IServiceProvider provider = (IServiceProvider) HttpContext. Current;
HttpWorkerRequest wr = (HttpWorkerRequest) provider. GetService (typeof (HttpWorkerRequest ));
Byte [] bs = wr. GetPreloadedEntityBody ();
....
If (! Wr. IsEntireEntityBodyIsPreloaded ())
{
Int n = 1024;
Byte [] bs2 = new byte;
While (wr. ReadEntityBody (bs2, n)> 0)
{
.....
}
}
3. Custom Multipart MIME parser
Automatically intercepts MIME delimiters
Write a file into blocks such as temporary files.
Update Appliaction status in real time (ReceivingData, Error, Complete)

/Example
HttpApplication application1 = sender as HttpApplication;
HttpWorkerRequest request1 = (HttpWorkerRequest) (IServiceProvider) HttpContext. Current). GetService (typeof (HttpWorkerRequest ));
Try
{
If (application1.Context. Request. ContentType. IndexOf ("multipart/form-data") <=-1)
{
Return;
}
// Check The HasEntityBody
If (! Request1.HasEntityBody ())
{
Return;
}
Int num1 = 0;
TimeSpan span1 = DateTime. Now. Subtract (this. beginTime );

String text1 = application1.Context. Request. ContentType. ToLower ();

Byte [] buffer1 = Encoding. ASCII. GetBytes ("\ r \ n --" + text1.Substring (text1.IndexOf ("boundary =") + 9). ToCharArray ());
Int num2 = Convert. ToInt32 (request1.GetKnownRequestHeader (11 ));
Progress progress1 = new Progress ();

Application1.Context. Items. Add ("FileList", new Hashtable ());

Byte [] buffer2 = request1.GetPreloadedEntityBody ();
Num1 + = buffer2.Length;

String text2 = this. AnalysePreloadedEntityBody (buffer2, "UploadGUID ");
If (text2! = String. Empty)
{
Application1.Context. Items. Add ("LionSky_UpLoadModule_UploadGUID", text2 );
}
Bool flag1 = true;
If (num2> this. UpLoadFileLength () & (0> span1.TotalHours) | (span1.TotalHours> 3 )))
{
Flag1 = false;
}
If (0> span1.TotalHours) | (span1.TotalHours> 3 ))
{
Flag1 = false;
}
String text3 = this. AnalysePreloadedEntityBody (buffer2, "UploadFolder ");
ArrayList list1 = new ArrayList ();
RequestStream stream1 = new RequestStream (buffer2, buffer1, null, RequestStream. FileStatus. Close, RequestStream. ReadStatus. NoRead, text3, flag1, application1.Context, string. Empty );
List1.AddRange (stream1.ReadBody );
If (text2! = String. Empty)
{
Progress1.FileLength = num2;
Progress1.ededlength = num1;
Progress1.FileName = stream1.OriginalFileName;
Progress1.FileCount = (Hashtable) application1.Context. Items ["FileList"]). Count;
Application1.Application ["_ UploadGUID _" + text2] = progress1;
}

If (! Request1.IsEntireEntityBodyIsPreloaded ())
{
Byte [] buffer4;
ArrayList list2;
Int Number3 = 204800;
Byte [] buffer3 = new byte [num3];
While (num2-num1)> = num3)
{
If (! Application1.Context. Response. IsClientConnected)
{
This. ClearApplication (application1 );
}
Num3 = request1.ReadEntityBody (buffer3, buffer3.Length );
Num1 + = num3;
List2 = stream1.ContentBody;
If (list2.Count> 0)
{
Buffer4 = new byte [list2.Count + buffer3.Length];
List2.CopyTo (buffer4, 0 );
Buffer3.CopyTo (buffer4, list2.Count );
Stream1 = new RequestStream (buffer4, buffer1, stream1.FileStream, stream1.FStatus, stream1.RStatus, text3, flag1, application1.Context, stream1.OriginalFileName );
}
Else
{
Stream1 = new RequestStream (buffer3, buffer1, stream1.FileStream, stream1.FStatus, stream1.RStatus, text3, flag1, application1.Context, stream1.OriginalFileName );
}
List1.AddRange (stream1.ReadBody );
If (text2! = String. Empty)
{
Progress1.ededlength = num1;
Progress1.FileName = stream1.OriginalFileName;
Progress1.FileCount = (Hashtable) application1.Context. Items ["FileList"]). Count;
Application1.Application ["_ UploadGUID _" + text2] = progress1;
}
}
Buffer3 = new byte [num2-num1];
If (! Application1.Context. Response. IsClientConnected & (stream1.FStatus = RequestStream. FileStatus. Open ))
{
This. ClearApplication (application1 );
}
Num3 = request1.ReadEntityBody (buffer3, buffer3.Length );
List2 = stream1.ContentBody;
If (list2.Count> 0)
{
Buffer4 = new byte [list2.Count + buffer3.Length];
List2.CopyTo (buffer4, 0 );
Buffer3.CopyTo (buffer4, list2.Count );
Stream1 = new RequestStream (buffer4, buffer1, stream1.FileStream, stream1.FStatus, stream1.RStatus, text3, flag1, application1.Context, stream1.OriginalFileName );
}
Else
{
Stream1 = new RequestStream (buffer3, buffer1, stream1.FileStream, stream1.FStatus, stream1.RStatus, text3, flag1, application1.Context, stream1.OriginalFileName );
}
List1.AddRange (stream1.ReadBody );
If (text2! = String. Empty)
{
Progress1.ReceivedLength = num1 + buffer3.Length;
Progress1.FileName = stream1.OriginalFileName;
Progress1.FileCount = (Hashtable) application1.Context. Items ["FileList"]). Count;
If (flag1)
{
Progress1.UploadStatus = Progress. UploadStatusEnum. Uploaded;
}
Else
{
Application1.Application. Remove ("_ UploadGUID _" + text2 );
}
}
}
Byte [] buffer5 = new byte [list1.Count];
List1.CopyTo (buffer5 );
This. PopulateRequestData (request1, buffer5 );
}
Catch (Exception exception1)
{
This. ClearApplication (application1 );
Throw exception1;
}