Httpworkerrequest for uploading large files to Asp.net

Httpworkerrequest can be used to upload large files
I have also uploaded files before, but they are all small files, which cannot exceed 2 MB. Upload more than MB. I couldn't find any information to study it. For web-based file uploads, FTP and HTTP protocols can be used. Although FTP is stable in transmission, security is a serious problem. In addition, the FTP server reads the user database to obtain permissions, this is not convenient for users. Only HTTP is left. There are three methods in http: Put, WebDAV, and rfc1867. The first two methods are not suitable for uploading large files, currently, we use form-based file uploads Based on rfc1867 standard HTML.
I. Briefly introduce rfc1867 (form-based file upload in HTML) standards:
1. HTML form with file submission Function
The current HTML Specification defines eight possible values for the Type attribute of the input element: checkbox, hidden, image, password, radio, reset, submit, text. in addition, when the form uses the POST method, the form has the "Application/X-WWW-form-urlencoded" enctype attribute by default.
Rfc1867 made two changes to HTML :.....
.....
ASP. NET large file upload Solution
The solution is to use the implicit httpworkerrequest and its getpreloadedentitybody and readentitybody methods to read data from the pipe created by IIS for ASP. NET in blocks. Chris Hynes provides us with such a solution (using httpmodule) that allows you to upload large files and display the upload progress in real time.
Lion. Web. uploadmodule and aspnetupload both use this solution.
Solution Principle:
Httphandler is used to implement functions similar to ISAPI extention, process request information and send response ).
Solution highlights:
1. httphandler or httpmodule
A. The request object is intercepted before the Asp.net process processes the request.
B. Read and Write Data in Parts
C. Track the upload progress in real time and update the meta information.
2. Use the implicit httpworkerrequest to process the file stream using its getpreloadedentitybody and readentitybody methods.
Iserviceprovider provider = (iserviceprovider) httpcontext. Current;
Httpworkerrequest wR = (httpworkerrequest) provider. getservice (typeof (httpworkerrequest ));
Byte [] BS = Wr. getpreloadedentitybody ();
....
If (! Wr. isentireentitybodyispreloaded ())
{
Int n = 1024;
Byte [] bs2 = new byte;
While (WR. readentitybody (bs2, n)> 0)
{
.....
}
}
3. Custom multipart mime parser
Automatically intercepts mime delimiters
Write a file into blocks such as temporary files.
Update Appliaction status in real time (receivingdata, error, complete)
/Example
Httpapplication application1 = sender as httpapplication;
Httpworkerrequest request1 = (httpworkerrequest) (iserviceprovider) httpcontext. Current). getservice (typeof (httpworkerrequest ));
Try
{
If (application1.context. Request. contenttype. indexof ("multipart/form-Data") <=-1)
{
Return;
}
// Check the hasentitybody
If (! Request1.hasentitybody ())
{
Return;
}
Int num1 = 0;
Timespan span1 = datetime. Now. Subtract (this. begintime );
String text1 = application1.context. Request. contenttype. tolower ();
Byte [] buffer1 = encoding. ASCII. getbytes ("\ r \ n --" + text1.substring (text1.indexof ("boundary =") + 9). tochararray ());
Int num2 = convert. toint32 (request1.getknownrequestheader (11 ));
Progress progress1 = new progress ();
Application1.context. Items. Add ("filelist", new hashtable ());
Byte [] buffer2 = request1.getpreloadedentitybody ();
Num1 + = buffer2.length;
String text2 = This. analysepreloadedentitybody (buffer2, "uploadguid ");
If (text2! = String. Empty)
{
Application1.context. Items. Add ("lionsky_uploadmodule_uploadguid", text2 );
}
Bool flag1 = true;
If (num2> This. uploadfilelength () & (0> span1.totalhours) | (span1.totalhours> 3 )))
{
Flag1 = false;
}
If (0> span1.totalhours) | (span1.totalhours> 3 ))
{
Flag1 = false;
}
String text3 = This. analysepreloadedentitybody (buffer2, "uploadfolder ");
Arraylist list1 = new arraylist ();
Requeststream stream1 = new requeststream (buffer2, buffer1, null, requeststream. filestatus. Close, requeststream. readstatus. noread, text3, flag1, application1.context, String. Empty );
List1.addrange (stream1.readbody );
If (text2! = String. Empty)
{
Progress1.filelength = num2;
Progress1.ededlength = num1;
Progress1.filename = stream1.originalfilename;
Progress1.filecount = (hashtable) application1.context. items ["filelist"]). count;
Application1.application ["_ uploadguid _" + text2] = progress1;
}

If (! Request1.isentireentitybodyispreloaded ())
{
Byte [] buffer4;
Arraylist list2;
Int Number3 = 204800;
Byte [] buffer3 = new byte [num3];
While (num2-num1)> = num3)
{
If (! Application1.context. response. isclientconnected)
{
This. clearapplication (application1 );
}
Num3 = request1.readentitybody (buffer3, buffer3.length );
Num1 + = num3;
List2 = stream1.contentbody;
If (list2.count> 0)
{
Buffer4 = new byte [list2.count + buffer3.length];
List2.copyto (buffer4, 0 );
Buffer3.copyto (buffer4, list2.count );
Stream1 = new requeststream (buffer4, buffer1, stream1.filestream, stream1.fstatus, stream1.rstatus, text3, flag1, application1.context, stream1.originalfilename );
}
Else
{
Stream1 = new requeststream (buffer3, buffer1, stream1.filestream, stream1.fstatus, stream1.rstatus, text3, flag1, application1.context, stream1.originalfilename );
}
List1.addrange (stream1.readbody );
If (text2! = String. Empty)
{
Progress1.ededlength = num1;
Progress1.filename = stream1.originalfilename;
Progress1.filecount = (hashtable) application1.context. items ["filelist"]). count;
Application1.application ["_ uploadguid _" + text2] = progress1;
}
}
Buffer3 = new byte [num2-num1];
If (! Application1.context. response. isclientconnected & (stream1.fstatus = requeststream. filestatus. Open ))
{
This. clearapplication (application1 );
}
Num3 = request1.readentitybody (buffer3, buffer3.length );
List2 = stream1.contentbody;
If (list2.count> 0)
{
Buffer4 = new byte [list2.count + buffer3.length];
List2.copyto (buffer4, 0 );
Buffer3.copyto (buffer4, list2.count );
Stream1 = new requeststream (buffer4, buffer1, stream1.filestream, stream1.fstatus, stream1.rstatus, text3, flag1, application1.context, stream1.originalfilename );
}
Else
{
Stream1 = new requeststream (buffer3, buffer1, stream1.filestream, stream1.fstatus, stream1.rstatus, text3, flag1, application1.context, stream1.originalfilename );
}
List1.addrange (stream1.readbody );
If (text2! = String. Empty)
{
Progress1.receivedlength = num1 + buffer3.length;
Progress1.filename = stream1.originalfilename;
Progress1.filecount = (hashtable) application1.context. items ["filelist"]). count;
If (flag1)
{
Progress1.uploadstatus = progress. uploadstatusenum. Uploaded;
}
Else
{
Application1.application. Remove ("_ uploadguid _" + text2 );
}
}
}
Byte [] buffer5 = new byte [list1.count];
List1.copyto (buffer5 );
This. populaterequestdata (request1, buffer5 );
}
Catch (exception exception1)
{
This. clearapplication (application1 );
Throw exception1;

}

The traditional method of uploading defective clients only submits data and file streams. It seems that there should be no difference. However, the biggest difference between IE and Firefox is that uploading files via IE will not increase the memory usage of the client, firefox needs to constantly read the file content into the memory before sending it. For example, if you use Firefox to upload a file client of MB, the memory consumption will increase by MB. It is surprising that Firefox has such a defect!
The IIS server is very cautious when processing file uploads. During the upload process, the server will not consume memory because the submitted data is a large file. Therefore, it is suggested that IIS stores data directly to a temporary file after the pipe is created to read data. However, the fatal problem is that our programmers need to process the uploaded files, and Asp.net needs to hand over the real file stream to you for processing. The final result is ASP. the net process will still load the entire file content to you for processing, such as in the bitaec framework.
It is also worth mentioning that ASP. the fileupload control in. net, as I analyzed above, when fileupload uploads a file, IIS constantly reads data from pipe and saves it to a temporary file, which will not increase the server content; when the upload is complete, you directly call the saveas method in fileupload to directly store the file to the location you need. The actual operation is to move the received temporary file to the location you need, therefore, it does not increase the consumption of server memory resources. However, if you need to operate on the uploaded file stream, it will increase the memory consumption because it needs to reload the temporary file to the memory.
The large file upload Solution uses the rfc1867 standard to process File Uploads in two ways:
1. Get the uploaded data at one time and analyze and process it.
After reading n multiple codes, I found that currently no component program and some COM components use the request. binaryread method. Obtain the uploaded data at a time, and then analyze and process it. This is why the upload of large files is slow. If IIS times out, even if hundreds of MB of files are uploaded, the analysis takes a while.
2. Write to the hard disk while receiving files.

Some of the commercial components outside China are widely used, including power-web, aspupload, activefile, abcupload, aspsmartupload, and Sa-fileup. Among them, the better is aspupload and SA-FILE, they claim to be able to process 2 GB of files (SA-FILE EE edition or even no file size restrictions), and the efficiency is also very good, is the programming language so much less efficient? I checked some information and thought they were all directly operating the file stream. In this way, the file size is not equal. However, it is not absolutely perfect for foreigners. After aspupload processes large files, the memory usage is astonishing. Around 1 GB is common. As for SA-FILE although it is good but difficult to find crack. Then we found two. Net Upload components, lion. Web. uploadmodule and aspnetupload, which are also operation file streams. However, the upload speed and CPU usage are not as good as the commercial components of foreigners.
A test was conducted to upload 1 GB files in the LAN. The average upload speed of aspupload is 4.4 Mb/s, the CPU usage is 10-15, and the memory usage is 700 mb. SA-FILE is almost like this. The aspnetupload is only 1.5 m/s at the fastest, with an average of 700 K/s and CPU usage of 15-39. test environment: piii800, 100 m memory, and m lan. I think the slow speed of aspnetupload may be caused by hard disk writing while receiving files. The low cost of resource occupation is to reduce the transmission speed. But I also have to admire the foreign program. The CPU usage is so low .....

Iii. ASP. NET File Upload Problems
We have encountered one or more problems when uploading large files using ASP. NET. Setting a large value of maxrequestlength does not completely solve the problem, because ASP. net blocks the entire file until it is loaded into the memory and then processes it. In fact, if the file is large, we often see that Internet Explorer displays "the page cannot be displayed-cannot find server or DNS error". It seems that this error cannot be caught. Why? Because this is a client side error, application_error on the server side cannot be handled.
Iv. ASP. NET large file upload Solution
The solution is to use the implicit httpworkerrequest and its getpreloadedentitybody and readentitybody methods to read data from the pipe created by IIS for ASP. NET in blocks. Chris Hynes provides us with such a solution (using httpmodule) that allows you to upload large files and display the upload progress in real time.
Lion. Web. uploadmodule and aspnetupload both use this solution.

The solution is to use the implicit httpworkerrequest and its getpreloadedentitybody and readentitybody methods to read data from the pipe created by IIS for ASP. NET in blocks.
Iserviceprovider provider = (iserviceprovider) httpcontext. Current;
Httpworkerrequest wR = (httpworkerrequest) provider. getservice (typeof (httpworkerrequest ));
Byte [] BS = Wr. getpreloadedentitybody ();
....
If (! Wr. isentireentitybodyispreloaded ())
{
Int n = 1024;
Byte [] bs2 = new byte [N];
While (WR. readentitybody (bs2, n)> 0)
{
.....
}
}

There are several ways to upload large files:
1. httpworkerrequest Method
2. Use the third-party control aspnetupload for money !! Forget it. We still like free.
3. Modify the Web. config file, but the error cannot be captured.
4. Upload through FTP. The server must provide the FTP service.

Third party:
Modify the webcong file:
<System. Web>
& Lt; httpruntime maxrequestlength = "40690"
Usefullyqualifiedredirecturl = "true"
Executi
Usefullyqualifiedredirecturl = "false"
Minfreethreads = "8"
Minlocalrequestfreethreads = "4"
Apprequestqueuelimit = "100"
Enableversi
/>
</System. Web>

Which is closely related to upload:
Maxrequestlength
Indicates the maximum file upload size supported by ASP. NET.
This restriction can be used to prevent DoS attacks caused by a large number of files being transferred to the server.
The specified size is in KB.
The default value is 4096 KB (4 MB ).

Executiontimeout
Indicates the maximum number of seconds allowed to execute a request before it is automatically disabled by ASP. NET.
The Unit is seconds. When uploading a large file, this setting is larger.


If the server memory is 512 MB, files with a size of MB can be uploaded. (I have never tried it. Comments from all the posts on csdn .)
'Www .knowsky.com
The Web. config settings are now complete.
However, once the size of the uploaded file exceeds the configured file size range, the following error occurs:
This page cannot be displayed
The page you want to view is currently unavailable. The website may encounter technical problems, or you need to adjust the browser settings.

If this problem cannot be solved, capture the error! What should we do?
I have eaten a few fish recently and thought about it. Because this error is caused by a file control foreground error, it is not feasible to use try... catch in the background.
So I thought of using the. NET error capture page mechanism for processing. Feasible.

1. Set web. config first
<Customerrors mode = "on"/>
2. Create an error. aspx file to capture errors.
3. Add the page command on the front-end page of The ASPX page of the uploaded file. Errorpage = "uploaderror. aspx"
4. Add some code in error. aspx to determine whether the error message is a file-caused foreground error.
Public class uploaderror: system. Web. UI. Page
{
Private void page_load (Object sender, system. eventargs E)
{
Exception EX = server. getlasterror ();
If (ex! = NULL)
{
Response. Redirect (".../error. aspx ");
}
Else // foreground error ex is null
{
Response. Redirect ("uploadexcel. aspx? Err = 1 "); // redirect to the upload page again, and add the err parameter to display the error message.
}
}

5. An error message is displayed.
Public class uploadexcel: system. Web. UI. Page
{
Private void page_load (Object sender, system. eventargs E)
{
If (request ["Err"] = "1 ")
{
Page. registerstartupscript ("budget", "<script language = JavaScript> alert ('upload file has failed! File size is too large! ') </SCRIPT> ");
}
}
}