Explain how MongoDB accelerates the storage of physical files and SQUID

When reading the MongoDB GFS document, I learned how to store physical files, including large files. In order to deepen your impression, I wrote an example of uploading files to Mongodb. Of course, since it is stored in the document database, you cannot access these physical files in normal ways, here, An aspx page is specially written to read these files or MP3 files). Therefore, after downloading the example, you will see two pages: uploadfile. aspx (upload), one is getfile. aspx reads the file from mongodb as a stream ). Of course, considering the access speed, SQUID is introduced here to accelerate files. Currently SQUID only caches static files by default, so we need to set the output of the ASPX page here, these will be introduced in this article)

First, let's introduce the development environment. I use VS2008 + SP1, And the mongodb client uses the latest version of samus-mongodb)

Add reference to the following namespace

 
 

  
  
using MongoDB;  
  
  
using MongoDB.GridFS; 
 
 

The following is the main code for uploading files:

Uploadfile. aspx. cs

 
 

  
  
HttpPostedFile myFile = FileUpload.PostedFile;  
  
  
     int nFileLen = myFile.ContentLength;  
  
  
       
  
  
     byte[] myData = new Byte[nFileLen];  
  
  
     myFile.InputStream.Read(myData, 0, nFileLen);  
  
  
       
  
  
     GridFile fs = new GridFile(DB, filesystem);  
  
  
 
  
  
     Random random = new Random(unchecked((int)DateTime.Now.Ticks));  
  
  
 string newfilename = string.Format("{0}{1}{2}", random.Next(1000, 99999), 
  
  
random.Next(1000, 99999), System.IO.Path.GetExtension(myFile.FileName));  
  
  
     GridFileStream gfs = fs.Create(newfilename);  
  
  
     gfs.Write(myData, 0, nFileLen);  
  
  
     gfs.Close(); 
 
 

Here we just give a random name to the uploaded file, so that if everything is normal, you can find the file in the database, such:

Next let's take a look at how to access the uploaded physical file getfile. aspx. cs by passing the filename parameter, the corresponding field structure in mongodb, such ):

 
 

  
  
Protected void Page_Load (object sender, EventArgs e)
  
  
{
  
  
If (! String. IsNullOrEmpty (Request. QueryString ["filename"])
  
  
{
  
  
String filename = Request. QueryString ["filename"];
  
  
Init ();
  
  
String filesystem = "gfstream ";
  
  
 
  
  
GridFile fs = new GridFile (DB, filesystem );
  
  
GridFileStream gfs = fs. OpenRead (filename );
  
  
 
  
  
Byte [] buffer = new Byte [gfs. Length];
  
  
HttpContext. Current. Response. AddHeader ("Expires", DateTime. Now. AddDays (20). ToString ("r "));
  
  
HttpContext. Current. Response. AddHeader ("Cache-Control", "public ");
  
  

  
  
// The length of the data to be read
  
  
Long dataToRead = gfs. Length;
  
  
Int length;
  
  
While (dataToRead> 0)
  
  
{
  
  
// Check whether the client is still connected
  
  
If (HttpContext. Current. Response. IsClientConnected)
  
  
{
  
  
Length = gfs. Read (buffer, 0, 10000 );
  
  
HttpContext. Current. Response. OutputStream. Write (buffer, 0, length );
  
  
HttpContext. Current. Response. Flush ();
  
  
Buffer = new Byte [10000];
  
  
DataToRead = dataToRead-length;
  
  
}
  
  
Else
  
  
{
  
  
// Jump out of the endless loop if the connection is no longer available
  
  
DataToRead =-1;
  
  
}
  
  
}
  
  
Gfs. Dispose ();
  
  
HttpContext. Current. Response. End ();
  
  
}
  
  
}
 
 

The following shows the final running effect of retrieving the file list from mongodb in a list:

Although MONGODB has good concurrent performance, it also suffers performance loss every time it is retrieved from mongodb. Especially for physical files that do not change frequently, SQUID is used for file caching. Currently, SQUID only supports static files by default. You need to set the stream information output from the ASPX page in this example to cache.

First, if the squid. conf file contains the following lines, you need to use # To comment it. It will disable caching all? ):

 
 

  
  
hierarchy_stoplist cgi-bin ? \.php \.html  
  
  
acl QUERY urlpath_regex cgi-bin \? \.php \.html     
  
  
cache deny QUERY 
 
 

In this way, modify the corresponding. aspx and add the following information to the Header:

 
 

  
  
HttpContext.Current.Response.AddHeader("Expires", DateTime.Now.AddDays(20).ToString("r"));  
  
  
HttpContext.Current.Response.AddHeader("Cache-Control", "public"); 
 
 

In this way, SQUID will faithfully cache the corresponding file based on the header information.

Of course, you can also use the following method to make the specified aspx files be squid cached:

 
 

  
  
Acl CACHABLE_PAGES urlpath_regex \ getfile. aspx
  
  
# Allow the aspx page on the cache
  
  
No_cache allow CACHABLE_PAGES
 
 

The following acl matches all dynamic pages and disables cache of all aspx pages.

 
 

  
  
# Acl NONE_CACHABLE_PAGES urlpath_regex \? \. Aspx
  
  
# Disable cache for other aspx pages
  
  
# No_cache deny NONE_CACHABLE_PAGES
 
 

The following lines set the page cache duration. The first line is cache for one day and the second line is cache for two minutes.

 
 

  
  
refresh_pattern ^http://10.0.4.114:1100/mongodbsample/getfile.aspx 1440 0% 1440 ignore-reload  
  
  
refresh_pattern ^http://10.0.4.114:1100/mongodbsample/getfile.aspx 2 0% 2 ignore-reload 
 
 

If you want to configure SQUID correctly, you only need to access the site requested by SQUID as http: // 10.0.4.85: 8989/mongodbspame/uploadfile. aspx). here it will go to http: // 10.0.4.114: 1100/mongodbspame/uploadfile. aspx obtains the page information and getfile on the link in the page. all aspx files are cached, for example:

Now, the content of today is here first. Download the sample source code and SQUID configuration file:/Files/daizhj/mongodbsample.rar

Original article title: Use Mongodb storage to upload physical files and perform SQUID acceleration (based on the aspx page)

Link: http://www.cnblogs.com/daizhj/archive/2010/08/19/1803454.html