Optimization and storage of static resources in large-scale website

Static resource optimization:

Merge

There are several advantages to reducing HTTP requests:

(1) Reducing the time spent on DNS requests

(2) Reduce server pressure (Cpu,io)

(3) Reduce the HTTP request header (when we make a request to the server, we carry the cookie and some other information from the domain name in the HTTP header, When the server responds to the request, it also brings back some header information such as cookies. This information can sometimes be very large, which can affect bandwidth performance when requested and responded to.

Merge Request:

Lvmama Home

<link &NBSP; rel Span style= "Font-family:monaco, Consolas, Courier, ' Lucida Console ', monospace;font-size:12px" >= " stylesheet< Span style= "Font-family:monaco, Consolas, Courier, ' Lucida Console ', monospace;font-size:12px" > "&NBSP; href ";

Gewara Home

<link rel= "stylesheet" type= "Text/css" href= "http://static5.gewara.com/css/gewara.css?n=ui_layout,index_new &v=2014070214 "/>

Compression

Gzip compression transfer can be more effective in saving bandwidth traffic. He first compresses the text into. GZ and then transmits it to the browser, and finally the browser is responsible for extracting it to the user.

Initial, dynamic, static resources on the same server, maintaining the original directory structure

Why would a static picture, js,css, be stored in a separate domain?

With a separate domain name, the browser does not pass the cookie from the primary domain to the domain, reducing network overhead and, to a certain extent, increasing the page loading speed.

Open Lvmama Home page (Dynamic request):

Home Static resource Request:

From the two graphs we come to the conclusion that:

There are cookies under http://www.lvmama.com

http://s1.lvjs.com.cn No Cookies

Using static and dynamic separation

Static and dynamic separation, which is to put the content in the Web application on different Web servers, and to deal with dynamic and static content, so as to achieve performance improvement.

After separation:

A picture request: Http://s1.lvjs.com.cn/pics/super/2014/06/S72XK.jpg

The corresponding Linux storage directory might be this:

Var/www/super/2014/06/s72xk.jpg

Var/www/super/2014/06/s73xk.jpg

Var/www/super/2014/06/s71xk.jpg

var/www/super/2014/06/.....

Var/www/super/2014/..../...

Var/www/super/........./..../...

Var/www/.........../........./..../...

What's the problem?

Data content in a geometric progression, especially those small files, a few k to hundreds of K, the number of large, traditional file system processing is very difficult, such a problem:

(1) High disk IO

Massive file retrieval efficiency is low, because the existing storage system can not effectively manage the large number of small files, and in a single directory of the amount of files stored there is a certain limit, once the number of files reached a certain size, the file retrieval speed decreased sharply.

The current user can only organize a large number of small files through a multilevel directory, as the depth of the directory increases, the retrieval cost of the file increases, and the retrieval efficiency decreases (a large number of disk addressing).

(2) difficult to back up

(3) Single point problem, capacity and read and write can not be extended horizontally, there is a possibility of failure

In the event of a continuous disk failure or server failure, the existing storage system will immediately interrupt the read and write process of the application, causing serious or even loss of data.

Small files are very slow to backup, and backing up a large number of small files can take a lot of time and system resources.

Users have to take a time-consuming and laborious approach when storing large amounts of small files, copying files to multiple sets of storage systems for increased security.

What to do?

Distributed filesystem (Distributed File System)

The physical storage resources managed by the file system are not necessarily connected directly to the local node, but are connected to the nodes through the computer network.

(1) expansion Capability : is the most important feature of a distributed file system.

(2) High availability: In Distributed file systems, high availability includes two tiers, one is the availability of the entire file system, and the other is the integrity and consistency of the data.

(3) Protocols and interfaces: Distributed file systems are provided to the application interface.

(4) Resilient storage: You can flexibly increase or decrease data storage and add or remove resources from storage pools based on business needs without disrupting system operation. The biggest challenge for resilient storage is the problem of data turbulence when resources are reduced or increased.

(5) compression , encryption, de-weight, caching , and storage quotas : These features often test whether a distributed file system is extensible, and a distributed file system can be easily added without compromising overall performance. Then this filesystem is a good design. A storage quota is a quota limit for the storage space your app can use, such as only the maximum quota range for each app.

Current programme

Gridfs

Gridfs is a distributed file system over MongoDB that leverages MongoDB's distributed storage mechanism and stores file data and file metadata through MongoDB, with the advantages of a document-based database and file system.

MongoDB is a database open source project based on distributed file storage. Written by the C + + language. A high-performance data storage solution that is designed to provide a protective display for Web applications.

It is characterized by high performance, easy to deploy, easy to use, and easy to store data. The main features are:
* For collection storage, easy to store object type data.
* Free mode. (Schema-free)
* Support Dynamic Query.
* Full index support, including internal objects.
* Support Query.
* Supports replication and recovery.
* Use efficient binary data storage, including large objects (such as video, etc.).
* Automatically process fragmentation to support scalability at the cloud level
* Support multiple languages such as ruby,python,java,c++,php.
* File storage format is Bson (an extension of JSON)
* Available via Network access

Final selection:

Load Balancing-->nginx

Cache-->squid

Glassfish-->java Web Image Server application (Spring mvc,spring data for MongoDB)

Storage-->gridfs (a mechanism for storing large binary files in MongoDB)

About Image Server Application

Demand:

Equal scale thumbnail Image:

Http://img5.gewara.com/cw120h160/images/movie/201405/s2800beec_14638bfd918__7ec5.jpg

cw120h160 here

Large thumbnail features are supported.

Reference:

Site performance Optimization group definition of 34 guidelines--yahoo!

Best practices to speeding up Your Web Site (https://developer.yahoo.com/performance/rules.html)

Java Web High Performance Development (HTTP://WWW.IBM.COM/DEVELOPERWORKS/CN/JAVA/J-LO-JAVAWEBHIPERF2/)

MongoDB Gridfs Best Practice Overview (http://blog.csdn.net/chszs/article/details/20123327)

Optimization and storage of static resources in large-scale website