Image server architecture evolution

Now almost any application, such as a website, a web app and a mobile app, needs a picture display function, which is very important for the picture function from the bottom up. Must have a forward-looking planning picture server, picture upload and download speed is of crucial importance, of course, this is not to say that it is to engage in a very NB architecture, at least with some scalability and stability. Although all kinds of architecture design, I am here to talk about some of my personal ideas.

For the picture server IO is undoubtedly the most serious resource consumption, for web applications need to do a certain separation of the picture server, otherwise it is likely because of the image server IO load causes the application to crash. Therefore, especially for large sites and applications, it is necessary to separate the image server and application server, build a separate image server cluster, build a separate image server The main advantages:

1) share the Web server's I / O load - the resource-consuming picture services separated to improve server performance and stability.

2) Can optimize the picture server specially - Set up the targeted caching scheme for the picture service, reduce the bandwidth network cost, improve the visit speed.

3) Improve the website scalability - through the increase of image server, improve image service throughput.

From the traditional Internet web1.0, after the web2.0 era and the development of the current web3.0, with the increase in the size of the picture storage, picture server architecture is gradually changing, the following mainly discusses the three stages of the picture server architecture Evolution.

1, the initial stage

Before introducing the early days of the small picture server architecture, let us first take a look at NFS technology. NFS is an abbreviation for Network File System, the network file system. NFS is a software developed and developed by Sun for sharing files between different machines and different operating systems over the Internet. An NFS server can also be thought of as a FILE SERVER for sharing files between UNIX-like systems, easily mounted to a directory, and manipulated just as easily as a local file.

If you do not want to synchronize all the pictures on each picture server, then NFS is the easiest file-sharing method. NFS is a distributed client / server file system. The essence of NFS lies in the sharing of computers among users. Users can connect to a shared computer and access files on the shared computer in the same way as accessing a local hard disk. The specific idea is:

1) All front-end web servers mount the directories exported by the three image servers via nfs to receive the images written by the web server. Then [Image 1] server mount two other image server export directory to local external ap to provide access.

2) user upload pictures

The user submits the upload request to the web server through the Internet access page, and the web server copies the image to the corresponding mount local directory after the web server finishes processing the picture.

3) user access pictures

When the user accesses the image, the image server reads the image in the corresponding mount directory via [Image 1].

Problems in the above architecture:

1) Performance: The existing structure over-reliance on nfs, when the picture server nfs server problems, may affect the front-end web server. The problem of NFS is mainly lock problem. It is easy to cause deadlock, only hardware restart can be solved. Especially when the picture reaches a certain level, nfs will have serious performance problems.

2) high availability: external download only one server image, prone to single point of failure.

3) Scalability: There is too much dependence between image servers, and there is not enough room for horizontal expansion.

4) storage: web server upload hot uncontrollable, resulting in the existing picture server space is not balanced.

5) Security: nfs way For those who have a web server password, you can freely modify the contents of the nfs, the security level is not high.

Of course, the image server picture synchronization can not use NFS, you can also use ftp or rsync, ftp so that each image server will save a copy of the picture, but also played a backup role. But the disadvantage is the picture ftp to the server more time-consuming, if you use the asynchronous way to synchronize the picture there will be a delay, but in general the small picture files are okay. Use rsync synchronization, when the data file reaches a certain level, each rsync scan will take a long time will bring some delay.

2, the stage of development

When the site reaches a certain size, the image server performance and stability have certain requirements, the NFS picture service architecture is facing a challenge, relying heavily on NFS, and the system exists single-point machine prone to failure, the need for the overall architecture Upgrade. So there picture server architecture, there is a distributed image storage.

The concrete idea of ​​its realization is as follows:

1) After the user uploads the image to the web server, the web server processes the image, and then the front-end web server posts the image to [Image1], [Image2] ... [ImageN], and the image server receives the post Came over the picture, and then write the picture to the local disk and return the corresponding success status code. The front-end web server determines the corresponding operation according to the returned status code. If successful, the front-end web server generates thumbnails of various sizes, prints watermarks, and writes the ID and the corresponding image path of the image server into the DB database.

2) upload control

We need to adjust the upload, you only need to modify the web server post to the destination image server ID, you can control which image storage server upload to the corresponding image storage server only need to install nginx at the same time provide a python or php service to receive and save Picture, if you do not want to open python or php services, you can write an nginx extension module.

3) user access process

When users visit the page, according to the request picture URL to the corresponding picture server to access the picture.

Such as: http://imgN.xxx.com/image1.jpg

This phase of the picture server architecture, an increase of load balancing and distributed image storage, to a certain extent, to solve the problem of high concurrency and storage capacity. Load balancing In the case of a certain amount of financial resources can be considered F5 hard load, of course, also consider the use of open source LVS soft load (and also open the cache). This will greatly enhance the amount of concurrent access, according to the situation at any time deployment of the server. Of course, this time there are some flaws, that is, there may be multiple Squid exist the same picture, because access pictures may be assigned to squid1 for the first time, LVS expired after a second visit to squid2 or other, of course, relative Concurrency problem solving, such a small amount of redundancy completely within our allowable range. In the system architecture can be used in the secondary cache Squid can also consider the use of Varnish or traffic server, open source software for the cache to be selected rate to test the following points

1) Performance: Varnish itself is technically superior to Squid. It uses "Visual Page Cache" technology. Varnish has advantages over Squid in terms of memory usage. It avoids Squid frequently exchanging files in memory and disk, Performance is higher than Squid. Varnish can not cache to the local hard disk. There is also a powerful Varnish management port that can use regular expressions to quickly and batch remove parts of the cache. nginx is a buffer that is made with third-party module ncache, whose performance is essentially varnish, but nginx is generally the reverse in the architecture (static files now use a lot of nginx, concurrency can support up to 20,000 +). In the static architecture, if the front-end is directly facing the cdn alive front-end load of 4 layers, the full use of nginx cache is enough.

2) to avoid the file system cache, the file data volume is very large, the file system performance is poor, such as squid, nginx proxy_store, proxy_cache way cache, when the cache up, the performance will be Can not meet the requirements. Open source traffic server directly with bare disk cache, is a good choice, the domestic large-scale application and announced mainly Taobao, not because of its poor, but open source late. Traffic Server has been used internally in Yahoo for more than four years, mainly for CDN services, and CDN for distributing specific HTTP content, usually static content like images, JavaScript, CSS. Of course, the use of leveldb cache, I guess it can achieve good results.

3) Stability: Squid cache as a veteran King, the stability is more reliable, some users from my feedback point of view varnish occasional crash situation. Traffic Server has no known data corruption during its current use and its stability is relatively reliable. For the future, I actually expect that Traffic Server will have more users in China.

The above picture service architecture design eliminates early NFS dependencies and single point problems, can balance the picture server space, and improves the security of the picture server. However, one problem poses is that the picture server scales horizontally and redundantly. Just want to store on a normal hard drive, first of all, still have to consider the actual handling capacity of the physical hard drive. Is 7200 turn or 15000 turn, the actual performance difference is very big. As for the file system selection xfs, ext3, ext4 or reiserFs, you need to do some performance testing, from some official test data, reiserFs is more suitable for storing some small picture files. Inode problems should also be considered when creating a file system, select the appropriate size of the inode size, because Linux for each file is assigned a number called inode number inode, inode can be simply understood as a pointer, it always points to this file Specific storage location. The number of inodes that a file system allows is limited. If the number of files is too large, even if each file is a 0-byte empty file, the system eventually ends up being unable to create files due to running out of nodes, And the speed of the trade-offs, a reasonable structure of the directory index.

3, cloud storage stage

In 2011, at the Baidu Union summit, Robin Li mentioned that the era of reading the map of the Internet has come. The picture service has become a large part of an Internet application and the processing power of pictures has accordingly become the responsibility of enterprises and developers. A basic skill, image download and upload speed is even more important, to deal with the picture, you need to face the three main problems are: high traffic, high concurrency, mass storage.

The OpenStorageService (OSS) is a massive, secure, low-cost and highly reliable cloud storage service provided by Aliyun. Users can upload and download data anytime, anywhere through a simple REST interface, or manage data using a web page. At the same time, OSS provides Java, Python, PHP SDK, to simplify the user's programming. Based on OSS, users can build a variety of multimedia sharing sites, network disk, personal data backup and other business-based large-scale data services. In the following picture, cloud storage is mainly introduced by Ali cloud cloud storage OSS. The picture shows a simple architecture diagram of OSS cloud storage.

The real "cloud storage", not storage but cloud services, the main advantages of using cloud storage services are the following:

1) users do not need to know the type of storage device, interface, storage media and so on.

2) No need to care about the data storage path.

3) No need to manage and maintain the storage device.

4) No need to consider data backup and disaster recovery

5) Simple access to cloud storage, enjoy the storage service.

4, the composition of the module

1) KV Engine

Object source information and data files in OSS are stored on the KV Engine. In version 6.15, V Engine will use version 0.8.6 and use OSSFileClient for OSS.

2) Quota

This module records Bucket and user correspondence and Bucket resource usage in minutes. Quota will also provide the HTTP interface for Boss system queries.

3) Security module

The security module mainly records the ID and Key corresponding to the User and provides the user authentication function accessed by the OSS.

OSS Terminology Glossary

1) Access Key ID & Access Key Secret

When a user registers for OSS, the system assigns a pair of Access Key ID & Access Key Secret to the user, which is called an ID pair, and is used to identify a user and perform signature verification for accessing the OSS.

2) Service

OSS provides users with virtual storage space, in which one or more Bucket can be owned by each user.

3) Bucket

Bucket is a namespace on OSS; Bucket names are globally unique across OSS and can not be modified; each Object stored on OSS must be included in a bucket. One application, such as a photo sharing site, can correspond to one or more Bucket. A user can create up to 10 buckets, but there is no limit on the number and size of objects stored in each Bucket. Users do not need to consider data scalability.

4) Object

In OSS, each user's file is an Object, each file needs less than 5TB. Object contains key, data and user meta. Among them, the key is the name of the Object; data is the Object data; user meta is the user's description of the object.

Its use is very simple, as follows java sdk:

OSSClient ossClient = new OSSClient (accessKeyId, accessKeySecret);

PutObjectResult result = ossClient.putObject (bucketname, bucketKey, inStream, new ObjectMetadata ());

Execute the above code to upload the image stream to the OSS server.

Picture access is also very simple The url is: http: //bucketname.oss.aliyuncs.com/bucketKey

Distributed file system

There are several benefits of using distributed storage, distributed automatically provide redundancy, do not need us to back up, worry about data security, in the case of particularly large number of files, backup is a very painful thing, rsync scan may be Just a few hours, there is a point that the dynamic expansion of distributed storage convenience. Of course, in some other domestic file system, TFS (http://code.taobao.org/p/tfs/src/) and FASTDFS also have some users, but the advantage of TFS is more for some small file storage, mainly Taobao Using. In addition, FASTDFS performance problems in the concurrent write higher than 300 cases, the stability is not friendly enough. OSS storage is based on Alias ​​cloud 5k platform independently developed highly available, highly reliable distributed file system Pangu. Distributed File System Pangu and Google GFS similar Pangu architecture Master-Slave master-slave architecture, Master responsible for metadata management, Sliave called Chunk Server, is responsible for reading and writing requests. Master is based on the Paxos multi-master architecture, a Master died, the other Master can quickly take over, the basic failure recovery can be achieved in less than a minute. Files are stored in slices, each divided into three copies, on different racks, and finally provide end-to-end data validation.

HAPROXY load balancing

This is a new caching architecture based on haproxy's automatic hash architecture, with nginx as the foremost proxy to the caching machine. Nginx followed by the cache group, by the nginx url hash request will be distributed to the cache machine.

This architecture is easy to upgrade pure squid cache, you can install nginx on the squid machine. nginx has a cache function, you can cache some of the large amount of traffic directly in nginx cache, you do not need to go through more than one proxy request, to ensure that the picture server high availability, high performance. For example favicon.ico and website logo. Load balancing is responsible for the load balancing of all the OSS requests, and the failure of the http server in the background will be switched automatically so as to ensure the continuous service of the OSS.

CDN

Alibaba Cloud CDN service is a distributed caching system all over the country that caches website files (such as images or JavaScript code files) to servers in multiple city rooms throughout the country. When a user visits your website, Get data on servers in cities near TA so that end users can access your services very quickly.

Alibaba CDN service in the country deployed more than 100 nodes, can provide users with excellent network acceleration. As business suddenly explodes and grows, there is no need to scramble to expand network bandwidth, and CDN services make it easy to respond. As with OSS services, using CDN requires CDN service provisioning on aliyun.com. After the opening, you need to create your distribution in the management center on the site (distribution channel), each distribution consists of two necessary parts: distribution ID and source address.

Use Alibaba Cloud OSS and CDN can be very convenient for each bucket content acceleration, because each bucket corresponds to an independent second-level domain name, CDN delete for each file, simple and economical solution to the problem of storage and network services, After all, most web sites or apps consume most of their storage and network bandwidth for images or videos.

From the perspective of the industry as a whole, the recent such cloud storage for individual users such as DropBox and Box.net abroad is very popular. At present, there are mainly seven types of cloud storage and cloud storage in China.

Upload and download the divide and rule

Image server image download ratio is much higher than the proportion of upload, business logic processing is also very different from the upload server to rename the image, record storage information, download server to add watermarks, modify the size of the dynamic processing. From a high availability point of view, we can tolerate part of the picture failed to download, but must not have pictures failed to upload, because the upload fails, means that the data is lost. Upload and download separately, to ensure that the pressure will not affect the download upload pictures, but also point out that the download entry and upload entry load balancing strategy is also different. Uploading requires the Quota Server to record the relationship between the user and the image. Logical Processing of Download If the front-end cache processing is bypassed and the backend business logic is processed, you need to obtain the image path information from the OSS. Alibaba recently launched CDN based on the nearest upload function, automatically select the nearest CDN node from the user, making the data upload and download speeds are optimized. Compared to traditional IDC, access speed increased several times.

Pictures security chain handling

If the service does not allow anti-theft links, then traffic will cause bandwidth, server stress and other issues. The more generic solution is to add refer ACLs to nginx or squid reverse proxy software, and OSS also provides referral-based anti-spoofing. Of course, OSS also provides a more advanced URL signature anti-theft chain, which implements the following ideas:

First, verify that your bucket privileges are private, that is, all requests for this bucket must be considered legitimate after the signature has been passed. Then dynamically generate a signed URL based on the type of operation, the bucket to be accessed, the object to access, and the timeout period. With this signed URL, your authorized user can perform the appropriate action before the signed URL expires.

Signed Python code is as follows:

h = hmac.new ("OtxrzxIsfpFjA7SwPzILwy8Bw21TLhquhboDYROV", "GET \ n \ n \ n1141889120 \ n / oss-example / oss-api.jpg", sha);

urllib.quote_plus (base64.encodestring (h.digest ()). strip ());

The method can be PUT, GET, HEAD, DELETE any; the last parameter "timeout" is the time out, in seconds. A through the above Python method, the calculated signature URL:

http://oss-example.oss-cn-hangzhou.aliyuncs.com/oss-api.jpg?OSSAccessKeyId=44CF9590006BF252F707&Expires=1141889120&Signature=vjbyPxybdZaNmGa%2ByT272YEAiv4%3D

With this method of dynamically calculating a signed URL, the data placed on the OSS can be effectively protected from being stolen by others.

Image editing API

GraphicsMagick (http://www.graphicsmagick.org/)) should be familiar to techies working on the Internet for the editing of online images. GraphicsMagick is branching from ImageMagick 5.5.2, but now that he has become more stable and good, GM is smaller and easier to install, GM is more efficient, GM's manual is very rich. GraphicsMagick's commands are basically the same as ImageMagick.

GraphicsMagick provides very rich interface APIs for cutting, scaling, compositing, watermarking, image conversion, and padding. The SDK for development package is also very rich, including JAVA, C, C ++, Perl, PHP, Tcl, Ruby, etc. support for over 88 image formats, including important DPX, GIF, JPEG, JPEG-2000, PNG, PDF, PNM and TIFF, GraphicsMagick can be used on most platforms, Linux, Mac, Windows no problem. However, the independent development of these image processing services, the server's IO requirements are relatively higher, but at present these open source image processing editor library, relatively speaking, is not very stable, I encountered the tomcat process crash when using GraphicsMagick, Need to manually restart the tomcat service.

Alibaba Cloud has now opened up the image processing API, including most common processing solutions: thumbnail, watermarking, text watermarking, style, pipelines and so on. Developers can very easily use the above image processing solutions, I hope more and more developers can open more excellent products based on OSS.