Twitter Architecture Optimization Path--twitter is how to handle 3000 images per second

Source: Internet
Author: User
Tags webp

Today, Twitter can create and save 3000 images (20GB) per second. In 2015, Twitter even saved $6 million from the optimization of its media storage strategy.

But that's not the case in the beginning, and Twitter was mostly text-based in 2012, like the Hogwarts School of Witchcraft and Wizardry in Harry Potter without the cool photos hanging on the walls. Now it is 2016 and Twitter has entered the era of rich media future. During the development of the new media platform, Twitter can support photo previews, multiple photos, gifs, vine, and online videos.

Twitter's software development engineer, Henna Kermani , said in a conversation with Mobile @Scale London that the media platform could handle 3000 images per second. Although the main topic of the conversation was to discuss the picture pipeline, she said that most of the details also apply to other media types.

Some of the most interesting excerpts from this conversation are summarized below :

In the simplest way possible, the result will really surprise you: sending a tweet with a picture is an all-or-nothing operation, and the simplest way to do that is to lock it. With the inability to scale well, especially with poor network conditions, it is difficult for Twitter to add new features.

separation Processing: by separating the tweets from the sending media and handling them in a decoupled way, Twitter optimizes each approach, while significantly increasing operational flexibility.

move the handle (handle)instead of moving the BLOB (Binary large object): Do not perform large chunks of data movement in the system, which consumes bandwidth and causes performance problems for each service that touches the data. Please store the data and use handle to refer to it.

Instead, a segmented, recoverable upload operation can significantly reduce the failure rate of media uploads.

Experiments and research: In the study, Twitter found that setting the TTL (Survival time) of various image variants (thumbnails, small plots, large images, etc.) to 20 days allows for the most efficient and optimal balance of storage and computing. After 20 days, the probability of access to the image is very low, and the storage space that can be saved every day after deletion is almost half the 4tb--to the compute server, which saves $ millions of per year.

on-demand: We can delete variants of old pictures because they can be rebuilt in an instant without pre-calculation. Performing services on demand can increase flexibility and be more intelligent in the way tasks are executed and more centralized in control.

progressive JPEG(Progressive jpeg) is the true winner of the standard picture format: Both the front and back end are excellent for their support, and in slower networks, this kind of picture works well.
In the future of Twitter as a rich media, there are a lot of good things happening, let's read one by one.

The past--2012 Twitter
How to Write
The user edits a tweet in the app, perhaps with a picture attached.

The client sends this WENFA to a single monolithic endpoint. When uploading, pictures and other meta-data in tweets are bundled together and sent to the individual services involved in the process.

In legacy designs, endpoints are the root cause of many problems.

问题一: 浪费大量带宽

Create a tweet and upload media, both tightly coupled in one operation.

The uploaded action is a whole, either all succeeds or all fails. Whatever the cause of the failure-temporary interruptions to the network, temporary errors, and so on-require the entire process to start over, including uploading the media again. If a failure occurs while uploading to 95%, it must be re-uploaded.

问题二: 对较大的新型媒体来说,缺乏良好的扩展

This approach lacks the scalability of large media, such as video. The larger the media, the greater the likelihood of failure, especially in the emerging markets of the IT industry, such as Brazil, India, Indonesia, and so on, due to the slow speed, poor network reliability, the problem is more serious, it is really necessary to increase the success rate of uploads.

问题三: 内部带宽的使用效率低下

The terminal is connected to the TFE (Twitter frontend), while TFE is responsible for user authentication and assigns the user to a different image server (image Service).

The picture server and the Picture Variant Generator (Variant Generator) session, and generate different sizes of picture instances (such as Tou, middle, large, thumbnail). Picture variants are stored in Blobstore, an Key-value storage system optimized for large payloads such as images and videos, and the images stored therein are permanent.

There are many other services involved in creating and saving tweets. Because the terminal is a single monolithic, the media and the tweet meta-data together, will also flow through all services. This large payload is sent to services that are directly responsible for the image, which are not part of the media pipeline, but are still being forced to perform optimizations for large payloads. This approach is very inefficient in the internal bandwidth.

问题四: 臃肿的存储空间

Images in tweets are no longer called after months or years, but still occupy space in Blostore. Sometimes even after tweets have been deleted, the picture still exists in Blobstore, which lacks a garbage collection mechanism.
Read mode
Users view tweets and associated pictures. Where is the source of the picture?

The client requests a variant of the picture from the CDN, which may need to request a picture to the origin and the Twitter front end, resulting in a direct search for a specific size, image on a particular URL in Blobstore.

问题五: 不可能引入新的变体

The design is not flexible enough to add a new variant is a different size of the picture, you need to fill in blobstore for each picture of the new image size, the lack of the need to increase the variation mechanism.

Because of the lack of flexibility, Twitter has a hard time adding functionality to clients.

Now--Twitter of 2016
How to Write
Decoupling the upload from the tweet.

Upload is set as the first, after the upload terminal is established, the only responsibility is to put the original media in the Blobstore.

Adds a lot of flexibility to uploading methods.

Client-to-TFE sessions, TFE and picture server sessions, picture servers place pictures in Blobstore, and add data to the metadata store, so there are no other related hidden services.

The media ID (mediaid) is a media-unique identifier that is returned by the picture server. If a user wants to create a tweet, DM, or upload a profile photo on the client, the system uses MediaID as a reference handle to the media, rather than using the media itself directly.

Let's say we want to create tweets with the media we just uploaded, the process is as follows:

The client finds the update endpoint and adds MediaID to the tweet, which is sent to the Twitter front-end, which is introduced into the appropriate service to perform the creation. For tweets, the service used is Tweetypie, and DM and personal data will use other services, all services will be dialogue with the picture server, image server has a tweet processing queue mechanism, can perform similar to face detection and child pornography detection and other functions; After the test is complete, The image server is responsible for the imagebird of the image or the Videobird session responsible for the video; Imagebird will generate a variant of the image; Videobird will perform some transcoding work; In any case, the generated media will be placed on the blobstore.

The media itself is not sent directly, thus saving a lot of bandwidth that was wasted before.

Multipart recoverable uploads:

The user walks into the subway, no signal, 10 minutes after the signal recovery, the upload process will automatically recover from the disconnected place. The whole process is seamless for the user.

The client initializes the upload process via the upload API and the backend sends it a mediaid that will be used as an identifier throughout the upload process.

A picture is divided into sections, such as three parts. With the API to append each part, each append call will have a segment index, and all append MediaID are the same. After uploading, it means that the upload process is finished and the media is ready for use.

This method is more adaptable to network failure situation, each individual part can be retried, if the network for some reason the interruption, then pause the upload, waiting for the network to resume after the upload of the part.

The simple approach brings huge gains. For files larger than 50KB, the failure rate of picture uploads in Brazil is as high as 33%, in India as high as 30%, and in Indonesia at 19%.

Read Mode
A CDN source server named Minabird was introduced (Origin server).

Minabird can talk to Imagebird, Videobird, so if not, you can immediately generate a corresponding size picture and video format.

Minabird is more dynamic and smoother when executing client requests. For example, you need to block a certain content because of copyright issues, and using Minabird can easily perform masking and recovery operations on a particular piece of media.

The ability to generate demand-sized images and video format transcoding in real-time, Twitter's intelligence on storage is also higher.

Generating the required media variants on demand means that you do not need to store all variants in Blobstore. This is a huge victory.

The original media is stored in blobstore until it is deleted, and the variant is saved for only 20 days. The media platform team did a lot of research on the best time-to-save, and in all the requested images, about 50% were saved for 15 days, decreasing the results by the rate of return and deleting older images. Very old media are likely to have no one to initiate the corresponding request, after 15 days there will be a long tail period.

If you do not set the TTL (survival time) and expiration time, the media storage is increased by 6TB per day. The lazy approach is to generate all media variants on demand, causing media storage to grow to 1.5TB. The 20-day TTL uses less storage space than lazy methods, so it doesn't take up too much storage space, but it's a huge success in computing. Using lazy method to calculate, read all variants need to set 150 Imagebird in each data center, and use 20 day TTL, only need 75 imagebird. So the 20-day TTL is the most efficient and balanced point of time for computing and storage.

Since saving storage space and computing resources is saving money and introducing a 20-day TTL, Twitter saved $6 million in 2015.

Client-side optimization (Android)

After 6 months of experimentation using WEBP (a picture format created by Google),

This image is 25% smaller than the corresponding PNG or JPEG image.

In this way, especially in emerging markets, user engagement is also higher because smaller images have less pressure on the network.

Due to the unsupported iOS system and support for Android 4.0 or more, the lack of platform support makes the WEBP format a huge expense.

So Twitter tried another option, the progressive JPEG format. The first scan may be blocky due to continuous scanning, but it will gradually become self-perfecting in the process of continuous scanning.

This format has a better performance.

The backend is easy to support.

This format is 60% slower than the traditional JPEG format, and is not a big problem because of the number of times the service is encoded.

Progressive JPEG images do not support transparent images, so transparent PNG images are preserved, and in addition, progressive JPEG is used.

The client is supported by the Facebook fresco Library, and fresco has many advantages. The effect is impressive under the 2G network. The first scan of the pjpeg image only uses 10kb of traffic, so the load time is not long, the local pipeline is still waiting to load, nothing shows, Pjpeg has shown a recognizable image.

There are ongoing implementation results showing that the load details are as follows: Reduced P50 load time by 9%, reduced P95 load times by 27%, reduced 74% failure rates, and a significant improvement for users with slow connections.

Twitter Architecture Optimization Path--twitter is how to handle 3000 images per second

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.