(Facebook open Source project) Fresco: A new Android image processing class Library

It's important to quickly and efficiently display images on Facebook's Android client. Over the years, however, we have encountered many problems in how to efficiently store images. The picture is too large and the device is too small. A single pixel occupies 4 bytes of data (representing R G B and Alpha, respectively). If you're on a 480*800-sized phone screen, a single full-screen image will occupy 1.5MB of memory. Usually the memory of the phone is very small, and the memory is occupied by a variety of apps. On some devices, the Facebook app is only 16MB, but just one image takes up 1/10 of the space.

What happens when your app runs out of your memory? Will crash. We set out to solve this problem by creating a class (Fresco) that manages the image and memory. Crash away!

Memory Area

To understand what Facebook is doing, we have to understand the various heap memory that is applied to Android.

　　Java Heap is strictly limited to the subject of each application, set by the device maker. All objects created by the new operator of the Java language will come here. This is a relatively safe area of memory. The memory is always recycled, so when the app has run out of memory, the system will automatically recycle it.

Unfortunately, there was a problem with the memory recovery phase. To reclaim memory, Android must stop running the app and then run the garbage collector. This is one of the reasons why the app you're using is stopping or slowing down. This is frustrating, perhaps the user is trying to scroll or press a button, the result of the app is not responding, only inexplicable wait.

Instead, the native heap is a heap that is used by the C + + new operator. There is a lot of available memory here. The app is limited to the available physical memory of the device, there is no garbage collection, and the app does not slow down. However, we need to free up memory space in C + + programs, otherwise they will overflow and the app will eventually crash.

The other area of memory in the Android Sea oil is called ashmem. This is much like a local heap, but requires additional system calls. Android is able to "unpin" memory instead of releasing them. This is a lazy release that is released only when the system really needs more memory space. When Android points to (PIN) memory again, the old data will still be there, as long as it is not released.

A bitmap that can be cleared

　　Ashmem is not a direct access to Java applications, because there are some exceptions, and the image is one of them. When you create a decoded (uncompressed) image, such as a Bitmap,android API allows you to specify that the image is "cleared":　

Newtrue= Bitmapfactory.decodebytearray (jpeg, 0, jpeg.length, options);

These erased images exist in Ashmem, however, the garbage collector does not automatically recycle them. When the system renders the image, the Android system library marks the memory, and when the rendering is complete, it is no longer marked. Memory that is not marked can be reclaimed by the system at any time. If an image that is not marked needs to be redrawn, the image is re-decoded.

This sounds like a perfect solution, but the problem is that the fast decoding image takes place in the UI thread. Decoding is a CPU-intensive operation that slows down the UI when it executes. For this reason, Google opposes the use of features, and they recommend using a different flag:inbitmap. However, this flag only appeared in Android3.0. Until now, this flag is not commonly used, unless all images in the app are of the same size, which is completely out of line with Facebook. This restriction is not removed until Android4.4. However, we need a solution that will make all Facebook users happy, including those who use Android2.3.

Have the cake and eat it

We found a solution that takes into account the fast UI and fast memory. If we mark memory in advance, in a non-UI thread, and ensure that the tag is not removed, then we can guarantee that the picture exists in Ashmem without causing the UI to slow down. With luck, there is a function in the NDK that can do exactly that, called "Androidbitmap_lockpixels." This function is called after the "unlockpixels" call, removing the memory tag again.

When we realized that we didn't have to do that, we made a breakthrough. If we call "lockpixels" without matching "unlockpixels", we create a security image that exists outside the Java heap and does not slow down the UI thread. A few lines of C + + code can do things.

Write code in Java, but think in C + +

As we learned from Spider-Man, "great power comes from great responsibility." The marked clear image is neither garbage collection nor a built-in cleanup tool in Ashmem to prevent memory leaks. We believe in ourselves.

In C + +, the usual approach is to create nice pointer classes to implement reference counts, which take advantage of some tools in C + +, such as copy constructors, assignment operators, and deterministic destructors. These dynamic features do not exist in Java, and in Java, the garbage collector handles everything. So we have to find a way to implement C + + style in Java.

We use two classes to achieve this, one is sharedreference, it has two methods, AddReference and Deletereference. Callers must call them whenever they get the underlying object or go out of scope. Resource cleanup (such as bitmap.recycle) occurs once the reference counter is zeroed.

It is obvious, however, that for Java developers to invoke these methods and their error-prone. Java was chosen as a language to avoid this situation. So at the top of the sharedreference, we built the closeablereference. It not only realizes the Java Closeable interface, but also realizes the cloneable. The constructor and the Clone () method call AddReference (), and the Close () method calls Deletereference. So Java developers only have to follow two simple rules.

1. When assigning a closeablereference to a new object, call the. Clone () method.

2. Before the scope is exceeded, call the. Close () method, usually in a finally block.

These rules are very effective in preventing memory leaks, while allowing us to enjoy local memory management, such as in larger Java applications such as Facebook for Android or Messenger for Android.

It's not just the loader-it's a pipe

There are a number of steps when displaying an image on a mobile device.

There are several great open source libraries that perform these steps, such as Picasso,universal Image Loader, Volley, and so on. These are important contributions to Android development. We believe that our open source library goes further in several important ways.

Given that these steps are different as a pipeline instead of a loader, each step should be as independent of the other parts as possible, and enter some parameters to output the results simultaneously. It should do something in parallel as much as possible. The other part is serial. Some executions are only under certain conditions. Some of the threads they perform have special requirements. In addition, if we value the image of innovation, the whole picture will become more complex. Many people use Facebook in very slow internet connections, and we want these users to be able to see their pictures quickly, even before the pictures actually load.

Don't worry, love the upper.

Traditionally, asynchronous code in Java has been executed through mechanisms such as the future. Code is uploaded and executed on another thread, like a future object that can be detected to be ready for the result. However, assuming there is only one result, when dealing with advanced images, we think that there will be a whole sequence of successive results.

Our solution is an approximate version of the future, called DataSource. He provides a subscription method that the caller must pass a data subscriber and an executor. Data subscribers receive notifications of DataSource from intermediate or final results, and provide a simple way to differentiate them. Because we often deal with objects that need to explicitly call Close, DataSource itself is a closeable.

Behind the scenes, each of the above boxes is implemented, and a new frame called Producer/consumer is used. Here we draw inspiration from the Reactivex framework. Our system has the same interface as Rxjava, and is more suitable for mobile devices and embedded closeable support.

Interface is very simple, producer has a separate method, is produceresults, it is responsible for getting a consumer object. Instead, consumer has a Onnewresult method.

We use such a system to link the producers together. Suppose we have a producer, whose job is to convert type I to type O, then the code is as follows.

public class Outputproducer<i, o> implements producer<o> {  private final producer<i> Minputproducer;  Public Outputproducer (producer<i> inputproducer) {    this.minputproducer = inputproducer;  }  public void Produceresults (consumer<o> outputconsumer, Producercontext context) {    consumer<i> Inputconsumer = new Inputconsumer (outputconsumer);    Minputproducer.produceresults (Inputconsumer, context);  }  private static class Inputconsumer implements Consumer<i> {    private final consumer<o> moutputconsumer;< C10/>public Inputconsumer (consumer<o> outputconsumer) {      moutputconsumer = Outputconsumer;    }    public void Onnewresult (I newresult, Boolean islast) {      O output = Doactualwork (Newresult);      Moutputconsumer.onnewresult (output, islast);}}}  

　　This allows us to link together a series of complex steps, and still keep them logically independent.

Animations-from one to many

Stickers, an animated WEBP format stored in GIF, is loved by Facebook users. But there are new challenges in supporting stickers. An animation is not an image but a series of combinations of images, each of which is decoded and stored in memory and displayed. It is unreasonable to store each individual frame in a large animation into memory.

We created animateddrawable, a class with rendering capability, with two backend-one GIF and the other webp. Animateddrawable implements the standard Android Animatable interface, so the caller can start and end the animation at any time. In order to optimize memory storage, we cache all the frames into memory when we have enough memory, but the frames are too long, we will decode in a hurry. This behavior can be fully adjusted by the caller.

Two backend are implemented in C + + code, we copy encoded data and parsed metadata, such as width and height. We reference the count data so that multiple drawable can access a separate WEBP image at the same time on the Java side.

How do I love you? Let me make a way

We want to display a placeholder when the picture is being downloaded from the network. If the download fails, we display an error indicator. When the picture arrives, we make a quick fade effect animation. We often scale the picture, or apply a display matrix and use a hardware accelerator to render it. And we don't always scale the picture--the useful focus may be somewhere else. Sometimes we want to show a rounded picture, or a circular picture. All such operations should be fast and smooth.

Our previous implementations were using the Android View Object--swapping placeholders for a imageview when time came, which would be very slow. Switching views forces Android to perform the entire layout, not something like a user scrolling. A more sensible approach is to use Android's Drawables, which can be exchanged quickly.

So we created the Drawee. This is an MVC-like framework for displaying images. This model is called Draweehierarchy. It is a drawables implementation hierarchy in which each applies a specific function-imaging, layering, fading in, or scaling-for the underlying image.

Draweecontrollers Connect the image pipe--or for any image loader--to take care of the background image control. They receive events from the pipeline and decide how to handle them. They control the actual display of the draweehierarchy-such as placeholders, error symbols, or finished pictures.

Draweeviews only has a few functions, but all of them play a decisive role. They monitor the system events that Android view no longer displays. When leaving the screen, Draweeview can tell Draweecontroller to close the resource used by the image. This avoids a memory leak. Also, if you haven't gone out, the controller will tell the image pipeline to cancel the network request. Therefore, scrolling a long list image (as it often does on Facebook) does not abort the network.

With these tools, it's not so difficult to display images. The calling program only needs to instantiate a Draweeview, specify a URI, and then arbitrarily set some other parameters. All the other work is automated. Developers don't have to worry about managing image memory or updating streams to images. Everything is done by the class library.

Fresco

A flexible toolset has been set up to display and manipulate images, and we want to share it to the Android developer community. We are pleased to announce that starting today, this project is open source code.

Fresco (wet mural) is a painting technique that has been popular for centuries. We are honored that many great artists have used this form, from Italian Renaissance masters such as Raphael, to Siguiriagu artists in Sri Lanka. We do not claim to have reached that level. We want Android app developers to enjoy the fun of using class libraries.

This article from: Https://code.facebook.com/posts/366199913563917/introducing-fresco-a-new-image-library-for-android/

(Facebook open Source project) Fresco: A new Android image processing class Library