Android Image Library released by FaceBook-Fresco

Android Image Library released by FaceBook-Fresco

Reprinted

 

• Original article: Introducing Fresco: A new image library for Android
• By tyrone Nicolas
• Translator: ZhaoKaiQiang
• Proofreader: Chaossss
• Proofreader: bboyfeiyu
• Proofreader: BillionWang
• Status: complete

 

It is extremely important to quickly and efficiently display images on Android devices. Over the past few years, we have encountered many problems in how to efficiently store images. The picture is too large, but the memory of the mobile phone is very small. The R, G, B, and alpha channels of each pixel occupy a total of 4 bytes of space. If the phone screen is 480*800, a screen size image will occupy MB of memory. Mobile phone memory is usually very small, especially Android devices need to allocate memory for each application. On some devices, the memory allocated to the Facebook App is only 16 MB. An image occupies a tenth of its memory.

What happens when your App memory overflows? Of course it will crash! We developed a library to solve this problem. We call it Fresco. It can manage the images and memory used, and the App will not crash.

Memory Zone

To understand what Facebook has done, we need to know the difference between the heap memory that Android can use. The Java heap memory size of each App in Android is strictly limited. Every object is instantiated in heap memory using Java new, which is a relatively safe area in memory. The memory has a garbage collection mechanism, so when the App does not use the memory, the system will automatically recycle the memory.

Unfortunately, the garbage collection process in memory is the problem. When the memory is garbage collected, the memory not only recycles garbage, but also completely terminates the Android Application. This is one of the most common causes of freezing or transient false positives when users use the App. This will make the users who are using the App very depressed, and then they may be anxious to slide the screen or click the button, but the only response of the App is: Before the App returns to normal, ask the user to wait patiently

In contrast, the Native heap is allocated by the new of the c ++ program. There is more available memory in the Native heap. The App is limited only by the physical available memory of the device, and there is no garbage collection mechanism or other things. However, c ++ programmers must recycle each allocated block of memory. Otherwise, memory leakage will occur and program crashes.

Android has another memory area called Ashmem. It operates more like the Native heap, but there are additional system calls. When operating the Ashmem heap, Android extracts the memory area of the heap that contains data from the Ashmem heap instead of releasing it. This is a weak memory release mode; the extracted memory is released only when the system actually needs more memory (the system memory is insufficient. When Android puts the extracted memory into the Ashmem heap, as long as the extracted memory space is not released, the previous data will be restored to the corresponding location.

Erasable Bitmap

Ashmem cannot be directly processed by Java applications, but there are some exceptions. images are one of them. When you create a non-compressed Bitmap, the Android API allows you to specify whether it can be cleared.

BitmapFactory.Options = new BitmapFactory.Options();options.inPurgeable = true;Bitmap bitmap = BitmapFactory.decodeByteArray(jpeg, 0, jpeg.length, options);

After the above Code processing, the erased Bitmap will reside in the Ashmem heap. No matter what happens, the garbage collector will not automatically recycle these bitmaps. When the Android rendering system renders these images, the Android system library extracts these bitmaps from the Ashmem heap. After the rendering, these bitmaps will be placed back to their original locations. If a extracted image needs to be drawn again, the system only needs to decode it again. This operation is very fast.

This sounds like a perfect solution, but the problem is that Bitmap decoding operations run on the UI thread. Bitmap decoding consumes a lot of CPU resources. When the consumption is too large, it will cause UI blocking. For this reason, Google does not recommend this feature. Now they recommend another feature-inBitmap. However, this feature is not supported until after Android3.0. Even so, this feature is not very useful unless all the images in the App are of the same size, which is obviously not applicable to Fackbook. This restriction was removed until version 4.4. But what we need is a general solution that runs in Android 2.3-the latest version.

Self-reliance

For the problem of "decoding operations causing UI to be suspended" mentioned above, we have found a solution that ensures both UI display and memory management. If we put the extracted memory area back to its original location before the UI thread performs rendering and make sure it is no longer extracted, we can put these images in Ashmem, at the same time, the UI will not be suspended. Fortunately, the Android NDK has a function that perfectly meets this requirement, called AndroidBitmap_lockPixels. The initial purpose of this function is to be executed after unlockPixels is called to extract the memory area again.

When we realized that we didn't have to do this, we made a breakthrough. If we only call lockPixels and do not call the corresponding unlockPixels, we can create a memory-safe image in the Java heap memory without causing slow UI thread loading. We only need a few lines of c ++ code to solve this problem perfectly.

Use C ++ to write Java code

As Spider-Man said: "the more powerful the ability, the more responsibility it takes ." Cleanup Bitmap is neither recycled by the garbage collector nor processed by the built-in cleanup mechanism of Ashmem, which may cause memory leakage. So we can only rely on ourselves.

In c ++, the common solution is to create a smart pointer class to implement reference counting. These need to take advantage of the c ++ language features-copy constructor, value assignment operator and a definite destructor. This syntax does not exist in Java, because the garbage collector can handle all this. Therefore, we must implement these guarantees in Java in some way.

We have created two classes to complete this process. One of them is "SharedReference", which has two methods: addReference and deleteReference. The caller must take the base class object or make it out of the scope when calling the call. Once the reference counter is set to zero, resource processing (Bitmap. recycle) will occur.

However, it is clear that it is easy for Java developers to call these methods. The Java language is designed to avoid such a problem! Therefore, the CloseableReference class is built on the SharedReference. It not only implements the Java Closeable interface, but also implements the Cloneable interface. Its constructor and clone () method call addReference (), while the close () method calls deleteReference (). Therefore, Java developers must follow the following two simple rules:

1. Call. clone () when allocating a new CloseableReference object ().
2. When the scope is exceeded, call. close (), which is usually in the finally code block.

   These rules can effectively prevent memory leakage and enable Native memory management and communication in large Java programs such as Fackbook Android clients.

   It is not just a program, it is a pipeline

   There are many steps to display images on mobile devices: several excellent open-source libraries are executed in this order, such as Picasso, Universal Image Loader, Glide, and Volley. These open-source libraries have contributed a lot to the development of Android. We believe that Fresco will do better in several important aspects.

   The difference is that we regard these steps as pipelines, not just loaders. Every step and other aspects should be as independent as possible. It is as simple as passing data and parameters and generating an output. It should be able to perform some operations, whether parallel or serial. Some operations can only be performed under feature conditions. Some threads have special requirements. In addition, when we consider improving images, all the images will become very complex. Many people use Facebook with low network speeds. We want these people to see images as soon as possible, even before the images are completely downloaded.

   Don't worry, embrace stream

   In Java, asynchronous code has always been executed through the Future mechanism. In another thread, the code is submitted for execution, and a Future-like object can check whether the execution result is complete. However, this works only if there is only one result. When processing progressive images, we hope to display results in a complete and continuous manner.

   Our solution is to define a broader Future version called DataSource. It provides a subscription method. The caller must input a DataSubscriber and Executor. DataSubscriber can obtain and process the results from DataSource, and provides a very simple method to differentiate. Because we need to process these objects very frequently, there must be a clear close call. Fortunately, DataSource itself is Closeable.

   In the background, a new framework called "producer/consumer" is implemented on each box. In this case, we are inspired by ReactiveX. Our system has interfaces similar to RxJava, but it is more suitable for mobile devices and has built-in support for Closeables.

   Keep simple interfaces. Producer has only one method called produceResults. This method requires a Consumer object. In turn, Consumer has an onNewResult method.

   We use a system like this to associate the Producer. Suppose we have a producer job that converts type I to type O, then it looks like this:

   public class OutputProducer
       
         implements Producer
        
          {  private final Producer
          mInputProducer; public OutputProducer(Producer inputProducer) { this.mInputProducer = inputProducer; } public void produceResults(Consumer  outputConsumer, ProducerContext context) { Consumer  inputConsumer = new InputConsumer(outputConsumer); mInputProducer.produceResults(inputConsumer, context); } private static class InputConsumer implements Consumer { private final Consumer  mOutputConsumer; public InputConsumer(Consumer  outputConsumer) { mOutputConsumer = outputConsumer; } public void onNewResult(I newResult, boolean isLast) { O output = doActualWork(newResult); mOutputConsumer.onNewResult(output, isLast); } }}   
        
       

   This allows us to concatenate very complex steps and maintain their logic independence.

   Full animation coverage

   Facebook users like Stickers very much because it can store GIF and Web formats in animated form. If these formats are supported, new challenges are required. Because each animation is composed of more than one image, you need to decode each image, store it in the memory, and then display it. For larger animations, placing each frame of image in the memory is not feasible.

   We have created an AnimatedDrawable, a powerful Drawable that can present animations, and supports both GIF and WebP formats. AnimatedDrawable implements the standard Android Animatable interface, so the caller can start or stop the animation at will. To optimize memory usage, if the image size is small enough, We cache the image in the memory, but if it is too large, we can quickly decode the image. These behavior callers are completely controllable.

   All the background is implemented using c ++ code. We maintain a piece of decoded data and metadata parsing, such as width and height. We reference technical data, which allows multiple Java-side Drawables to simultaneously access a WebP image.

   How to love you? Let me tell you...

   When an image is downloaded from the internet, we want to display a bitmap. If the download fails, an error mark is displayed. After the image is loaded, we have a gradient animation. By using hardware acceleration, we can scale in or out proportionally, or convert the matrix into the desired size and then render it. We do not always scale down according to the center of the image, so we can define the focal point for the scale-down. Sometimes we want to display rounded or even circular images. All these operations should be fast and smooth.

   Our previous implementation was to use the Android View object -- the time was up. You can use ImageView to replace the placeholder View. This operation is very slow. Changing the View will force Android to refresh the entire layout. When the user slides, this is definitely not what you want to see. It is wise to use Android Drawables, which can be quickly replaced.

   So we created Drawee. This is an image display framework like MVC Architecture. This model is called DraweeHierarchy. It is implemented as a layer of Drawables. For underlying images, each layer has a specific function-imaging, cascade, gradient, or shrinkage.

   DraweeControllers connects to images through pipelines-or other image loading libraries-and processes background image operations. They receive events from the MPs queue and decide how to handle them. They control the actual operations of DraweeHierarchy-whether it is a placeholder image, an incorrect condition, or a completed image.

   DraweeViews has few functions, but they are all crucial. They listen to system events that no longer display Android views on the screen. When an image leaves the screen, DraweeView can tell DraweeController to disable the image resources used. This avoids Memory leakage. In addition, if it is no longer within the screen range, the Controller will tell the image pipeline to cancel the network request. Therefore, when scrolling a long string of images like Fackbook, there will be no frequent network requests.

   With these efforts, the hard work of displaying images is gone forever. To call the code, you only need to instantiate a DraweeView and specify a URI and other optional parameters. All the rest will be completed automatically. Developers do not need to worry about managing image memory or updating image streams. Fresco has done everything for them.

   Fresco

   After this complex tool library is displayed and operated, we want to share it with the Android developer community. We are very happy to announce that from today on, this project has been used as the source code!

   Murals are painting techniques and have been welcomed by people around the world for centuries. Many of our great artists use this name, from the Italian renaissance master Rafael to the mural artist Sri Lanka. We do not pretend to have reached this great level. We really hope that Android Developers will enjoy using this open-source library just as much as they did before.

   More

   Fresco Chinese Document