How to avoid the time overhead of image Decompression

This is a translation (the original article "avoiding image decompression sickness"). The original Article is a very good article I have read about IOS image display, I have solved some of my doubts and problems, so I have translated and shared some of them to ensure some smoothness. Some of the content is somewhat different from the original content, but I try to ensure consistency of meaning, thank you for your correction and criticism. The translation is displayed on the horizontal line:

When icatalog. when working with the framework, I found that using a large image may cause some annoying problems. "large" means that the image has a large enough resolution (1024 × 768) to overwrite the entire screen of the iPad, or to overwrite the screen of the future Retina Display iPad (if any) with double resolution (2048x1536.

Imagine a magazine-type app with a uiscrollview page. Each page displays a uiimageview, once a page enters the screen area, you need to create or reuse a uiimageview for the page and place it in the current display area of scrollview, even if the page only has one pixel to enter the screen area, you still need to do the work. This runs very well on the simulator, but when you test it on a real machine, you will find a significant delay every time you enter the next page. This delay comes from extracting images from files and rendering them to the screen. Unfortunately, uiimage only decompress the image when it is to be displayed.

Because adding a View to the current view hierarchy must be performed on the main thread, decompression and rendering of the image to the screen are also performed on the main thread, this is the cause of this delay. This problem can also be found in other apps with similar effects in store.

Generally, we use two main formats: JPEG and PNG. Apple usually recommends that you use PNG as the image format on the user interface. These images will be optimized by an open-source tool called pngcrush. platform/developer/usr/bin/pngcrush). This allows iOS devices to decompress and render images faster during display. The first batch of magazine applications on the iPad platform, such as wired, used PNG as the format of magazine content and images, this causes a version of this application to exceed 500 mb (Link) [http://www.cocoanetics.com/2010/05/saturday-morning-breakfast-wired-emag/]

Although images in PNG format will be optimized in advance, this does not mean that in all cases, PNG is the best image format, PNG is very good for those pictures in the app, but what will happen to images that need to be down from the Internet. Both PNG and JPEG formats have their respective advantages and disadvantages:

PNG images have alpha channels, while JPEG images do not. PNG lossless compression, JPEG allows you to choose a compression quality of 0-100%. If you need an alpha channel (transparent), you can only use PNG format. However, if you do not need a perfect image, you can use the JPEG format. The JPEG format ignores the information that you cannot see, for most images, we can use a compression quality of 60-70%, which does not significantly affect the image quality. For images with "sharp pixels" such as text, we may need a high compression quality, low compression quality can be used for photos.

Let's take a look at the space consumption of the next image:

1. Disk space or space consumed by transmission over the Internet
2. Decompression space, usually x 4 bytes long (rgba)
3. When displayed in a view, the view itself also needs space to store the Layer

There is a possible Optimization Method for the first problem: copying compressed files to the memory is not as good as ing them to the memory. nsdata has the ability to assume that a disk space is in the memory, in this way, the image is actually accessed from the disk instead of the memory. It is said that cgimage knows which access method is the most efficient. uiimage only encapsulates cgimage.

For "How soon will it take to display these pixels on the screen ?" The time consumed by an image is determined by the following three factors:

1. Time of alloc/init uiimage from disk
2. Decompression Time
3. The time when the decompressed bits are converted to cgcontext. Generally, you need to change the size, mix the bits, and perform anti-aliasing.

To answer each question one by one, we need a benchmark for measurement.

Test environment and content

 

I made a benchmark app running on iOS devices. We have tested the optimized and unoptimized PNG images and images of different sizes. The minimum time unit is 1 ms, which is not very accurate, but has enough reference value.

Tests include 128x96,256x192,512x384,102 4x768 and 2048x1536. These include representative resolutions, optimized PNG, and unoptimized PNG, the compression quality ranges from 10 to 100% JPEG. Benchmark runs on iPad 1 + 2, iPhone 3G, iPhone 3G, and iPhone 4.

The following are the hardware parameters of each device, from Wikipedia:

• IPhone 3G: Samsung 32-bit Proteus arm11 620 MHz processor (downsample to 412 MHz), powervr mbx lite 3D GPU, 128 MB edram
• IPhone 3gs: Samsung apl0298c05 chip, arm Cortex-A8 architecture, downsample to 600 MHz (from 833 MHz), Integrated powervr SGX 535 GPU, 256 MB edram.
• IPhone 4: Apple A4 chip based on ARM Cortex-A8, integrated powervr SGX 535 GPU, 1 GHz in iPad, iPhone 4 frequency not disclosed: don't they all say it's 512 MHz, and mb ram has forgotten to write it, but you know it if you don't know it)
• IPad 1: 1 GHz Apple A4 chip 256 MB DDR RAM !)
• IPad 2: 1 GHz dual-core Apple A5 chip, 512 MB DDR2 Ram

Among them, iPad 1 and iPhone 4 have the same processor. After Apple uses its own chip, we can see a significant improvement in performance, the locking A4 is twice faster than the previous Cortex-A8 + powervr SGC 535 GPU

Test Data

1024x768 resolution, 90% Compression Quality JPEG image from loading, decompression to rendering time:

• IPhone 3G: 527 MS
• IPhone 3gs: 134 MS
• IPad: 79 MS
• IPhone 4: 70 MS
• IPad 2: 51 MS

It is also a 1024x768 resolution. It compares the optimized and unoptimized PNG images.

• IPhone 3G: 866 MS-1032 MS = fast 16%
• IPhone 3gs: 249 MS-458 MS = fast 46%
• IPad: 130 MS-256 MS = fast 49%
• IPhone 4: 179 MS-309 MS = fast 42%
• IPad 2: 105 MS-208 MS = fast 49%

The 3gs data fully reflects the optimized PNG benefits: twice faster than the unoptimized ones.

The following charts record all test data.

First, we can find that different compression quality in the same chart (with the same resolution) has similar time overhead. For data integrity, I still provide all the data:

These two charts represent the images that may be used on the user interface. Ignore the old device. We can see that all the formats are displayed around 20 ms. This time is the lower limit of the image size for instant display. If you have an old device that doesn't want your app to be stuck, you need to extract the package from the main thread.

The following three charts test images with resolutions that can be filled with iPhone and iPad screens.

Until now, if we ignore the old devices, most of the images will be quickly displayed. We can see that using PNG images under such conditions will have an impact on performance, and we will be more inclined to use JPEG.

Under these resolutions, even the fastest device can only process 2 (IPAD retina full screen)/10 (IPAD full screen) large images per second.

Even if the image is decompressed on a Non-main thread, it may take a considerable amount of time to draw the image. You may want to divide the image into small pieces and use the ledlayer for display.

In summary, we can see that the red area (decompression) always consumes the most time, and the rendering (rendering) time only depends on the resolution rather than the compression quality, Because pixels occupy a group of factors.

Generally, the time overhead of 100%-quality JPEG images is roughly the same as that of optimized PNG images. I can think of two reasons for using JPEG: 1) the optimized PNG cannot be dynamically created on the device. 2) You just need a perfect image without considering the disk space overhead.

Compare the file size horizontally (the file size chart is shown below). You can see that the time overhead increases linearly from JPEG 10% to JPEG 90%.

There is an interesting question in the last figure. In the future, the retina display iPad device may, the image we use should be decoded 3-4 times faster than the iPad 2 (because we need to display the image on a double-resolution screen, that is, three times more pixels than before ), the performance is only doubled from iPad 1 to iPad 2, which is why iPad 2 does not support Retina display.

File Size

Let's take a look at the file size. The optimized PNG image is faster to render, but does it reduce the file size?

The optimized PNG format only reduces the file size of large-sized images in a small amount. The 100% Compression Quality JPEG image file size is smaller than the optimized PNG file size, but 100% of the quality seems to have lost the goal of compression. We can see that the size of 90% compressed files is more than half the size of 100% compressed files, and the size of JPEG files increases linearly from 10% to 90%, however, there was a huge increase from 90% to 100%.

Instant?

For full-screen images, we can find a problem that has occurred since Apple launched the tablet: a 70%-compressed full-screen image on iPad 1 needs to be displayed up to 75 ms, the iPad is still 49 Ms On 2 and cannot meet the requirements of instant display. This is far from our goal of 60fps, and there is a big gap between 13fps or 20fps and the 30 FPS that makes people feel smooth. This will cause us to drag the scrollview. When the new image enters the screen, it will be stuck for one second, and then the scrollview must beat to catch up with your gesture.

If the decompression time is excluded, the results will be much better than before, and the 17-18 ms time will bring about 55 FPS smoothness. Interestingly, there is no difference in the time when two generations of iPad mix pixels to the layer. The difference is only about decompression.

One simple solution for creating a catalog page in icatalog is to use catiledlayer and disable fading. In this way, the image display can be processed in the background thread without affecting the performance of scroll. Of course, if you quickly slide to the right (scroll), there will be a significant delay in the display of the corresponding page. The disadvantage of this solution is that it is difficult to switch between the horizontal and vertical screens.

A more advanced method is to forcibly extract images in advance.

Force Extract

When you use an image for the first time, IOS decompress it. Generally, the decompressed version will be stuck for a period of time (memory allowed ). This is meaningless, but you can decompress the image by rendering it into a new one. In this way, you will getTwoDecompressed version.

- (void)decompressImage:(UIImage *)image
{
UIGraphicsBeginImageContext(CGSizeMake(1, 1));
[image drawAtPoint:CGPointZero];
UIGraphicsEndImageContext();
}


This piece of code will decompress this image and instantly it only has one pixel.

It is strange that if the uiimage is created only through initwithcontentsoffile, I cannot keep the decompressed version. Therefore, I must use an option provided by the ImageIO Framework (available after IOS 4) to explicitly maintain the decompressed version:

NSDictionary *dict = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:YES]
forKey:(id)kCGImageSourceShouldCache];

Cgimagesourceref source = cgimagesourcecreatewithurl (cfurlref) URL, null );
Cgimageref cgimage = cgimagesourcecreateimageatindex (source, 0, (cfdictionaryref) dict );

UIImage *retImage = [UIImage imageWithCGImage:cgImage];
CGImageRelease(cgImage);
CFRelease(source);


In this way, the image can be extracted only once: it takes a long time to decompress the image for the first time, but it does not consume the image for the second time. The key here is kcgimagesourceshouldcache. You can use this option for cgimagesource and cgimagesourcecreateimageatindex. the header file describes this as follows:

Specifies whether the image should be cached in a decoded form. The value of this key must be a CFBooleanRef; the default value is kCFBooleanFalse.

If this option is set to no, the image painting time increases with the decompression time. If it is set to yes, only one decompression is performed.

Conclusion

If you need an alpha channel or PNG format, we recommend that you install pngcrush on your Web server and process all the PNG images. In other cases, high-quality JPEG images can provide smaller file sizes and faster decompression and rendering.

It turns out that the PNG format is very good for small images that are used in the UI element, but not for those applications that display images in full screen mode. The replacement of PNG is usually 60-80% of the compression quality of JPEG, as to how much the compression quality is appropriate, it depends on your image content.

You may want to maintain the decompressed version of all the images displayed, but this will also bring a lot of memory overhead and cause your app process to be killed. Using nscache is a good solution. It can automatically look at these images when memory is insufficient.

Unfortunately, we don't know whether an image needs to be decompressed, we will not receive any notification when the decompressed version of the same image disappears (this may be suitable for submitting it to Apple's bug feedback website ). Fortunately, the decompressed images accessed through the preceding method do not consume any time. So you can use this method at the right time and under the right conditions without causing additional overhead.