(reproduced) Introduction to Android Display principle

Android app shows the process:android app calls the Surfaceflinger service to render the measured, layout, and painted surface onto the display screen.

noun explanation

Surfaceflinger : Android System Services, responsible for managing Android the frame buffer of the system, which is the display screen.

Surface : Android each window of the app corresponds to a canvas (canvas ), which is surface that can be understood as Android a window for the application.

Android app display process contains two parts ( application side drawing, System side rendering ), two mechanisms ( inter-process communication mechanism, display refresh mechanism ), then we will come to one by one channel.

Application side

An Android application window contains a number of UI elements that are organized in a tree-like structure, where there is a parent-child relationship in which child UI elements reside within the parent UI element, such as:

Therefore, before drawing the UI of an Android application window, we first determine the size and position of each child UI element within the parent UI element. The process of determining the size and position of individual child UI elements within a parent UI element is also known as the measurement process and layout process. As a result, the UI rendering process for the Android application window can be divided into three stages: measurement , layout , and drawing . As shown in the following:

Measurement: Recursive (depth first) determines the size of all views (high, wide)

Layout: Recursive (depth first) determines the position of all views (upper-left coordinate)

Drawing: Drawing all the views of the application window on canvas canvases

Measurement, layout There is not much to say, here to focus on the drawing. Android currently has two drawing models: a software-based drawing model and a hardware-accelerated drawing model (fully supported starting with Android 3.0).

In a software-based drawing model, the CPU dominates the drawing, and the view is drawn in two steps:

1. Make the View hierarchy invalid

2. Drawing the View hierarchy

When an application needs to update its part of the UI, it invokes the invalidate () method of the View object where the content has changed. An invalid (invalidation) message request is passed in the View object hierarchy to calculate the screen area (dirty area) that needs to be redrawn. The Android system then draws all the areas that intersect the dirty area in the view hierarchy. Unfortunately, there are two drawbacks to this approach:

1. Drawing a view that does not need to be redrawn (the area that intersects the dirty area)

2. Masking some of the bugs that are applied (due to redrawing areas that intersect dirty areas)

Note: When the View object's properties change, such as the background color or the text in the TextView object, the Android system automatically calls the view The invalidate () method of the object .

In the hardware-accelerated drawing mode, GPU-led drawing is drawn in three steps:

1. Make the View hierarchy invalid

2. Record and update the display list

3. Drawing the Display list

In this mode, the Android system will still use the invalidate () method and the Draw () method to request screen updates and display the View object. But the Android system does not immediately execute the draw command, but instead begins by recording the view's drawing function as a drawing instruction in a display list, and then reading the drawing instructions in the display list to invoke the OpenGL related function to complete the actual drawing. Another optimization is that the Android system only needs to record and update the display list for the dirty area of the View object that is tagged by the invalidate () method call. A view object that has no expiration can replay the drawing instructions from the previous display list record to perform a simple redraw job.

The purpose of using the display list is to save the various drawing functions of the view to the drawing instructions, and to re-read the previously saved operation instruction to replay once without changing the view, and improve the display speed of the view. For a view that needs repainting, the display list is updated for the next reuse, and then OpenGL is finished drawing.

Hardware acceleration improves the speed of Android display and refresh, but it's not everything, it has three flaws:

1. Compatibility (partial drawing function not supported or incomplete hardware acceleration, see article end)

2. Memory consumption (OpenGL API call takes 8MB and actually consumes more memory)

3. Power consumption (GPU consumption)

System side

After the Android application draws a good view hierarchy in the graphics buffer, the graphics buffer is handed over to the Surfaceflinger service, and the Surfaceflinger service uses the OpenGL graphics library API to render the graphics buffer to the hardware framebuffer.

Since Android applications rarely involve the underlying Android system, the execution of the Surfaceflinger service is not too much to be described.

Inter-process communication mechanisms

Android applications must communicate with the Surfaceflinger service in order to be able to draw their own UI on the frame buffer of the system:

Android applications and Surfaceflinger services are running in different processes, so they use some kind of interprocess communication mechanism to communicate. Since the Android application notifies the Surfaceflinger service to draw its own UI, the UI data needs to be passed to the Surfaceflinger service, such as the area, location, and so on, to draw the UI. An Android application may have many windows, and each window has its own UI data, so the anonymous shared memory mechanism of the Android system comes in handy.

Between each Android application and the Surfaceflinger service, the UI Data is passed through an anonymous shared memory , as follows:

However, there is a lack of effective management in the simple anonymous sharing that passes multiple window data, so the anonymous shared memory is abstracted into a higher data structure sharedclient, as shown in:

In each sharedclient, there are up to 31 Sharedbufferstack, and each sharedbufferstack corresponds to a surface, which is a window. In this way, we can see why each sharedclient contains a series of sharedbufferstack instead of a single sharedbufferstack: A sharedclient corresponds to an android application, An Android application may contain multiple windows, surface. As you can see from here, an Android app can contain up to 31 windows.

N buffers are included in each sharedbufferstack (<4.1 n=2; >=4.1 n=3), which shows the double buffering and triple buffering techniques that are about to be mentioned in the refresh mechanism.

Display refresh mechanism

In general, when we draw the UI, we use a technique called "double buffering". Double buffering means using two buffers (in Sharedbufferstack), one of which is called front buffer, and the other is called back buffer. The UI is always drawn in the back buffer and then swapped with front buffer to render to the display device. Ideally, such a refresh will be done within 16ms (60FPS), which is described as a refresh process (display processing before front buffer,cpu, GPU processing back Buffer.

But the reality is not so ideal.

1. Time starting from 0, enter the first 16ms:display display frame No. 0, after the CPU finishes processing the first frame, the GPU is processed to continue the first frame immediately thereafter. Three of them do not interfere, everything is normal.

2. Time into the second 16ms: since the previous 16ms time, the 1th frame has been processed by CPU,GPU. So display can show the 1th frame directly. show no problem. However, during the 16ms, the CPU and GPU did not draw the 2nd frame of data in time (note the previous white space), but at the end of the cycle, CPU/GPU only processed the 2nd frame of data.

3. Time into the 3rd 16ms, the display should show the 2nd frame of data, but because the CPU and GPU have not finished processing 2nd frame data, so display can only continue to show the first frame of data, the result is the 1th frame more than once (the corresponding time period labeled a jank).

According to the above analysis, the key question of the occurrence of jank here is, why the 1th 16ms paragraph, Cpu/gpu did not deal with the 2nd frame of data in a timely manner? The reason is simple, the CPU may be busy doing other things, do not know the time to handle the UI drawing. But once the CPU wants to deal with the 2nd frame of data, time is missed!

To solve this problem, Android 4.1 introduced the VSYNC, which is similar to clock interrupts. The results are as follows:

It is known that each received VSync interrupt, the CPU starts processing each frame data. The whole process is perfect.

However, after careful pondering, you will find a new problem: in the CPU and GPU processing data can seem to be done in 16ms, and there is still time to spare, that is, Cpu/gpu fps (frame rate, Frames Per Second) is higher than display FPS. That's true. Since CPU/GPU only begins processing when the VSync is received, their FPS is pulled down to the same FPS as display. However, this processing is not a problem, because the Android device display FPS is generally 60, the corresponding results are very smooth.

What happens if the fps of CPU/GPU is smaller than the display FPS? Please see:

By:

1. In the second 16ms time period, display should show B-frames, but because the GPU is still processing B-frames, a frame is repeatedly displayed.

2. Similarly, in the second 16ms time period, the CPU is idle, because a buffer is used by display. B buffer is used by the GPU. Note that once the vsync time point is over, the CPU cannot be triggered to handle the drawing work.

Why can't the CPU start drawing work at the second 16ms? The reason is that only two buffer (before Android 4.1). If there is a third buffer, the CPU can use it directly without being idle. This leads to a triple buffer (Android 4.1). The results are as follows:

By:

The second 16ms time period, the CPU uses C buffer drawing. Although a frame will still be displayed more than once, the subsequent display will be more smooth.

Is the more buffer the better? The answer is in the negative. It is known that in the second time period, the CPU to draw the C-frame data to the fourth 16ms to display, which is more than the dual buffer condition 16ms delay. So, the best buffer is two, three is enough.

attached: Different API level , the support of the drawing function to the hardware acceleration mode