Android Display System-surface flinger

Good learning.

From: http://blog.csdn.net/yili_xie/archive/2009/11/12/4803527.aspx

Android display system --- surface flinger

Surfaceflinger is a part of Android multimedia. In Android implementation, surfaceflinger is a service that provides the surface composer function within the system, it combines 2D and 3D surfaces of various applications. Before we talk about surfaceflinger, let's take a look at some basic display knowledge.

1. Principle Analysis

Let's first take a look at the brief screen below GHOST:

Each application may correspond to one or more graphic interfaces, and each interface is called a surface or a window. In the preceding figure, we can see four surfaces and one is the home interface, there are three Surfaces Represented by Red, green, and blue, and the two buttons are actually the content in the home surface. Here we can see the problems we need to solve for graphic display:

A. First, each surface has its position and size on the screen. Then, each surface contains the content, content, and size to be displayed, location, these elements may change when we change the application. What should we do?

B. Then there may be overlapping surfaces. For example, in the simple figure above, green overwrites blue, while red overwrites green, blue, and home, it also has a certain degree of transparency. How should we describe the relationship between these layers?

Let's first look at the second problem. We can imagine there is a Z axis in the vertical direction of the screen plane. All the surfaces are determined based on the coordinates on the Z axis, in this way, we can describe the upper and lower overwrites between various surfaces, and the order on the Z axis has a professional term on the graph called Z-order.

For the first problem, we need a structure to record the location, size, and buffer of the application interface to record the content to be displayed. So this is our concept of surface, surface can be understood as a container, which records the control information on the application interface, such as the size, location, it also has a buffer to specifically store the content to be displayed.

There is still a problem here, that is, how to deal with the overlap of images, and some surface may also contain transparent information. here we need to solve the problem of surfaceflinger, it combines various surfaces (compose/merge) into a main surface, and finally sends the content of the main surface to FB/v4l2 output, so that we can see the desired effect on the screen.

In practice, merge can be applied to these surfaces in two ways. One is to use merge in the form of software, the other is to use hardware, and the other is to use surfaceflinger in software, the hardware method is overlay.

2. Overlay

Because the hardware merge content is relatively simple, let's first look at overlay. There are many methods to implement overlay, but they all require hardware support. Taking imx51 as an example, when IPU applies for FB from the kernel, it will apply for three FB, one for the main screen, the other for the secondary screen, and the other for the overlay. To put it simply, overlay is to send the format data and control information that the hardware can accept to this overlay framebuffer, and the hardware driver is responsible for merge overlay buffer and the content in the main screen buffer.

Generally, the current hardware only supports one overlay, which is mainly used for video playback and camera preview. Because the video content is constantly changing, the hardware merge is much more efficient than the software merge, the following is the process of using overlay and not using overlay:

Overlay Hal is added to surfaceflinger. You only need to implement this overlay Hal function. This HeaderFileIn:/hardware/libhardware/include/harware/overlay. H, you can use FB or v4l2 output. This may be the content of our future work. After overlay Hal is implemented, the sequence of the overlay interface will be:/frameworks/base/libs/surfaceflinger/tests/overlays. CPP. This sequnce is very important and will be discussed later.

However, in practice, we do not need to implement overlay Hal. If we know the hardware, we can directly send this information to the overlay buffer in the driver, instead of going to the upper-layer android. FSL uses this method in current camera preview, And I have roughly read the content of the R3 patch. It should also implement overlay in the video playback of opencore.

3. surfaceflinger

Now let's take a look at the most complex surfaceflinger. First of all, we need to make it clear that surfaceflinger is only responsible for controlling merge surface, such as calculating the overlapping areas of two surfaces. As for the content to be displayed on the surface, it is calculated using skia, OpenGL, and pixflinger. Therefore, before introducing surfaceflinger, we should first ignore what the content stored in it is, first figure out a series of control processes for merge, and then look at the processing process with the 2D and 3D engines.

3.1 surface creation process

As mentioned above, each application may have one or more surfaces. We need some data structures to store our window information and buffer to store our window content, the most important thing is that we should determine a solution to interact with surfaceflinger. Let's first look at the class diagram of the surface creation process below:

On the left side of ibinder is the client, that is, the application that needs to be displayed in the window, and on the right side is our surface flinger service. Creating a surface is divided into two processes: one is to create a management structure for each application (client) on the surfaceflinger side, and the other is to create a buffer for the storage content, and a series of operations such as drawing on the buffer.

Surfaceflinger manages multiple Window Interfaces of multiple applications. To manage multiple window interfaces, a client class is provided. Each application requesting services corresponds to a client. Because the surface is created in surfaceflinger, a structure must be returned to let the application know the surface information it applied for. Therefore, surfaceflinger encapsulates the control structure per_client_cblk_t created by the client and then returns it to surfacecomposerclient, the application is provided with a set of operations to create and destroy the surface:

After creating a client for the application, what needs to be done below is to allocate the surface to the client. flinger provides 8 Mb space for each client, including the buffer for controlling information and storing content. Before creating a surface, we should first understand the concept of layer and return to the simple screen diagram we have seen. In fact, each window is a layer on the Z axis, layer provides operations on window control information and content processing (calling OpenGL or skia). That is to say, surfaceflinger only controls when to process and process the information, all actual operations are performed in the layer. It can be understood that creating a surface means creating a layer. I have to say that the messy names of Android have left me around for a long time ......

When creating a layer, the client of the application generates a unique layer id based on the PID of the application, and then creates a layer based on the size, location, format, and other information. There is a nested surface class in the layer, which mainly contains an isurfaceflingerclient: surface_data_t, which contains the unified identifier of the surace and the buffer information, and is provided to applications. Finally, the application creates its own Surface Based on the returned isurface information.

Android provides four types of layers for selection. Each layer corresponds to one type of window and corresponds to the corresponding operations of the window: layer, layerblur, layerbuffer, and layerdim. I have to say that the name of Android is messy. layerbuffer can easily be understood as a layer buffer, which is actually a layer type. For the effects of each layer, refer to the description in surface. Java:/frameworks/base/CORE/Java/Android/View/surface. java. Here we will focus on two types of layers: layer (norm layer) and layerbuffer.

Norm layer is the most widely used layer for Android. Generally, applications use this layer when creating a surface, understanding normal layer allows us to understand some basic principles in the display process of Android. The normal layer allocates two buffers for each surface: front buffer and back buffer. This is a relative concept before and after, and they can perform flip. The front buffer is used for surfaceflinger display, while the back buffer is used for drawing applications. When the back buffer fills up data (dirty), it will flip, and the back buffer will become the front buffer for display, the front buffer becomes the back buffer for drawing. The size of the two buffers varies dynamically based on the surface size format. I did not take a closer look at the implementation of this dynamic change. For details, refer to setbuffers () in/frameworks/base/lib/surfaceflinger/layer. cpp ().

The two buffer flip methods are an important implementation method in Android display, not only for each surface, but also for the main surface written to FB.

Layerbuffer is also a layer that will be used in the future. I personally think it is also the most complicated layer. It does not have a render buffer and is mainly used on camera preview/video playback. It provides two implementation methods: POST buffer and overlay. The overlay interface is actually implemented on this layer. Whether it is overlay or POST buffer, the data of the layer is from other places, but the POST buffer is the final method of the software or the FB of the layer merge master, overlay is implemented by means of hardware merge. The isurface interface is closely associated with this layer and is used to register data sources. The following is an example to describe how to use these two methods:

The previous steps are common:

// To use surfaceflinger services, you must first create a client

Sp <surfacecomposerclient> client = new surfacecomposerclient ();

// Apply for a surface from surfaceflinger. The surface type is pushbuffers.

Sp <surface> surface = client-> createsurface (getpid (), 0,320,240,

Pixel_format_unknown, isurfacecomposer: epushbuffers );

// Then obtain the isurface interface. If the getisurface () function has permission restrictions during calling, it must be on the surface. h:/framewoks/base/include/UI/surface. h

Sp <isurface> isurface = test: getisurface (surface );

// Create overlay in the overlay mode, and then you can use the overlay interface.

Sp <overlayref> ref = isurface-> createoverlay (320,240, pixel_format_rgb_565 );

Sp <overlay> verlay = new overlay (REF );

// In POST buffer mode, you must first create a buffer and then register the buffer on isurface.

Isurface: bufferheap buffers (W, H, W, H,

Pixel_format_ycbcr_420_sp,

Transform,

0,

Mhardware-> getpreviewheap ());

Msurface-> registerbuffers (buffers );

3.2 application control and drawing of Windows

After the surface is created, the application can draw images in the buffer. Here we are faced with two problems. One is how to know which buffer to draw images on, the other is how to notify surfaceflinger to perform flip after drawing. Apart from drawing, how can we tell surfaceflinger to process the window when we move the window and change the window size? Before understanding these issues, we should first understand how surfaceflinger operates:

From the class diagram, we can see that surfaceflinger is a thread class that inherits the Thread class. When a surfaceflinger service is created, a surfaceflinger listening thread is started. This thread will wait for the event to occur, for example, sruface flip or the window location has changed, once these events are generated, surfacecomposerclient sends a signal through ibinder. This thread ends waiting to process these events and continues to wait after the processing is complete.

Surfacecomposerclient and surfaceflinger use the surfaceflingersynchro class to synchronize signals. It is actually a conditional variable. When the value of the waiting condition of the listener thread changes to open, the wait ends and the condition is set to close for event processing. After the processing is complete, the value of the waiting condition changes to open, once the client's surface changes, it notifies surfaceflinger through ibinder to change the value of the conditional variable to open and wake up the waiting thread. In this way, a dynamic processing mechanism is implemented through the Thread class and conditional variable.

After learning about surfaceflinger's event mechanism, let's look back at the problems mentioned above. First, you must lock the surface layer before drawing the surface. In fact, the swapstate variable in layer_cblk_t is locked. Surfacecomposerclient uses the value of swapsate to determine which buffer to use for drawing. If swapstate is the following value, it will block the client and copy it directly without translation:

// We block the client if:

// Enextflippending: We 've used both buffers already, so we need

// Wait for one to become availlable.

// Eresizerequested: the buffer we're re going to acquire is being

// Resized. block until it is done.

// Efliprequested & ebusy: the buffer we're re going to acquire is

// Currently in use by the server.

// Einvalidsurface: this is a special case, we don't block in this

// Case, we just return an error.

Therefore, the application first calls locksurface () to lock the layer's swapstate, and obtains the drawing buffer. Then, the painting can be performed on the layer. After the painting is completed, unlocksurfaceandpost () will be called () to notify surfaceflinger to perform flip. Or just call unlocksurface () without notifying surfaceflinger.

In general, all pixels on the surface need to be re-painted during the painting process, because normally the displayed pixels are not saved, but some pixels can be saved through settings, only part of the pixels are drawn. here we need to copy the content of the front buffer to the back buffer. In surfaceflinger service implementation, pixel copying is often required, and may also involve the conversion of the copy process, such as screen rotation and flip. Therefore, Android provides Hal for copying pixels, which may be implemented in the future, because hardware is used to copy pixels and possible matrix transformations during the copying process, it is more efficient and resource-saving than memcpy. The Hal header file is in:/hardware/libhardware/hardware/include/copybit. h.

The processing of window status changes is a very complicated process. First of all, it should be noted that surfaceflinger only executes the Windows Manager Command, and Windows Manager determines what is the size and position of the occasionally changed window, set transparency and how to adjust the order between layers. surfaceflinger only executes its commands. PS: Windows Manager is a service on the Java layer. It provides management functions for all windows. I have not read this part carefully and I think it is something I need to know in the future.

Window status changes include moving the position, window size, transparency, and Z-order. First, let's take a look at how surfacecomposerclient interacts with surfaceflinger. When the application needs to change the window state, it packs all the state changes and sends them together to surfaceflinger. After surfaceflinger changes the state information, it will wake up the waiting listening thread, set a flag to indicate that the status of the listening thread window has changed and must be processed. In Android implementation, the packaging process is a transaction, all changes to the window status (layer_state_t) must be in a transaction.

The processing process of the application client has been completed, which is basically divided into two parts: drawing in the window and processing of window state changes.

4. surfaceflinger processing process

After understanding the interaction between flinger and the client, let's take a closer look at the surfaceflinger processing process. We have already said that the surfaceflinger service will start a listening thread when it is created, this thread is responsible for processing every window update. Next we will take a closer look at the processing of this thread event, which is roughly shown in the figure below:

First, let's take a rough look at how Android combines various windows: Android actually calculates the visible area of each window, it is the window area visible on the screen (visibleregionscreen In the android vocabulary), and then draws the visible area of each window to the corresponding part of a main layer, finally, a complete screen is spliced, and the main layer is delivered to the FB display. A hardware implementation and a software implementation are involved in painting visible areas of each window to the main layer. If it is software implementation, it is re-drawn through OpenGL, it also includes Alpha computing with transparency. If copybit Hal is implemented, you can directly copy this part of the data in the window and complete possible rotation and flip, and alhpa computing.

Next let's take a look at the specific process of combining various layers in Android and sending them to the FB display:

4.1 handleconsoleevent

After receiving the signal or singalevent event, the thread stops waiting to start processing the client's request. The first step is handleconsoleevent, this step is related to the/dev/console device. It gets the screen or releases the screen. Only when you get the screen can you draw a picture on the screen.

4.2 handletransaction

As mentioned above, window status changes can only be performed in one transaction. Because the window status may change the visible area of the current window and other windows, you must calculate the visible area of the window again. In this processing sub-process, Android will traverse all layers based on the flag, once the status of the window changes, set the flag to recalculate the visible area of the window in the future. After traversing all sub-layers, Android will process the main layer based on the flag. For example, if the sensor senses that the mobile phone is crossed, the window will be displayed horizontally, in this case, you need to reset the direction of the primary layer.

4.3 handlepageflip

Here, we will process the flip between the surface buffer in each window. Based on the swapsate of layer_state_t, we can determine whether to flip. When the value of swapsate is enextflippending, it will flip. After processing the flip, it will re-calculate the visible area of each layer. I haven't seen this re-computing process clearly yet, but it is roughly like this process:

Calculate from the layer with the largest Z value, that is, from the top layer calculation, remove its own transparent area and the opaque area covered in it, and obtain the visible area of the layer. Then the opacity area of the layer will be accumulated to the opacity coverage area, the visible area of the layer will be placed in the visible area of the main layer, and then calculate the next layer, until the visible regions of all layers are calculated. The intermediate computation is implemented through a logical and non-graphical operation defined in skia, similar to the logical and non-logical diagrams in mathematics.

4.4 handlerepaint

After calculating the visible area of each layer, this step is to draw the content of all visible areas to the corresponding part of the main layer, that is to say, the corresponding content in each surface buffer is copied to the corresponding buffer of the primary layer, which may also involve Alpha operations, pixel flip, rotation, and other operations, as I mentioned earlier, hardware or software can be used for implementation. Pixflinger is also used for Pixel synthesis in the process of using software OpenGL for computation. I have no time to take a closer look at this part of the content.

4.5. postframebuffer

The final task is to flip the two buffers of the master layer and put the content just written into the FB.