Android Display System

 

Android Display System introduce (Qualcomm 8x60 Platform) (2. Sw Architecture -- 1)

I. Overview

The prototype is taken from Qualcomm's document. Since the source image cannot describe the existing architecture, I made some modifications to the source image, mainly adding the overlay part, in addition, some changes were made based on the existing software. Next, let's give a rough introduction to the key part, and then make a detailed analysis.

The top layer is the application, which can be divided into several types based on the data type and application.

The first is the most common application, such as the display of the UI interface. The data type is usually RGB, and no special processing is required. This application can be said to be spread across various applications, and almost exists in real time.

The second type is for the application of large YUV data, such as the preview of camera and video playback. This application is only applicable to specific applications.OverlayDirectly send large volumes of YUV data to the kernel for display.

The third type is similar to the first one, but the application needs to process data in 2D and 3D (using OpenGL, openvg, SVG, and skia) before display ), the process after processing is no different from the general display. It is generally used in game, MAP, Flash, and other applications.

Under an application is the framework, the core of which is surfaceflinger, which provides services for the display of all applications. Because the overlay interface is mounted in surfaceflinger (although the two are not relevant in functions), all overlay APs need to access overlay through surfaceflinger. In addition, surfaceflinger needs to use OpenGL to compose.
Surface, which is why surfacelfinger will call EGL wrapper. EGL wrapper is an encapsulation of graphics Hal, except that surfaceflinger will call it to compose.
In addition to the surface, 2D and 3D applications on the upper layer also call it for graphic processing.

The next layer is Hal.

The first is the overlay module, which provides the control channel and data channel on the upper side. The lower part is the MDP driver called to the kernel through the system.

The other is the gralloc module. Note that it is tied with overlay. It contains two parts. One is to provide the pmem interface for the upper layer, and the other is to refresh the framebuffer, the framebuffer here is actually the UI data. It can be seen that the upper layer has two channels to send the display data to the kernel. framebuffer is a traditional method, and overlay is added after Android (after é Clair.

The red part and the right part are the Hal of OpenGL. The red part represents HW solution, which is provided by Qualcomm and does not have source code. The software graphics library on the right is Sw.
Solution: Android. HW and SW solution can exist at the same time or only one, which will be explained later.

The next step is the driver in the kernel. The most important thing is the FB device driver and the mdp4 overlay driver. From the hardware perspective, the two are parallel, and framebuffer is finally sent to MDP through overlay. Pmem and kgsl correspond to the driver of pmem (/dev/pmem) and adreno220 in the kernel respectively.

Ii. surfaceflinger

1. Overview

Surfaceflinger can be said to be the core of the android display system. In Android, surfaceflinger is a service that providesSystemSurface in the range
The composer FunctionApplicationThe 2D and 3D surface of the program are combined, and the merged main surface data is sent to the video memory. To put it simply, surfaceflinger is like a canvas. surfaceflinger does not care about the content to be drawn, but simply performs the merging function. Of course, surfaceflinger must be based on parameters such as the position, size, and effect of the painting. This is similar to the layers in Photoshop. You can draw any content on different layers. You can set the position, size, and effect parameters for each layer, and finally synthesize a layer through merge.

From the application perspective, each application may correspond to one or more graphic interfaces, and each interface can be seen as a surface. First, each surface has its position, size, content, and other elements, which can be changed at will. In addition, the positions of different surface may overlap, which may involve effect processing problems such as transparency, these are all done through surfaceflinger. Of course, surfaceflinger is responsible for management. It uses OpenGL to process and Synthesize effects, but the premise is that surfaceflinger needs to calculate relevant parameters, such as overlapping positions.

2. surfaceflinger's position in the system

The image system in Android adopts the client/server architecture. The server is responsible for surface synthesis and other processing. The client provides interfaces to the upper layer to operate its own surface and sends messages to the server to complete actual processing. The server (surfaceflinger) is mainly written by C ++ code. Client-side code is divided into two parts: one is the supplied APIs provided by Java, and the other is the underlying implementation written by C ++. As shown in:

Except for applications at the top layer, the interface at the top of the surface layer is Java surface. The file path is as follows:

Frameworks/base/CORE/Java/Android/View/surface. java. The interface in this file is indirectly called by the application.

We can see from JNI that the JNI file path of the surface is as follows:

Frameworks/base/CORE/JNI/android_view_surface.cpp. The interfaces in frameworks/base/CORE/JNI/android_view_surface.cpp are classified into two types: one is responsible for managing ibinder communication, and the other is related to display control, the second type of interface directly calls the C implementation function.

The file path implemented by C is as follows:

Frameworks/base/libs/UI/surface. cpp

Let's take a look at some important interfaces in JNI:

Surfacesession_init: This interface is called only once and is responsible for creating surfacecomposerclient, mainly preparing for inter-process communication. The corresponding destroy functions include surfacesession_destroy and surfacesession_kill.

Surface_init:Create a surface and call createsurface in surfaceflinger. The corresponding destroy functions include surface_destroy and surface_release.

Surface_lockcanvas: Called before plotting a surface, lock the surface and obtain the back buffer of the surface. The application can plot the surface.

Surface_unlockcanvasandpost: After the upper layer is drawn, this function notifies the lower layer that the back buffer has been drawn and can be updated to the video memory.

Surface_setlayer/

Surface_setposition/

Surface_setsize/

Surface_hide/

Surface_show/

Surface_setorientation/

Surface_freeze/

Surface_unfreeze

Surface_setflags/

Surface_setalpha/

Surface_setmatrix:Set some properties of the surface, such as size, position, orientation, truncation range, and Z-order. The final change is the structure attribute of the surface, as shown below:

Uint32_t what; // which attribute changes

Int32_t X; // display position

Int32_t y; // display position

Uint32_t Z; // Layer Sequence

Uint32_t W; // width

Uint32_t h; // height

Float Alpha; // transparency

Uint32_t tint; // color, not used

Uint8_t flags; // flag

Uint8_t mask; // blocking command

Uint8_t reserved;

Matrix22_t matrix; // truncation range

Region transparentregion; // transparency settings

3. Connection and communication between JNI and surfaceflinger

Since JNI and C functions are not implemented in the same process as surfaceflinger (one on the application-client and the other on the server), in Android, inter-process communication is implemented through IPC (binder, it comes from the Internet, but I modified some of the errors. It demonstrates the process of establishing a connection between JNI and surfaceflinger and creating a surface.

The implementation of JNI and C functions is considered as a part.

Surfacecomposerclient, which is the client of surfacelinger, can be associated with surfaceflinger using the binder through its upper layer. Both isurfacecomposer and isurfaceflingerclient are used to implement binder communication. The specific process is described as follows:

The application uses the JNI interface surfacesession_init to create the surfacecomposerclient. Use the surfacecomposerclient function to call getcomposerservice to obtain the isurfacecomposer ibinder object, and then use createconnection of this object to obtain the producer ibinder. Through this ibinder, JNI can call interfaces in surfaceflinger, such as createsurface. Because the binder method is used, the Code is a little more complex. You need to read the code several times to clarify the process.

4. surfaceflinger's connection with libui, OpenGL, and display devices

I have to mention that android is a very important part of the media framework, that is, libui, which is an interface provided by the Framework library for underlying operations, for example, you can call Hal interfaces such as gralloc and overlay. Other library classes inherit to call libui. surfaceflinger is connected to the display device in this way (including writing video memory and pmem usage)

Surfaceflinger uses OpenGL to synthesize the surface. Therefore, surfaceflinger directly calls the OpenGL interface.

Their architecture is as follows:

 

This part of the process is complicated, mainly because the inheritance of various classes is relatively large. I also read the code many times and used some materials to figure it out, the figure below is explained in detail:

Surfaceflinger is designed to support multiple screens, but the current version only supports one. One of surfaceflinger display devices corresponds to displayhardware in the same figure, surfaceflinger will create displayhardware during initialization (refer to surfaceflinger. readytorun function in CPP). One of its main tasks is to establish framebuffernativewindow and determine the data output device interface (see framebuffernativewindow. CPP), and then initialize OpenGL and create main
Surface. In the future, all layers in surfaceflinger will be painted on this main surface (see the init function of displayhardware. cpp ). In this way, the framebuffernativewindow interfaces in main surface, OpenGL, and libui are bound together.

Because libegl is responsible for the final synthesis of all layers, the final data sent to Hal must be triggered by libegl. The corresponding function flow is:

Postframebuffer (surfaceflinger )->Flip (Displayhardware)-> Eglswapbuffers (OpenGL)->Queuebuffer (Libui)-> Fbpost (Gralloc)

In addition, graphicbuffer in the figure is the pmem operation interface provided by libui. It directly calls the gralloc module. There will be separate chapters on OpenGL and gralloc.