Basic knowledge of android graphics

 

AndroidGDI basic framework

The concepts and Code involved in Android are the most complex, that is, the Code related to GDI. But in essence, in terms of abstraction, so many codes and frameworks do one thing: the operation and management of the display buffer.

GDI mainly manages the output of graphic images. In the overall direction, GDI can be considered as a manager used by a physical screen. Because in actual products, we need to output different windows on the physical screen, and each window considers itself exclusive to the use of the screen, for all windows output, applications do not care about whether the physical screen is occupied by other windows. Instead, they only care about the output in the current window. If the output can be seen on the screen, it needs to be managed by GDI.

From the analysis of the data stream at the top to the bottom, we can see that in fact, GDI provides an abstract concept for the GUI at the top, just like the files provided by the file system in the operating system, similar to abstract concepts such as directories, GDI outputs are abstracted into device-independent operations such as text, paint brushes, and bitmap operations, so that application programmers only need to perform output operations in the logical device context, do not involve the management of specific output devices and output boundaries. GDI maps conceptual objects such as text, lines, and bitmaps to specific physical devices. Therefore, GDI can be divided into the following elements in the general direction:

Canvas

Font

Text output

Painting object

Bitmap output

Android GDI System

Android's GDI system involves too many concepts. In addition, OpenGL makes the Android hierarchy and code complicated. However, what we need to know about the Android GDI system is not its static code relationship, but its dynamic object relationship. We need to understand the GDI in the logic running architecture. First, we need to understand the code structure.

Frameworks/Libs/Surfaceflinger

Frameworks/base/core/jni/android_view_Surface.cpp

Frameworks/base/core/java/android/view/surface. java

Frameworks/base/Graphics: drawing interface

Frameworks/Libs/Ui

External/Skia

External/Skia is a C ++ 2D graphics engine library. The Android 2D rendering system is built on this base. skia has completed the following functions: text output, bitmap, point, line, and image decoding.

Here I will provide the basic framework of AndroidGDI.

We only have a rough understanding of the above GDI architecture diagram. We have too many problems to solve and have too many questions to answer, so I keep thinking, why are there so many concepts proposed by designers? What is the background of this concept? What does he want to manage and abstract? As we know before, the entire design concept of Android is borderless. How does it penetrate the gap between Linux processes to achieve borderless development? What are Surface, Canvas, Layer, LayerBase, NativeBuffer, SurfaceFlinger, and SurfaceFlingerClient? How to manage and pass what? What is created? These are abstract concepts. How is the final buffer of painting managed? Where is the buffer?

Let's take a look at the ultimate and most essential design concepts. Starting from these concepts, we will discuss the formation process of these concepts and whether it is necessary to generate write concepts. What is SurfaceFlinger essentially? SurfaceFlinger: The application throws its "Surface" to the screen buffer through SurfaceFlinger. We will describe how to throw it in detail later.

 

Android core analysis (24) ----- display buffer management for Android GDI
24466 people read comments (22) collect reports

Android GDI screen device management-Dynamic Link Library
The tall building starts from the ground and starts from the most rooted hardware frame buffer zone. We know that the display of FrameBuffer is a piece of memory in the system, and the work of GDI is to put the content to be output to a certain position in the memory of this segment. We start with basic points (pixels) and basic buffer operations.

1. Basic Knowledge
1.1 point format
For different LCD, the binary format of FrameBuffer is different and can be divided into two parts:

 

1) Point format: Usually Depth, that is, the number of bits to represent a point.
1 bit indicates a vertex

Two digits indicate a vertex.

A 16-bit Vertex

32-bit indicates a point (Alpha channel)

2) Intra-point format: RGB component distribution representation.
For example, for a common 16-bit representation of a vertex

1. 2. Conversion between formats
Therefore, the screen output is actually a value ing relationship. We can convert the following vertex format,

The source format may come from the monochrome bitmap and color bitmap. For a specific target machine, our target format may be a type, such as the 16-bit (5/6/5) format. In fact, there is only one format conversion, that is, the slave target format is a 16-bit format.

However, when designing GDI, there is a basic requirement for good portability. Therefore, we must consider the conversion between different formats of LCD. Therefore, the following main operations are involved in the driver of GDI:

Regional Operations: The most common operation we do on the display buffer zone is block handling. As a result, many application processors use the hardware Graphics Accelerator to perform regional shipping: B. From the perspective of our main operations, there are two directions:

1) memory area to screen area

2) screen area to screen area

3) screen area to memory area

4) memory area to memory area

In this case, we need to note that because the memory between different processes in Linux cannot be accessed freely, the usage of each Android Application in the memory area and the screen buffer becomes very complex. In Android design, there are many layers of management in the screen buffer and Display memory buffer. The top layer objects can be freely transmitted between processes, however, the buffer content uses the shared memory mechanism.

Based on the above basic knowledge, we can know:

(1) The meaning of Config and its Format in the code. It also understands the significance of compatibility: the use of the same hardware point description object

(2) moving all screen shapes shows the result of buffer movement.

1.2 Graphics Accelerator
All application Processors may have Graphics accelerators. Different Application Processors may have different processing methods for their Graphics accelerators. For 2D acceleration, they can all be attributed to B. Most of them are data handling, amplification, and reduction, and rotation.

 

2. Buffer abstraction definition for Android
 
Different hardware has different hardware graphics acceleration devices and buffer memory implementation methods. The abstract task of the Android Gralloc dynamic library is to eliminate the differences between different devices. In the upper layer, the same method and object appear. Hide the buffer operation details on the Moudle layer. Android uses the dynamic link library gralloc. xxx. so to encapsulate the underlying details.

2.1 local definition @ hardware/libhandware/modules/gralloc

Each dynamic link library calls an interface with the same name:

1) Hardware Graphics Accelerator Abstraction: acceleration of BlitEngine and CopyBit.

2) Hardware FrameBuffer Memory Management

3) shared Cache Management

The abstract behavior of the dynamic link library is investigated from the data relationship: the content of the dynamic link library is completely packaged in the hierarchy: Hardware. c @ hardware/libhardware. /System/lib/hw/gralloc. xxx. so dynamic library file. The abstract result is obtained from the Gralloc. h (handware/libhardware/include/hardware) file: hw_get_module extracts HAL_MODULE_INFO_SYM (SYM variable) from gralloc. xxx. so)

From the external data structure, we can see this layout in @ Gralloc. cpp:

Static structhw_module_methods_t gralloc_module_methods = {

Open: gralloc_device_open

};

Structprivate_module_t HAL_MODULE_INFO_SYM = {

Base :{

Common :{

Tag: HARDWARE_MODULE_TAG,

...

Id: GRALLOC_HARDWARE_MODULE_ID,

Name: "Graphics Memory Allocator Module ",

Author: "The Android Open Source Project ",

Methods: & gralloc_module_methods

},

RegisterBuffer: gralloc_register_buffer,

UnregisterBuffer: gralloc_unregister_buffer,

Lock: gralloc_lock,

Unlock: gralloc_unlock,

},

Framebuffer: 0,

Flags: 0,

NumBuffers: 0,

BufferMask: 0,

...

};

What object have we set up to support buffer operations?

Buffer_handle_t: external interface.

Methods. open, registerBuffer, unregisterBuffer, lock, unlock

The following is the structure relationship between the external interface and the internal object. This type of structure makes full use of the data arrangement feature of CStruct: the basic structure is placed at the beginning, and the local private is placed at the end, this satisfies the abstract needs.

Typedef constnative_handle * buffer_handle_t;

Private_module_t HAL_MODULE_INFO_SYM refers to the dynamic link library interface exposed. Through this interface, we can directly use this object.

You cannot see the figure above clearly. You can look at it with a different look:

Several interface functions:

(1) fb_post

The actual address of the frame buffer does not need to be reported to the upper layer. All operations are completed through fb_post.

 

The fp_post task is to pass the content of a Buffer to the hardware Buffer. There are two implementation methods:

 

 

 

 

(Method 1) You do not need to copy the buffer. You need to switch the buffer after Framebuffer to the front buffer, and then tell the FB driver to switch the DMA source address through IOCTRL. The premise of this implementation is that the Linux kernel must allocate at least two physical memory buffers and ioctrol for switching, which enables fast switching.

 

(Method 2) use Copy. If the kernel is not modified, the copy method is used in the adaptation layer, but this is time-consuming.

 

 

 

(2) The main function of gralloc is to complete:

1) enable the screen device "/dev/fb0" and map the hardware display buffer.

2) provides an interface for allocating shared display caches.

3) provide the BiltEngine interface (to package the hardware accelerator)

(3) gralloc_alloc outputs the buffer_handle_t handle.

This handle is the basic basis for sharing. Its basic principles are described in detail in the following sections.

3. Conclusion
To sum up,/system/lib/hw/gralloc. xxx. so is a dynamic link library related to the hardware system. It can also be called the hardware abstraction layer of Android. It implements the Hardware Abstraction interface standard for Android and provides Display memory allocation mechanisms and CopyBit acceleration. How to implement these functions is related to the hardware platform, so we can see that there are different configuration relationships with different hardware architectures.

Android core analysis (25) ------ Android GDI's shared buffer mechanism
17969 people read comments (13) collect reports

Androird GDI's shared buffer mechanism
1 native_handle_t package for private_handle_t
Private_handle_t is the private data structure of the local buffer used by gralloc. so, while Native_handle_t is the abstract data structure that can be passed between processes. On the client, how does one restore the transmitted data structure? First, let's take a look at the abstract packaging of native_handle_t on private_handle_t.

NumFds = sNumFds = 1;

NumInts = sNumInts = 8;

This is the abstract mode used to describe the handle in Parcel. Actually, Native_handle points to the specific content of the handle object:

NumFds = 1 indicates that there is a file handle: fd/

NumInts = 8 indicates that the data following the eight INT types are magic, flags, size, offset, base, lockState, writeOwner, and pid;

In the upper-layer system, do not care about the data content in buffer_handle_t. Passing the buffer_handle_t (native_handle_t) handle between processes is actually to pass the handle content to the Client. Read readNativeHandle @ Parcel. cpp through the Binder on the client to generate an native_handle.

Native_handle * Parcel: readNativeHandle () const

{

...

Native_handle * h = native_handle_create (numFds, numInts );

For (int I = 0; err = NO_ERROR & I <= "" font = "">

H-> data [I] = dup (readFileDescriptor ());

If (h-> data [I] <0) err = BAD_VALUE;

}

Err = read (h-> data + numFds, sizeof (int) * numInts );

....

Return h;

}

Here we need to mention the processing of the file handle passed by the other party when constructing the native_handle of the client. Because it is not in the same process, dup (…) is required (...) For the client. In this way, the client interested in the Native_handle handle is copied from the server. In this way, the Private_native_t data is copied to the client, including magic, flags, size, offset, base, lockState, writeOwner, and pid.

The client uses this new Native_buffer to be transferred back to gralloc by Mapper. xxx. so, get the sharing buffer ing address associated with native_handle to get control of the buffer, and achieve the Memory sharing between the client and the Server. SurfaceFlinger shows that the plot area is shared.

2 What is Graphic Mapper?
The server (SurfaceFlinger) allocates a piece of memory as the Surface drawing buffer. How does the client work in this drawing buffer? This is what Grapher (GraphicBufferMapper) y is going to do. How do two processes share the memory and obtain the shared memory? This is what Mapper does. Two information needs to be used: the allocated offset of the device handle of the shared buffer. Mapper uses the following principle:

The client only has lock and unlock, which are essentially mmap and ummap operations. For the same shared buffer, the offset is always, and the starting address is not important. They actually operate on memory blocks of the same physical address. We have discussed the native_handle_t package process for private_handle_t and learned what the service end has passed to the client.

Process 1 pre-allocates 8 MB of memory on the shared memory device. All future allocations will be made in this 8 m space. For this file device, after 8 Mb physical memory is submitted, it actually occupies 8 MB of memory. Each process can share the 8 M memory with the same memory device, and the tool they use will be mmap. Since mmap uses 0 to get the ing address, all client processes have the same physical actual address, so the offset and size can identify a piece of memory. The offset and size are numeric values, which can be directly used from the service process to the client.

 
3 GraphicBuffer (buffer proxy object)
Typedef structandroid_native_buffer_t

{

Struct android_native_base_t common;

Intwidth;

Intheight;

Intstride;

Intformat;

Intusage;

...

Buffer_handle_thandle;

...

} Android_native_buffer_t;

Link Chart:

GraphicBuffer: EGLNativeBase: android_native_buffer_t

GraphicBuffer (parcel &) creates the native_buffer_t data of the local GraphicBuffer. Construct GraphicBuffer by receiving the native_buffer_t passed by the other party. Let's take a look at the function call of the client Surface: lock to get the operation Buffer:

Surface: lock (SurfaceInfo * other, Region * dirtyIn, bool blocking)

{Int Surface: dequeueBuffer (android_native_buffer_t ** buffer) (Surface)

{Status_t Surface: getBufferLocked (int index, int usage)

{

Sp buffer = s-> requestBuffer (index, usage );

{

Virtual sp requestBuffer (intbufferIdx, int usage)

{Remote ()-> transact (REQUEST_BUFFER, data, & reply );

Sp buffer = newGraphicBuffer (reply );

Surface: Lock creates a new GraphicBuffer object on the Client. This object uses the principle described in (1) to construct the buffer_handle_t data of SurfaceFlinger into the new Client buffer_handle_t data. On the client's Surface object, you can use GraphicMapper to perform mmap on the client's buffer_handle_t to get the starting address of the shared buffer.

4. Summary
Android uses shared memory in this section to manage and display the relevant buffer. It is designed to be two layers. The upper layer is the buffer management proxy GraphicBuffer and Its Related native_buffer_t, the lower layer is the allocation management of specific buffers and the buffer itself. Objects on the upper layer can be transmitted frequently through the Binder. In the process, instead of passing the buffer itself, mmap is used to obtain the ing address pointing to the common physical memory.

Android core analysis (26) ----- SurfaceFlinger of Android GDI
36394 people read comments (28) collect reports

SurfaceFlinger of Android GDI
SurfaceFinger is translated as a Surface shipper by English. The structure of SufaceFlinger is not too complex, but the complexity of SufaceFlinger is its client construction. The main functions of SufaceFlinger are:

1) refresh the content of Layers (Surfaces) to the screen.

2) maintain the Zorder sequence of the Layer and perform the cropping calculation on the final output of the Layer.

3) In response to Client requirements, create a Layer to establish a connection with the Client's Surface.

4) receive Client requirements and modify Layer attributes (output size, Alpha, and other settings)

But as the actual meaning of shipping, what we need to know first is how to deliver, throw, shipping route, and shipping destination.

1 basic architecture of SurfaceFlinger

SurfaceFlinger management object:

MClientsMap: manages the connection between the client and the server.

ISurface, IsurfaceComposer: AIDL call interface instance

MLayerMap: The management object of the server Surface.

MCurrentState. layersSortedByZ: Layer array arranged in the Z-order sequence of the Surface.

GraphicPlane buffer output management

OpenGL ES: Graphics libraries such as graphic computing and image synthesis.

Gralloc. xxx. so this is a platform-related graphics buffer manager.

Pmem Device: Provides shared memory, which is only visible in gralloc. xxx. so and abstracted by gralloc. xxx. so on the upper layer.

2 SurfaceFinger Client and server object Relationship Diagram

The connection between the Client and SurfaceFlinger is shown in the following figure:

 

Client object: Generally, SurfaceComposerClient is used on the Client to deal with SurfaceFlinger.

 

3 main objects
3.1 DisplayHardware & FrameBuffer
First, SurfaceFlinger needs to operate on the screen, and a screen hardware buffer management framework needs to be established. When designing and supporting multiple screens, Android introduces the graphicPlane concept. SurfaceFlinger has a graphicPlane array. Each graphicPlane object corresponds to a DisplayHardware. In the current Android (2.1) design, the system supports a graphicPlane, so it also supports a DisplayHardware.

SurfaceFlinger: Data Structure Diagram of the Hardware buffer.

3.2 Layer

Method: setBuffer creates a display buffer on the SurfaceFlinger side. The buffer here refers to the HW property, PMEM Device File ing memory.

1) layer Rendering

VoidLayer: onDraw (const Region & clip) const

{

Intindex = mFrontBufferIndex;

GLuint textureName = mTextures [index]. name;

...

DrawWithOpenGL (clip, mTextures [index]);

}

3.2mCurrentState.layersSortedByZ
LayerBase array arranged in the Z-order sequence of the Surface, which is the basis for layer display occlusion. When each layer calculates its own visible area, it starts from the top layer of Z-order, considering the reduction of the blocked area, the visible area of its previous layer is its own invisible area. When the Layer is drawn, it is drawn from the bottom Layer of Z-order, which takes into account the superposition of the transparent Layer.

4 SurfaceFlinger runtime framework
We can see from the basic principles of the previous chapter <Android Service> that the SurfaceFlinger runtime framework exists in threadLoop, which is the main loop body of SurfaceFlinger. SurfaceFlinger will first run: SurfaceFlinger: readyToRun () before entering the main body loop ().

4.1 SurfaceFlinger: readyToRun ()
(1) Create GraphicPanle

(2) establish FrameBufferHardware (determine the output target)

Initialization: OpenGL ES

Build compatible mainSurface. Use eglCreateWindowSurface.

Create the OpenGL ES process context.

Create a main Surface (OpenGL ES ). The Init () @ DisplayHardware. cpp function of DisplayHardware initializes OpenGL and creates the primary Surface. Why is it the main Surface? Because all layers need to be drawn on the main Surface before the system throws the content of the main Surface to the real screen.

(3) binding the primary Surface

1) After DisplayHandware is started, hw. makeCurrent () binds the context of the main Surface and OpenGL ES processes to the context of SurfaceFlinger,

2) then all the operation destinations using EGL in all SurfaceFlinger processes are mSurface @ DisplayHardware.

In this way, when OpenGL draws a graph, the main Surface is recorded in the context of the process, so the displayed parameters of the main Surfce cannot be transmitted. The following are the actions of Layer-Draw and Hardware. flip:

4.2 ThreadLoop

(1) handleTransaction (...) : It mainly calculates whether each Layer has any Attribute Modification. If there is any modification, it needs to be re-painted.

(2) handlePageFlip ()

ComputeVisibleRegions: calculates the visible and overwritten regions of each Layer based on the Z-Order sequence. Crop output range calculation-

When generating a cropping area, each Layer must perform the following steps to calculate its display area on the screen according to Z_order:

1) Use your own W and H to show your initial visible regions

2) subtract the area covered by the window above

During painting, Layer copies the data of the corresponding region based on its own region.

(3) handleRepaint ()

ComposeSurfaces (refresh area required ):

Based on the visible area of each Layer and the intersection area of the area to be refreshed, the Z-Order sequence is drawn from the bottom to the main Surface.

(4) postFramebuffer ()

(DisplayHardware) hw. flip (mInvalidRegion );

EglSwapBuffers (display, mSurface): delivers mSruface to the screen.

5. Summary
What SurfaceFlinger does now is shown below:

 

Android core analysis (27) ----- dynamic structure of SurfaceFlinger in Android GDI
18942 people read comments (12) collect reports

SurfaceFlinger object creation process
 

1SurfaceSession Creation
When a client requests to create a Surface, it first establishes a Session with SurfaceFlinger, and then creates a Connection on the Session to return the Bclient object through the concept. WindowManagerService checks whether a SurfaceSession is established before adding the first window. If no SurfaceSession is established, a new instance is created to represent a connection with SurfaceFlinger.

New SurfaceSession () @ windowAddedLocked () @ WindowManagerService. java.

The SurfaceSession creation process is mostly completed in the C ++ Native space, as shown in the SurfaceSession initialization function: init () Local function. You can see from the initialization function below:

Init () <-> SurfaceSession_init@android_view_Surface.cpp

New SurfaceComposerClient

SurfaceSession creates a SurfaceComposerClient instance in the C ++ Native space. The establishment of this instance implements the following communication basis with SurfaceFlinger:

(1) established a proxy server for the proxy SurfaceFlinger Service

(2) establish an IsurfaceFlingerClient connection, establish a corresponding Client on SurfaceFlinger, and return the BClient to WindowManagerService.

 

2Surface Creation
In the WindowState class of WindowManagerService, we know that a Surface corresponds to each child of the main window. Win. createSurfaceLocked () @ relayoutWindow

Surface. java

Init () <--> Surface_init (...., Session, pid, dpy, w, h, format) @ android_view_Surface.cpp

SurfaceControl surface (client-> createSurface

On the mClient connection: Create the ISurface interface:

M_Client-> greateSurface (...)@

Bclient: createSurface (mId...) @ SurfaceFlinger. cpp

MFlinger-> createSurface (clientid ....)

CreateNormalSurfaceLocked

* CreateNormalSurfaceLocked: Creates a Layer to allocate the Display memory.

* CreatePushBuffersSurfaceLocked: Creates a LayBuffer without allocating the Display memory.

 

Android core analysis (28) ----- Surface & Canvas of Android GDI

20162 people read comments (41)

Surface & Canvas

Canvas indicates the meaning of the Canvas. The upper-layer plotting of Android is almost done through the Canvas instance. In fact, Canvas is more of an interface packaging. DrawPaints, drawPoints, drawRect, drawBitmap...

1 Essential Relationship Between Canvas and Surface

For this section, we will not study the Skia graphics engine itself. What we need to know is how the image we make is placed in that place and how the Canvas is connected to the Surface.

Canvas (Java) has a Native Canvas C ++ object in the C ++ Native layer.

LockCanvas () @ java

Surface_lockCanvas@android_view_Surface.cpp

SurfaceControl-> newSurface (control) @ Surface. cpp

Surface: lock operation:

GraphicBuffer: lock

GetBufferMapper (). lock <-> GraphicBufferMapper: lock

MAllocMod-> lock <-> gralloc_module_t: lock

SurfaceLock can be used to obtain the graphic buffer address corresponding to the Surface (mLockedBuffe.

(1) create a bitmap device connected to SkCanvas. The bitmap uses the image buffer address obtained above as its own bitmap memory.

(2) set the drawing target device of SkCanvas to this bitmap.

Through this process, the connection between SurfaceControl and Canvas is established.

 

2 View: OnDraw Source

When OpenGL is not used for rendering, the View attribute of Android changes. When a new View or Z-order changes, you need to re-draw the View on the System screen, at this time, our View will execute OnDraw (canvas). Where is the root cause?

ViewRoot. Java

Extends mtraversals (..)

...

Draw ()

Canvas = surface. lockCanvas (dirty );

...

MView. draw (canvas );

Draw (cavas) @ view. java

Background. draw (canvas );

OnDraw (cavas)

DispatchDraw (cavas)

OnDrawScrolbars (cavas)

Surface. unlockCanvasAndPost (canvas );