Android4.4 General SurfaceFlinger Structure

Android4.4 General SurfaceFlinger Structure

SurfaceFlinger of Android4.4 GUI system framework

I. Android GUI framework:

SurfaceFlinger: Whenever a user program refreshes the UI, it will request a buffer (dequeueBuffer) through the intermediary BufferQueue, then fill in the UI information, and then drop it to SurfaceFlinger. SurfaceFlinger synthesize visibleRegion through multiple computations, it is thrown to the openGL layer for processing and then sent to the display.

According to the GUI Design Concept of the entire Android system, it is not difficult to guess that at least two local windows are required:

? SurfaceFlinger)

As SurfaceFlinger acts as the manager of all the UI interfaces in the system, SurfaceFlinger must directly or indirectly hold the "local window", which is the FramebufferNativeWindow.

? Application-oriented

This type of window is Surface (the difference between this window and the previous version is relatively large, and the local window of the previous version is SurfaceTextureClient)

The second Window can be directly displayed on the terminal screen-it uses the frame buffer, and the first Window is actually the space allocated from the memory buffer. When multiple applications exist in the system, this ensures that both of them can obtain a "Local window ", and these windows can finally be displayed on the screen-SurfaceFlinger will collect the display requirements of all programs and perform unified image mixing for them.

Ii. SurfaceFlinger and BufferQueue

SurfaceFlinger plays an important role in the process where the UI is completely displayed as a diplay, but it is responsible for "Flinger ", that is, the final "Drawing result" of all applications in the system is mixed and displayed on the physical screen in a unified manner. Other aspects, such as the painting process of each program, this is done by other things. This glorious task naturally falls on the shoulders of BufferQueue, a "one-to-one" tutor for every application, guides the UI program's "canvas application", "Painting Process" and a series of details. The following figure describes the relationship between the three elements:

Although they are related to the three, they only belong to two layers. The app belongs to the java layer, and BufferQueue/SurfaceFlinger belongs to the native layer. That is to say, BufferQueue is also affiliated with SurfaceFlinger, and all work is centered on SurfaceFlinger.

IGraphicBufferProducer is an important bridge between the app and BufferQueue. GraphicBufferProducer is responsible for the UI display requirements in a single application process, and it deals with BufferQueue. The workflow is as follows:

BpGraphicBufferProducer is the proxy object of GraphicBufferProducer on the client side. It is responsible for interacting with SF. GraphicBufferProducer obtains the buffer from BufferQueue through gbp (IGraphicBufferProducer Class Object) and then fills in the UI information, SF will be notified when the filling is complete, and the SF will perform the next operation on the Buffer after it is known. Typical production-consumer model.

The following describes the working modes of the client (producer) and server SurfaceFlinger (consumer:

First, the buffer here is a shared buffer, so it will certainly involve mutex locks, so there will be a variety of buffer states, generally, the buffer goes through the FREE-> DEQUEUED-> QUEUED-> ACQUIRED-> FREE process, as shown in the figure on the right:

? BufferQueue

It can be considered that BufferQueue is a service center, and the other two owners must use it to manage the buffer. For example, when the producer wants to obtain a buffer, it cannot directly contact consumer over BufferQueue, and vice versa.

? Producer

The producer is the person who fills in the buffer space. Normally, it is an application. Because the application constantly refreshes the UI, the generated display data is continuously written to the buffer. When the Producer needs to use a buffer, it will first initiate a dequeue application to the mediation BufferQueue before it can operate on the specified buffer. In this case, the buffer belongs to the producer. It can perform any necessary operations on the buffer, and other owners cannot intervene at the moment.

When the producer deems that a buffer has been written, it further calls the queue of BufferQueue. Literally, this function refers to "column" and visually expresses the buffer operation at this time-return the buffer to the BufferQueue queue. Once the queue succeeds, the owner changes to BufferQueue.

? Consumer

Consumers correspond to producers, and their operations are also controlled by BufferQueue. When a buffer is ready, the Consumer can start to work. It should be noted that, from the perspective of the roles played by each object, BufferQueue is an intermediary and belongs to the service provider; Producer is the Producer of the buffer content, its operations on the buffer zone are an "active" process; otherwise, consumer's buffer processing is "passive" and "waiting"-it must wait until a buffer is filled. In this model, how can we ensure that the Consumer can process the buffer in a timely manner? In other words, after a piece of buffer Data ready, how can we tell the Consumer to operate it?

After careful observation, we can see that BufferQueue also provides a special class named ProxyConsumerListener. The function interfaces include:

ClassProxyConsumerListener: public BnConsumerListener {public: // omitting the constructor virtual void onFrameAvailable ();/* When a buffer can be consumed, this function will be called, note that no shared lock protection */virtual voidonBuffersReleased ();/* BufferQueue notifies consumer that it has released one or more GraphicBuffer references in its slot */private: wp
 
  
MConsumerListener ;}
 

This makes it clear that when a frame of data is ready, BufferQueue will call onFrameAvailable () to notify the Consumer to consume the data.

The communication modes between BufferQueue and SurfaceFlinger are as follows:

There is also a pair of BpGraphicBufferConsumer/BnGraphicBufferConsumer that supports information transmission between them.

Iii. Detailed analysis of BufferQueue

First, describeBufferQueue class relationship:

The Core Function Analysis in BufferQueue is as follows:

Core member functions	Description
SetBufferCount	SetBufferCount updates the number of available buffer slots.
RequestBuffer	RequestBuffer returns the GraphicBuffer for slot N.
DequeueBuffer	DequeueBuffer gets the next buffer slot index for the producer to use.
QueueBuffer	QueueBuffer returns a filled buffer to the BufferQueue.
CancelBuffer	CancelBuffer returns a dequeued buffer to the BufferQueue
AcquireBuffer	AcquireBuffer attempts to acquire ownership of the next pending buffer BufferQueue.
ReleaseBuffer	ReleaseBuffer releases a buffer slot from the consumer back to the BufferQueue.

BufferQueue is the specific implementation of IGraphicBufferProducer and IGraphicBufferConsumer. During the request and SurfaceFlinger connection, the user requests SF to create a Layer. IGraphicBufferProducer obtains a BufferQueue object in this process, it is converted into an IGraphicBufferProducer class object to further interact with BufferQueue. The following is the key code:

status_t SurfaceFlinger::createNormalLayer(constsp
 
  & client,        const String8& name, uint32_t w,uint32_t h, uint32_t flags, PixelFormat& format,        sp
  
   * handle, sp
   
    *gbpsp
    
     * outLayer){    switch (format) {    case PIXEL_FORMAT_TRANSPARENT:    case PIXEL_FORMAT_TRANSLUCENT:        format = PIXEL_FORMAT_RGBA_8888;        break;    case PIXEL_FORMAT_OPAQUE:    } #ifdefNO_RGBX_8888    if (format == PIXEL_FORMAT_RGBX_8888)        format = PIXEL_FORMAT_RGBA_8888;#endif     *outLayer = new Layer(this, client, name,w, h, flags);    status_t err =(*outLayer)->setBuffers(w, h, format, flags);    if (err == NO_ERROR) {        *handle = (*outLayer)->getHandle();        *gbp =(*outLayer)->getBufferQueue();    }     ALOGE_IF(err, "createNormalLayer()failed (%s)", strerror(-err));    return err;}
    
   
  
 

The following is the implementation of getBufferQueue. It is very simple to get the BufferQueue object:

sp
 
  Layer::getBufferQueue() const {    return mBufferQueue;}
 

IGraphicBufferProducer is an interface class, and its implementation must be implemented in the sub-class BpGraphicBufferProducer. Let's look at this class:

classBpGraphicBufferProducer : public BpInterface
 
  {public:    BpGraphicBufferProducer(constsp
  
   & impl)        :BpInterface
   
    (impl)    {    }    virtual status_t requestBuffer(intbufferIdx, sp
    
     * buf) {        Parcel data, reply;       data.writeInterfaceToken(IGraphicBufferProducer::getInterfaceDescriptor());        data.writeInt32(bufferIdx);        status_t result =remote()->transact(REQUEST_BUFFER, data,&reply);        if (result != NO_ERROR) {            return result;        }        bool nonNull = reply.readInt32();        if (nonNull) {            *buf = new GraphicBuffer();            reply.read(**buf);        }        result = reply.readInt32();        return result;    }    virtual status_t dequeueBuffer(int*buf, sp
     
      * fence, bool async,            uint32_t w, uint32_t h, uint32_tformat, uint32_t usage) {        Parcel data, reply;       data.writeInterfaceToken(IGraphicBufferProducer::getInterfaceDescriptor());        data.writeInt32(async);        data.writeInt32(w);        data.writeInt32(h);        data.writeInt32(format);        data.writeInt32(usage);        status_t result = remote()->transact(DEQUEUE_BUFFER, data,&reply);        if (result != NO_ERROR) {            return result;        }        *buf = reply.readInt32();        bool nonNull = reply.readInt32();       if (nonNull) {            *fence = new Fence();            reply.read(**fence);        }        result = reply.readInt32();        return result;    }     virtual status_t queueBuffer(intbuf,            const QueueBufferInput& input,QueueBufferOutput* output) {        Parcel data, reply;       data.writeInterfaceToken(IGraphicBufferProducer::getInterfaceDescriptor());        data.writeInt32(buf);        data.write(input);        status_t result = remote()->transactQUEUE_BUFFER, data, &reply);        if (result != NO_ERROR) {            return result;        }        memcpy(output,reply.readInplace(sizeof(*output)), sizeof(*output));        result = reply.readInt32();        return result;
     
    
   
  
 

Some member functions are removed and only key member functions are attached. First, dequeueBuffer and queueBuffer do not actually operate on the Buffer. Pay attention to the red part, he only sends a "message" to notify the recipient

Dequeue a Buffer or queue a Buffer. Corresponding BnGraphicBufferProducer to receive messages.

OnTransact in BnGraphicBufferProducer is responsible for this matter:

status_tBnGraphicBufferProducer::onTransact(    uint32_t code, const Parcel& data,Parcel* reply, uint32_t flags){    switch(code) { case DEQUEUE_BUFFER: {           CHECK_INTERFACE(IGraphicBufferProducer, data, reply);            bool async      = data.readInt32();            uint32_t w      = data.readInt32();            uint32_t h      =data.readInt32();            uint32_t format = data.readInt32();            uint32_t usage  = data.readInt32();            int buf;            sp
 
   fence;            int result = dequeueBuffer(&buf, &fence, async, w, h,format, usage);            reply->writeInt32(buf);            reply->writeInt32(fence !=NULL);            if (fence != NULL) {                reply->write(*fence);            }            reply->writeInt32(result);            return NO_ERROR;        } break;        case QUEUE_BUFFER: {           CHECK_INTERFACE(IGraphicBufferProducer, data, reply);            int buf = data.readInt32();            QueueBufferInput input(data);            QueueBufferOutput* const output =                    reinterpret_cast
  
   (                           reply->writeInplace(sizeof(QueueBufferOutput)));            status_t result = queueBuffer(buf, input, output);            reply->writeInt32(result);            return NO_ERROR;        } break;                 }    return BBinder::onTransact(code, data,reply, flags);}
  
 

Some branches are omitted. Pay attention to the red part to find that dequeueBuffer and queueBuffer are called here,

Their implementation is in BufferQueue, and it is only in the BufferQueue field. Here, the client establishes a connection with BufferQueue, And the next thing is the internal processing of BufferQueue, as is the relationship between BufferQueue and SuefaceFlinger.

4. SurfaceFlinger processes the buffer

Here we will use two pictures to introduce the overall message processing mechanism and workflow of SurfaceFlinger:

More specific code flows have been analyzed before.

Next we will continue to analyze handleMessageRefresh, which is the core processing function of SuefaceFlinger.

VoidSurfaceFlinger: callback () {ATRACE_CALL (); preComposition (); rebuildLayerStacks (); events (); doDebugFlashRegions (); doComposition (); postComposition ();//......... Omitted}

PreComposition (); prepare in advance the buffer of the UI information sent from the client side;

RebuildLayerStacks (); recreate the visible area on each screen;

SetUpHWComposer (); initialize a hardware container;

DoDebugFlashRegions (); this function is generally returned;

DoComposition (); the actual merging process, and the merged BUFFER is processed by opengl es. After processing, postFramebuffer () is sent to display;

DoComposition ()

voidSurfaceFlinger::doComposition() {    ATRACE_CALL();    const bool repaintEverything =android_atomic_and(0, &mRepaintEverything);    for (size_t dpy=0 ; dpy
 
  &hw(mDisplays[dpy]);        if (hw->canDraw()) {            // transform the dirty region intothis screen's coordinate space            const Region dirtyRegion(hw->getDirtyRegion(repaintEverything));             // repaint the framebuffer (ifneeded)            doDisplayComposition(hw, dirtyRegion);            hw->dirtyRegion.clear();            hw->flip(hw->swapRegion);           hw->swapRegion.clear();        }        // inform the h/w that we're donecompositing      hw->compositionComplete();    }    postFramebuffer();}
 

DoDisplayComposition (hw, dirtyRegion); the core function responsible for rendering is:

DoDisplayComposition-> doComposeSurfaces-> draw-> onDraw-> drawWithOpenGL. always go to the OPENGL layer. After Opengl pasted the image, the flip function was called, which is very different from the previous version. The previous version of flip is in postFramebuffer, and the function content has been greatly changed, only count 1.

It is explained here that the UI display is a dual-buffer mechanism. Every time a buffer is drawn, flip is required, that is, switching. However, this version has been integrated into postFramebuffer:

Paste key code

R = hwc. commit ();

The member variable hwc is an HWComposer object generated in DisplayHardware: init. As long as the HWC_HARDWARE_MODULE_ID module can be loaded normally and hwc_open can enable the hwc_composer_device device, initCheck () will return NO_ERROR, otherwise it will be NO_INIT.

At this point, we use HWComposer: commit to execute flip. This function directly calls the following hardware interface:

MHwc-> set (mHwc, mNumDisplays, mLists );

Set () is basically equivalent to eglSwapBuffers. The prototype is as follows:

Int (* set) (struct hwc_composer_device * dev, hwc_display_t dpy,

Hwc_layer_list_t * list );

The last list must be exactly the same as the previous prepare () list. If the list is empty or the number of lists is 0, SurfaceFlinger has used OpenGL ES for composition. set is the same as eglSwapBuffers. If the list is not empty and the compositionType of the layer is = HWC_OVERLAY, HWComposer needs to merge hardware.

If the execution is successful, set returns 0; otherwise, it is HWC_EGL_ERROR.

If it fails, there will be another sentence:

If (r)

{

Hw-> hwcSwapBuffers ();

}

It also serves the same purpose as flip. Its function trend is:

HwcSwapBuffers-> eglSwapBuffers-> swapBuffers-> advanceFrame-> fbPost-> post.

Once the switch is complete, the display is displayed at the underlying layer.

Here we mainly study the swapBuffers function:

EGLBooleanegl_window_surface_v2_t::swapBuffers(){    //………….    nativeWindow->queueBuffer(nativeWindow,buffer, -1);     // dequeue a new buffer    if (nativeWindow->dequeueBuffer（nativeWindow, &buffer, &fenceFd)== NO_ERROR) {        sp
 
   fence(new Fence(fenceFd));        if(fence->wait(Fence::TIMEOUT_NEVER)) {           nativeWindow->cancelBuffer(nativeWindow, buffer, fenceFd);            return setError(EGL_BAD_ALLOC,EGL_FALSE);        }//。。。。。。}
 

This is consistent with the process of the figure I started. I joined the queue through queueBuffer, and then applied for a new buffer through dequeueBuffer FOR THE NEXT refresh.