Data Flow and control flow of the preview of Android4.2.2 and the final preview display

This article is a summary of the source code. Please specify the source for the transfer. Thank you.

Welcome to your discussion. Qq: 1037701636 email: gzzaigcn2012@gmail.com

Android source code Version: 4.2.2; hardware platform A31

Step 1: pass in a window to HAL in setPreviewWindow at CameraService.

Status_t setPreviewWindow (const sp & buf) {ALOGV ("% s (% s) buf % p", _ FUNCTION __, mName. string (), buf. get (); if (mDevice-> ops-> set_preview_window) {mPreviewWindow = buf; mHalPreviewWindow. user = this; ALOGV ("% s & mHalPreviewWindow % p mHalPreviewWindow. user % p ", _ FUNCTION __, & mHalPreviewWindow, mHalPreviewWindow. user); return mDevice-> ops-> set_preview_window (mDevice, buf. get ()? & MHalPreviewWindow. nw: 0); // call the underlying hardware hal interface} return INVALID_OPERATION ;}

The parameter buf passed to HAL is a Surface, and another variable is about the data stream operation in the preview window nw.

    struct camera_preview_window {        struct preview_stream_ops nw;        void *user;    };

The initial values of this variable are as follows: These function interfaces seem familiar. Indeed, in SurfaceFlinger, the client's Surface uses these interfaces to apply for a graphic cache from SurfaceFlinger and process the graphic cache display, only the previous operations have handed in the eglswapbuf () of OpenGL ES to perform the following dequeueBuffer and enqueuebuffer operations on this local window. In the preview of Camera, these operations are manually completed.

    void initHalPreviewWindow()    {        mHalPreviewWindow.nw.cancel_buffer = __cancel_buffer;        mHalPreviewWindow.nw.lock_buffer = __lock_buffer;        mHalPreviewWindow.nw.dequeue_buffer = __dequeue_buffer;        mHalPreviewWindow.nw.enqueue_buffer = __enqueue_buffer;        mHalPreviewWindow.nw.set_buffer_count = __set_buffer_count;        mHalPreviewWindow.nw.set_buffers_geometry = __set_buffers_geometry;        mHalPreviewWindow.nw.set_crop = __set_crop;        mHalPreviewWindow.nw.set_timestamp = __set_timestamp;        mHalPreviewWindow.nw.set_usage = __set_usage;        mHalPreviewWindow.nw.set_swap_interval = __set_swap_interval;        mHalPreviewWindow.nw.get_min_undequeued_buffer_count =                __get_min_undequeued_buffer_count;    }

Step2. continue the preceding preview processing. The CameraClinet at CameraService has called the startPreview function of CameraHardwareInterface. The actual operation is to operate the Camera device at HAL as follows:

    status_t startPreview()    {        ALOGV("%s(%s)", __FUNCTION__, mName.string());        if (mDevice->ops->start_preview)            return mDevice->ops->start_preview(mDevice);        return INVALID_OPERATION;    }

Step go to HAL to view the processing of Preview

Status_t CameraHardware: doStartPreview (){........... res = camera_dev-> startDevice (mCaptureWidth, mCaptureHeight, org_fmt, video_hint); // start the device ......}

Call the V4L2 device to start video stream collection. The startDevice () function better explains the start of preview, that is, the start of video collection.

Status_t V4L2CameraDevice: startDevice (int width, int height, uint32_t pix_fmt, bool video_hint) {LOGD ("% s, wxh: % dx % d, fmt: % d ", _ FUNCTION __, width, height, pix_fmt); Mutex: Autolock locker (& mObjectLock); if (! IsConnected () {LOGE ("% s: camera device is not connected. ", _ FUNCTION _); return EINVAL;} if (isStarted () {LOGE (" % s: camera device is already started. ", _ FUNCTION _); return EINVAL;} // VE encoder need this formatmVideoFormat = pix_fmt; mCurrentV4l2buf = NULL; mVideoHint = video_hint; mCanBeDisconnected = false; // set capture mode and fps // CHECK_NO_ERROR (v4l2setCaptureParams (); // do not check this er Rorv4l2setCaptureParams (); // set v4l2 device parameters, it maybe change the value of mFrameWidth and mFrameHeight. CHECK_NO_ERROR (v4l2SetVideoParams (width, height, pix_fmt); // v4l2 request buffersint buf_cnt = (mTakePictureState = TAKE_PICTURE_NORMAL )? 1: NB_BUFFER; CHECK_NO_ERROR (v4l2ReqBufs (& buf_cnt); // buf applies for mBufferCnt = buf_cnt; // v4l2 query callback (v4l2QueryBuf (); // buffer query, complete mmap and other operations // stream on the v4l2 deviceCHECK_NO_ERROR (v4l2StartStreaming (); // start the video stream collection mCameraDeviceState = STATE_STARTED; duration = 1000000/10; mFaceDectectAfter = 1000000/15; mPreviewAfter = 1000000/24; return NO_ERROR ;}

This is a complete reference to the video collection and processing process of V4L2:

1. v4l2setCaptureParams () sets collection parameters;

2. v4l2QueryBuf (): obtains the kernel image cache information and maps all kernel image caches to the current process. Convenient processing of user space

3. v4l2StartStreaming (): Enable the video collection process of V4L2.

Step 4: Image Collection thread bool V4L2CameraDevice: captureThread ();

The content of this function is complex, but the core is ret = getPreviewFrame (& buf) to obtain the current frame image:

Int V4L2CameraDevice: getPreviewFrame (v4l2_buffer * buf) {int ret = UNKNOWN_ERROR; buf-> type = disabled; buf-> memory = disabled; ret = ioctl (mCameraFd, VIDIOC_DQBUF, buf); // get a frame of data if (ret <0) {LOGW ("GetPreviewFrame: VIDIOC_DQBUF Failed, % s", strerror (errno); return _ LINE __; // can not return false} return OK ;}

A typical VIDIOC_DQBUF command is called to extract a frame of image cache to the user space for display.

The current platform defines a V4L2BUF_t struct to represent the currently collected frame of the image, respectively recording the physical address of Y and C and the virtual address of the user space. The virtual address is the ing of the kernel collection cache.

typedef struct V4L2BUF_t{unsigned intaddrPhyY;// physical Y address of this frameunsigned intaddrPhyC;// physical Y address of this frameunsigned intaddrVirY;// virtual Y address of this frameunsigned intaddrVirC;// virtual Y address of this frameunsigned intwidth;unsigned intheight;int index;// DQUE id numberlong longtimeStamp;// time stamp of this frameRECT_tcrop_rect;intformat;void*           overlay_info;// thumb unsigned charisThumbAvailable;unsigned charthumbUsedForPreview;unsigned charthumbUsedForPhoto;unsigned charthumbUsedForVideo;unsigned intthumbAddrPhyY;// physical Y address of thumb bufferunsigned intthumbAddrVirY;// virtual Y address of thumb bufferunsigned intthumbWidth;unsigned intthumbHeight;RECT_tthumb_crop_rect;int thumbFormat;int refCnt; // used for releasing this frameunsigned intbytesused;      // used by compressed source}V4L2BUF_t;

Let's take a look at the initialization code of this struct:

V4L2BUF_t v4l2_buf; if (mVideoFormat! = V4L2_PIX_FMT_YUYV & mCaptureFormat = V4L2_PIX_FMT_YUYV) {v4l2_buf.addrPhyY = mVideoBuffer. buf_phy_addr [buf. index]; v4l2_buf.addrVirY = mVideoBuffer. buf_vir_addr [buf. index];} else {v4l2_buf.addrPhyY = buf. m. offset & 0x0fffff; // kernel physical address v4l2_buf.addrVirY = (unsigned int) mMapMem. mem [buf. index]; // virtual address} v4l2_buf.index = buf. index; // The index v4l2_buf.timeStamp = mCurFrameTimestamp; v4l2_buf.width = mFrameWidth; v4l2_buf.height = mFrameHeight; v4l2_buf.crop_rect.left = mRectCrop. left; v4l2_buf.crop_rect.top = mRectCrop. top; v4l2_buf.crop_rect.width = mRectCrop. right-mRectCrop. left + 1; v4l2_buf.crop_rect.height = mRectCrop. bottom-mRectCrop. top + 1; v4l2_buf.format = mVideoFormat;

AddrPhy and addrViry respectively record the physical addresses of Y and C and the virtual addresses of user spaces. This address is directly set through the index of the current Buf. Why? Because the mmap operation in the image cache area of the kernel maps each cache to the user space one by one with its Index, and records the physical and virtual addresses of the cache, this is mainly to facilitate subsequent image display.

Step 5: bool V4L2CameraDevice: previewThread () // preview thread
When a frame of data is obtained, the preview thread must be notified to display the image. The collection thread and display thread are waiting through pthread_cond_wait (& mPreviewCond, & mPreviewMutex.

Bool preview: previewThread () // preview thread {V4L2BUF_t * pbuf = (V4L2BUF_t *) OSAL_Dequeue (& mQueueBufferPreview); // obtain preview frame buffer information if (pbuf = NULL) {// LOGV ("picture queue no buffer, sleep... "); pthread_mutex_lock (& mPreviewMutex); pthread_cond_wait (& mPreviewCond, & mPreviewMutex); // wait for pthread_mutex_unlock (& mPreviewMutex); return true;} Mutex :: autolock locker (& mObjectLock); if (mMapMem. mem [pbuf-> index] = NULL | pbuf-> addrPhyY = 0) {LOGV ("preview buffer have been released... "); return true;} // callbackmCallbackNotifier-> onNextFrameAvailable (void *) pbuf, mUseHwEncoder); // call back to collect frame data // previewif (isPreviewTime ()) // preview {mPreviewWindow-> onNextFrameAvailable (void *) pbuf); // frames can be displayed} // LOGD ("preview id: % d", pbuf-> index ); releasePreviewFrame (pbuf-> index); return true ;}

The preview thread mainly implements two tasks: one is to complete the callback of the image cache data for the upper-layer to use, and the other is to deliver the display.

Step 6: how to display the preview thread?

Bool PreviewWindow: onNextFrameAvailable (const void * frame) // use the local window surface for initialization {int res; Mutex: Autolock locker (& mObjectLock ); v4L2BUF_t * pv4l2_buf = (V4L2BUF_t *) frame; // address of the image of a frame ...... res = mPreviewWindow-> set_buffers_geometry (mPreviewWindow, mPreviewFrameWidth, mPreviewFrameHeight, format); // You can familiarize yourself with the buffer ry of the local window ...... res = mPreviewWindow-> dequeue_buffer (mPreviewWindow, & buffer, & stride); // apply SF caches bufferqueue images. Return the current process address to buffer .................. res = grbuffer_mapper.lock (* buffer, GRALLOC_USAGE_SW_WRITE_OFTEN, rect, & img); // put the address in the ing back buffer information into img, used to fill ............. mPreviewWindow-> enqueue_buffer (mPreviewWindow, buffer); // submit it to surfaceFlinger for display ............}

The above code real-time delivery of the local window image to SurfaceFlinger. Why? Let's look at the following analysis:

1. What is the mPreviewWindow member variable in the PreviewWindow class?

This is set from setPreviewDisplay () on the application end. The place passed to HAL is in the initialize function of CameraHardwareInterface:

Return mDevice-> ops-> set_preview_window (mDevice, buf. get ()? & MHalPreviewWindow. nw: 0); // call the underlying hardware hal interface}

The operation of nw has been described in step 1, and some operations related to initialization are described.

2. dequeue_buffer is used as an example:

    static int __dequeue_buffer(struct preview_stream_ops* w,                                buffer_handle_t** buffer, int *stride)    {        int rc;        ANativeWindow *a = anw(w);        ANativeWindowBuffer* anb;        rc = native_window_dequeue_buffer_and_wait(a, &anb);        if (!rc) {            *buffer = &anb->handle;            *stride = anb->stride;        }        return rc;    }

Call the local window and obtain the ANativeWindow object through w to see the implementation of the macro:

    static ANativeWindow *__to_anw(void *user)    {        CameraHardwareInterface *__this =                reinterpret_cast
 
  (user);        return __this->mPreviewWindow.get();    }#define anw(n) __to_anw(((struct camera_preview_window *)n)->user)
 

First, get the user object as the CameraHardwareInterface object, and use it to obtain the previously initialized Surface object, namely, the member variable mPreviewWindow (which belongs to the ANativeWindow class of the local window ).

3. Local window operations

static inline int native_window_dequeue_buffer_and_wait(ANativeWindow *anw,        struct ANativeWindowBuffer** anb) {    return anw->dequeueBuffer_DEPRECATED(anw, anb);}

The above process is actually to call the Surface object created at the application layer, which has been fully packaged and passed to CameraService for plotting and rendering. BpCamera

// Pass the buffered Surface to the camera service status_t setPreviewDisplay (const sp
 
  
& Surface) {ALOGV ("setPreviewDisplay"); Parcel data, reply; data. writeInterfaceToken (ICamera: getInterfaceDescriptor (); Surface: writeToParcel (surface, & data); // data Packaging remote ()-> transact (SET_PREVIEW_DISPLAY, data, & reply ); return reply. readInt32 ();}
 

At BnCamera, a Surface at the CameraService is created internally, but the parameters at the client are initialized. That is, the two are in different processes, but the information contained is exactly the same.

Case SET_PREVIEW_DISPLAY: {ALOGV ("SET_PREVIEW_DISPLAY"); CHECK_INTERFACE (ICamera, data, reply); sp
 
  
Surface = Surface: readFromParcel (data); reply-> writeInt32 (setPreviewDisplay (surface); // set sueface return NO_ERROR;} break;
 

Surface::Surface(const Parcel& parcel, const sp
 
  & ref)    : SurfaceTextureClient(){    mSurface = interface_cast
  
   (ref);    sp
   
     st_binder(parcel.readStrongBinder());    sp
    
      st;    if (st_binder != NULL) {        st = interface_cast
     
      (st_binder);    } else if (mSurface != NULL) {        st = mSurface->getSurfaceTexture();    }    mIdentity   = parcel.readInt32();    init(st);}
     
    
   
  
 

Here, the Surface is established through mSurface to communicate with SurfaceFlinger, because the Surface at the Camera client previously communicates with SurfaceFLinger through Binder, now we need to write the original Bpxxx-related data to CameraService for further Binder communication with SurfaceFlinger, such as Bufferqueue communication between queueBuffer () and SurfaceFlinger.

4. therefore, the anw-> dequeueBuffer function is exactly the same as the previous client that understands SurfaceFlinger from Android Bootanimation, only the Surface created by the Bootanimation process is handed over to OpenGL Es for plotting at the underlying level, such as dequeue (cache application, filling the current buffer) and enqueue (column rendering, see the article dequeueBuffer application and allocation in the graphic cache area of Android4.2.2 SurfaceFlinger. The specific drawing is not described here. This method has been used to establish a connection with SurfaceFlinger and is finally displayed by SurfaceFlinger.