Brief introduction to the new Camera FW/HAL architecture in Android Camera API2.0

Brief introduction to the new Camera FW/HAL architecture in Android Camera API2.0

 

Frontier:

In the previous blog, the system architectures of Camera HAL1 and HAL3 are summarized. However, these architectures are still in the standard of Camera API1.0 for Camera APP development. With the constant updates of Camera3 and HAL3.0, Google first changed the entire architecture in the Framework to match the processing logic of Camera API1.0. Over time, google directly upgraded the Camera API to remove the original Camera. java interfaces are replaced by Camera API2 to fully match the previously designed Camera3 and HAL3. The advantage is that the entire architecture looks simpler.

This article briefly describes the processing logic of Camera in the Framewrok layer in API2.0, and compares what he gave up and added in API1.0?

 

1. Brand new Camera API2.0

In API2.0, you can no longer see Standard Operation interfaces such as startPreview, takePicture, and AutoFocus. Instead, there are a large number of Apis related to CaptureRequest/CaptureResult, google is used in API Level21, that is, Android5.0, and deprecate the old Camera. java-related interfaces.

 

2. The emergence of AIDL Technology in CameraService

AIDL is a method for implementing the C/S architecture at the Android Java layer. With the help of the Native Binder Mechanism, AIDL directly establishes a communication between processes at the Java layer. In Camera API2.0, we can see that a large number of ADIL processing methods appear in the Java layer. Instead, API1.0 needs to enter the Native layer to complete communication.

For CameraService, no matter which architecture or method it is, it should satisfy the following processes:

(1) Start CameraService;

(2) A Client connects to CameraService through CameraService Proxy and obtains a CameraClient Proxy. In the future, CameraClient Proxy will directly interact with CameraService.

(3) The Client must provide the Callback object interface to the Service, that is, each CameraClient of the Service needs a Callback Proxy to complete the data and message Callback.

No matter how Android is upgraded, the Camera module is basically in this mode, but the specific implementation method is different. In addition, the C/S architecture mentioned above is basically implemented through Binder IPC.

 

The traditional CameraService architecture requires the Service to create a Camera on the client under API1.0. It is a very obvious C/S architecture at the C ++ layer, however, in the API2.0 architecture, the original Camera on the Client layer is directly maintained by the Java-layer CameraDevice. The interface ICameraDeviceUser is implemented through AIDL, and a Camera proxy is maintained at the Java layer, the advantage is that the response speed is faster:

 

interface ICameraDeviceUser{    /**     * Keep up-to-date with frameworks/av/include/camera/camera2/ICameraDeviceUser.h     */    void disconnect();    // ints here are status_t    // non-negative value is the requestId. negative value is status_t    int submitRequest(in CaptureRequest request, boolean streaming);    int cancelRequest(int requestId);    int deleteStream(int streamId);    // non-negative value is the stream ID. negative value is status_t    int createStream(int width, int height, int format, in Surface surface);    int createDefaultRequest(int templateId, out CameraMetadataNative request);    int getCameraInfo(out CameraMetadataNative info);    int waitUntilIdle();    int flush();}

 

Similarly, we can see the ICameraService. AIDL file of CameraSevice at the Android Java layer:

 

interface ICameraService{    /**     * Keep up-to-date with frameworks/av/include/camera/ICameraService.h     */    int getNumberOfCameras();    // rest of 'int' return values in this file are actually status_t    int getCameraInfo(int cameraId, out CameraInfo info);    int connect(ICameraClient client, int cameraId,                    String clientPackageName,                    int clientUid,                    // Container for an ICamera object                    out BinderHolder device);    int connectPro(IProCameraCallbacks callbacks, int cameraId,                              String clientPackageName,                              int clientUid,                              // Container for an IProCameraUser object                              out BinderHolder device);    int connectDevice(ICameraDeviceCallbacks callbacks, int cameraId,                              String clientPackageName,                              int clientUid,                              // Container for an ICameraDeviceUser object                              out BinderHolder device);    int addListener(ICameraServiceListener listener);    int removeListener(ICameraServiceListener listener);    int getCameraCharacteristics(int cameraId, out CameraMetadataNative info);}

 

 

3. The Camera2Client disappears and the CameraDeviceClient is born.

CameraDeviceClient can be said to replace the upgraded Camera2Client in API1.0. In addition, in API2.0, the version of the Camera HAL Module is not allowed to be CAMERA_DEVICE_API_VERSION_1_0, whether Camera2Device or Camera3Device is used to connect to HAL3 is specified through the CAMERA_DEVICE_API_VERSION of HAL. In addition, the VERSION in HAL must be in CAMERA_DEVICE_API_VERSION_2_0 or above to allow the establishment of CameraDeviceClient.

 

4. Various Stream creators whose Native disappears

In my previous blog posts, I have been emphasizing the emergence of various types of Camera2Client files. Currently, these files can only stay in the world of API1.0, with the passage of time, the upgrade of the Android version may gradually disappear, which directly tells us that the implementation method of CameraHardwareInterface of HAL1.0 will no longer exist. Of course, everything depends on the vendor's implementation method.

In this case, CallbackProcessor, FrameProcessor, StreamingProcessor, and other modules appear in the Camera2Client. Each module is responsible for processing different services and processing and callback of related underlying video image data, the architecture of CPUConsumer and Surface is established for data processing. More operations on Buffer queue and dequeue are performed from the Consumer perspective, finally, the processing logic under the Camera3Device standard is implemented.

In the Framework layer of API2, these modules will no longer be used, instead of various Consumer that appear in the Java layer of Android5.0, similar to SurfaceFlinger In the Preview mode in the Java layer, this mode creates different types of Consumer, and then creates a BufferQueue in the Native layer, the IGraphicBufferConsumer of this BufferQueue is used to build the CPUConsumer, and IGraphicBufferProducer is added to CameraDevice through createStream.

Of course, in essence, it seems that the two implementation methods are the same. They both need to create a Stream, and then Stream needs the corresponding ANativeWindow Surface to obtain data from HAL3, once the data is obtained, the Consumer bound to the Surface can receive and process the buffer through OnFrameAvailable. The following interface illustrates that for different data processing modules in API2, you only need to get a Surface and use the AIDL implementation method to create a stream interface for receiving data.

Status_t CameraDeviceClient: createStream (int width, int height, int format, const sp
 
  
& BufferProducer) {ATRACE_CALL (); ALOGV (% s (w = % d, h = % d, f = 0x % x), _ FUNCTION __, width, height, format); status_t res; if (res = checkPid (_ FUNCTION __))! = OK) return res; Mutex: Autolock icl (mBinderSerializationLock); if (bufferProducer = NULL) {ALOGE (% s: bufferProducer must not be null, _ FUNCTION _); return BAD_VALUE;} if (! MDevice. get () return DEAD_OBJECT; // Don't create multiple streams for the same target surface {ssize_t index = mStreamMap. indexOfKey (bufferProducer-> asBinder (); if (index! = NAME_NOT_FOUND) {ALOGW (% s: Camera % d: Buffer producer already has a stream for it (ID % zd), _ FUNCTION __, mCameraId, index ); return ALREADY_EXISTS ;}}............. int32_t disallowedFlags = GraphicBuffer: Protocol | Protocol; int32_t allowedFlags = GraphicBuffer: Protocol | GraphicBuffer: USAGE_HW_TEXTURE | GraphicBuffer: USAGE_HW_COMPOSER; bool flexib LeConsumer = (consumerUsage & disallowedFlags) = 0 & (consumerUsage & allowedFlags )! = 0; sp
  
   
Binder; sp anw; if (bufferProducer! = 0) {binder = bufferProducer-> asBinder (); anw = new Surface (bufferProducer, useAsync); // create a local surface for Product} // TODO: remove w, h, f since we are ignoring them .......... res = mDevice-> createStream (anw, width, height, format, & streamId); // create stream return res ;}
  
 

In API2.0, we can see that different modules are provided in the Android Java layer to process different video image data. This process is similar to various Processor modules in Camera2Client, however, the latter packages the data processing and then returns it back to the APP. The former directly responds asynchronously to different modules at the Java layer and processes different data streams directly, data Streams in different modes, such as PREVIEW, RRCORD, STILL_CAPTURE, VIDEO_SNAPSHOT, and ZERO_SHUTTER_LAG, will be processed directly by MediaRecoder, SurfaceView, and ImageReader. In general, the efficiency will be better.

5. The FrameProcessorBase class is still used by FrameProcessor in the old version to process 3A-related information. It mainly refers to the ExtraResult of each frame of Callback to the APP, that is, 3A-related data information. This is the only module in API1 and API2 that requires manual CallBack. The other data streams are automatically processed by the aforementioned modules. The implementation method is described in section 2 (3, ICameraDeviceCallbacks is used to return the data of each video frame:

interface ICameraDeviceCallbacks{    /**     * Keep up-to-date with frameworks/av/include/camera/camera2/ICameraDeviceCallbacks.h     */    oneway void onCameraError(int errorCode);    oneway void onCameraIdle();    oneway void onCaptureStarted(int requestId, long timestamp);    oneway void onResultReceived(int requestId, in CameraMetadataNative result);}

6. The architecture of Camera3 in API2