Introduction to the camera architecture of Android [shown in the figure]

[Android developer]

Section 1 Overview of camera

Android's camera includes the viewfinder and photo taking functions. Currently, the camera program of the android release version has simple functions, but its program architecture is divided into two parts: client and server. They are built on the binder structure for inter-process communication between android.
Taking open-source android as an example, the camera code is mainly in the following directory:
The path of the Java program of camera:
Packages/apps/camera/src/COM/Android/camera/

Among them, camera. Java is the main implementation file.

Java local call part of camera (JNI ):
Frameworks/base/CORE/JNI/android_hardware_camera.cpp
This part of content is compiled to libandroid_runtime.so.

The main header file is in the following directory:
Frameworks/base/include/UI/

The underlying camera library is in the following directory:
Frameworks/base/libs/UI/
This part of content is compiled into library libui. So.

Camera Service Section:
Frameworks/base/camera/libcameraservice/
This part of content is compiled into the library libcameraservice. So.

To implement a specific function of camera, a hardware-related camera library is also required at the underlying level (for example, by calling the video for Linux driver and JPEG encoding program ). This library will be called by the service library libcameraservice. So of camera.

Part 2 interfaces and architecture of camera

2.1 Overall framework of camera
The structure of each database in camera can be expressed as follows:

In each library of the camera system, libui. so is at the core. It provides the camera class for the upper layer. The class libandroid_runtime.so provides the Java interface by calling the camera class, and implements android. hardware. camera class. Libcameraservice. so is a camera server program, which inherits libui. so class to implement the server function, and with libui. in so, another part of the content is communicated through inter-process communication (that is, the Binder Mechanism.

The libandroid_runtime.so and libui. So libraries are public. Apart from camera, there are other functions.
The header files of the camera part are in the frameworks/base/include/UI/directory, which corresponds to the directory of the libmedia. So Library source file frameworks/base/libs/UI.
The main header files of camera include the following:
■ Icameraclient. h
■ Camera. h
■ Icamera. h
■ Icameraservice. h
■ Camerahardwareinterface. h

In these header files. H provides interfaces for the upper layer, while several other header files provide some interface classes (that is, classes containing pure virtual functions ), these interface classes must be inherited by implementation classes before they can be used.
The whole camera can be divided into two parts, client and server, which run in two processes respectively. They use the binder mechanism to implement inter-process communication. In this way, when the client calls an interface, the function is implemented on the server, but the call in the client is like directly calling the function on the server. The communication between processes is invisible to the upper-Layer Program.

From the framework structure, icameraservice. h. icameraclient. H and icamera. class H defines the interface and architecture of meidaplayer, icameraservice. CPP and camera. CPP files are used to implement the camera architecture. The specific functions of camera are implemented by calling hardware-related interfaces at the lower layer.

In terms of the overall structure of camera, the class camera is the core of the entire system, and the icamera class provides interfaces for the main functions of camera. In terms of client calls, cameraservice is a camera service, it implements functions by calling the actual camera hardware interface. In fact, the red dotted box in the figure is the framework of the camera program. It mainly uses the Binder Mechanism of the Android system to complete communication. The blue part calls the camera hardware-related interfaces to complete specific camera service functions, and the other part provides the JNI interface for the upper-layer Java program. In the overall structure, the left side can be regarded as a client, and the right side can be regarded as a server.
Binder to implement inter-process communication.

2.2 header file icameraclient. h
Icameraclient. h is used to describe the interface of a camera client, which is defined as follows:
Class icameraclient: Public iinterface
{
Public:
Declare_meta_interface (cameraclient );
Virtual void shuttercallback () = 0;
Virtual void rawcallback (const sp & picture) = 0;
Virtual void callback (const sp & picture) = 0;
Virtual void framecallback (const sp & frame) = 0;
Virtual void errorcallback (status_t error) = 0;
Virtual void autofocuscallback (bool focused) = 0;
};
Class bncameraclient: Public bninterface
{
Public:
Virtual status_t ontransact (uint32_t code,
Const parcel & data,
Parcel * reply,
Uint32_t flags = 0 );
};
2.3 header file camera. h

Camera. H is the interface header file of camera. It is called by android_hardware_camera.cpp, which implements camera JNI. Camera. H mainly defines a camera class:

Class camera: Public bncameraclient, public ibinder: deathrecipient
{
Public:
Static SP connect ();
~ Camera ();
Void disconnect ();
Status_t getstatus () {return mstatus ;}
Status_t setpreviewdisplay (const sp & surface );
Status_t startpreview ();
Void stoppreview ();
Status_t autofocus ();
Status_t takepicture ();
Status_t setparameters (const string8 & Params );
String8 getparameters () const;
Void setshuttercallback (shutter_callback CB, void * cookie );
Void setrawcallback (frame_callback CB, void * cookie );
Void setjpegcallback (frame_callback CB, void * cookie );
Void setframecallback (frame_callback CB, void * cookie );
Void seterrorcallback (error_callback CB, void * cookie );
Void setautofocuscallback (autofocus_callback CB, void * cookie );
// Icameraclient Interface
Virtual void shuttercallback ();
Virtual void rawcallback (const sp & picture );
Virtual void callback (const sp & picture );
Virtual void framecallback (const sp & frame );
Virtual void errorcallback (status_t error );
Virtual void autofocuscallback (bool focused );
//......
}
From the interface, we can see that the camera class just implements the basic operations of a camera, such as startpreview, stoppreview, and takepicture. In the camera class, connect () is a static function used to obtain a camera instance. In this class, there are several functions for setting callback functions: setshuttercallback, setrawcallback, and setjpegcallback. These functions are provided for upper-layer use, and the upper-layer uses these set callback functions, these callback functions are called in the corresponding callback function. For example, the callback function pointer set by setshuttercallback is called by shuttercallback.
In the definition, the camera class dual inherits bncameraclient and ibinder: deathrecipient, and bncameraclient inherits bninterface, which is built for the Android-based basic class Binder Mechanism to implement process communication. In fact, according to the definition of the bninterface class template, the interface class inherits icameraclient and bbinder. This is a common definition method for Android.
After the camera class inherits the deathnotifier class, when this class is used as the ibinder, The binderdied function is called when the binder is about to died. The inheritance class basically implements a callback function.

2.4 header file icamera. h
The content described by icamera. H is an interface for implementing the camera function. Its definition is as follows:

Class icamera: Public iinterface
{
Public:
Declare_meta_interface (CAMERA );
Virtual void disconnect () = 0;
Virtual status_t setpreviewdisplay (const sp & surface) = 0;
Virtual void sethasframecallback (bool installed) = 0;
Virtual status_t startpreview () = 0;
Virtual void stoppreview () = 0;
Virtual status_t autofocus () = 0;
Virtual status_t takepicture () = 0;
Virtual status_t setparameters (const string8 & Params) = 0;
Virtual string8 getparameters () const = 0;
};
Class bncamera: Public bninterface
{
Public:
Virtual status_t ontransact (uint32_t code,
Const parcel & data,
Parcel * reply,
Uint32_t flags = 0 );
};
In the camera class, we mainly define the functional interfaces of camera. This class must be inherited before it can be used. It is worth noting that these interfaces are similar to those of the camera class, but they are not directly related. In fact, in various implementations of the camera class, it is generally done by calling the implementation class of the icamera class.

 

2.5 header file icameraservice. h
Icameraservice. h is used to describe a camera service. The definition is as follows:
Class icameraservice: Public iinterface
{
Public:
Declare_meta_interface (cameraservice );
Virtual SP connect (const sp & cameraclient) = 0;
};
Class bncameraservice: Public bninterface
{
Public:
Virtual status_t ontransact (uint32_t code,
Const parcel & data,
Parcel * reply,
Uint32_t flags = 0 );
};

Because of pure virtual functions, icameraservice and bncameraservice must be inherited for use. In icameraservice, only one connect () interface is defined, and the return value type is sp, this icamera is an interface that provides implementation functions. Note that icameraservice only supports the connect () function and does not disconnect the function. The disconnected function is provided by the icamera interface.

2.6 header file camerahardwareinterface. h
Camerahardwareinterface. h defines a camera underlying interface. The implementers of this class are ultimately implemented by camera.
Camerahardwareinterface is set to the camera hardware interface, as shown below:
Class camerahardwareinterface: Public Virtual refbase {
Public:
Virtual ~ Camerahardwareinterface (){}
Virtual SP getpreviewheap () const = 0;
Virtual status_t startpreview (preview_callback CB, void * User) = 0;
Virtual void stoppreview () = 0;
Virtual status_t autofocus (autofocus_callback, void * User) = 0;
Virtual status_t takepicture (shutter_callback, raw_callback, pai_callback, void * User) = 0;
Virtual status_t cancelpicture (bool cancel_shutter, bool cancel_raw, bool cancel_jpeg) = 0;
Virtual status_t setparameters (const cameraparameters & Params) = 0;
Virtual cameraparameters getparameters () const = 0;
Virtual void release () = 0;
Virtual status_t dump (int fd, const vector & ARGs) const = 0;
};

Use the C language to export symbols:
Extern "C" SP opencamerahardware ();

In other parts of the program, you can use opencamerahardware () to obtain a camerahardwareinterface, and then call the interface of camerahardwareinterface to complete the functions of camera.

Part 3 Main Implementation Analysis of camera

3.1 Java program
In packages/apps/camera/src/COM/Android/camera/
The camera. Java file in the directory contains the call to camera.
Reference to the package in camera. Java:
Import Android. Hardware. Camera. picturecallback;
Import Android. Hardware. Camera. size;

The camera class defined here inherits the activity class, which contains an android. Hardware. Camera
Public class camera extends activity implements view. onclicklistener, surfaceholder. Callback {
Android. Hardware. Camera mcameradevice;
}

Some functions of camera are called as follows:
Mcameradevice. takepicture (mshuttercallback, mrawpicturecallback, m?picturecallback );
Mcameradevice. startpreview ();
Mcameradevice. stoppreview ();

Startpreview, stoppreview, and takepicture interfaces are implemented through Java local call (JNI.

The camera. Java file in the frameworks/base/CORE/Java/Android/hardware/directory provides a Java class: Camera.
Public class camera {
}
Public void setparameters (parameters Params ){
Log. E (TAG, "setparameters ()");
// Params. Dump ();
Native_setparameters (Params. Flatten ());
}

In this class, most of the Code is obtained by calling the lower layer using JNI, for example:

In addition, for example, the following code:
Public final void setpreviewdisplay (surfaceholder holder ){
Setpreviewdisplay (holder. getsurface ());
}
Private native final void setpreviewdisplay (surface );

The two setpreviewdisplay parameters are different. The latter is a local method. The parameter is of the surface type. The former is implemented by calling the latter method, but its own parameters are of the surfaceholder type.

3.2 Java local call of camera
The Java local call (JNI) of camera is implemented in the file android_hardware_camera.cpp in the frameworks/base/CORE/JNI/directory.
Android_hardware_camera.cpp defines a gmethods array of jninativemethod (Java local call method) type, as shown below:
Static jninativemethod cammethods [] = {
{"Native_setup", "(ljava/lang/object;) V", (void *) android_hardware_camera_native_setup },
{"Native_release", "() V", (void *) android_hardware_camera_release },
{"Setpreviewdisplay", "(landroid/View/surface;) V", (void *) android_hardware_camera_setpreviewdisplay },
{"Startpreview", "() V", (void *) android_hardware_camera_startpreview },
{"Stoppreview", "() V", (void *) android_hardware_camera_stoppreview },
{"Sethaspreviewcallback", "(z) V", (void *) android_hardware_camera_sethaspreviewcallback },
{"Native_autofocus", "() V", (void *) android_hardware_camera_autofocus },
{"Native_takepicture", "() V", (void *) android_hardware_camera_takepicture },
{"Native_setparameters", "(ljava/lang/string;) V", (void *) android_hardware_camera_setparameters },
{"Native_getparameters", "() ljava/lang/string;", (void *) android_hardware_camera_getparameters}
};

The first member of jninativemethod is a string that represents the name of the Java local call method. This name is called in the Java program. The second member is also a string, indicates the parameters and return values of the local call method in Java. The Third Member is the C language function corresponding to the local call method in Java.
The register_android_hardware_camera function registers gmethods as a class "android/Media/camera". Its main implementation is as follows.
Int register_android_hardware_camera (jnienv * env)
{
// Register native functions
Return androidruntime: registernativemethods (ENV, "android/hardware/camera ",
Cammethods, nelem (cammethods ));
}

"Android/hardware/camera" corresponds to the Java class android. Hardware. Camera.

3.3 camera local library libui. So
The camera. cpp file in frameworks/base/libs/UI/is used to implement the interfaces provided by camera. H. An important part is as follows:

• Const sp & camera: getcameraservice ()
• {
• Mutex: autolock _ L (mlock );
• If (mcameraservice. Get () = 0 ){
• SP Sm = defaservicservicemanager ();
• SP binder;
• Do {
• Binder = Sm-> getservice (string16 ("media. Camera "));
• If (binder! = 0)
• Break;
• Logw ("cameraservice not published, waiting ...");
• Usleep (500000); // 0.5 s
• } While (true );
• If (mdeathnotifier = NULL ){
• Mdeathnotifier = new deathnotifier ();
• }
• Binder-> linktodeath (mdeathnotifier );
• Mcameraservice = interface_cast (binder );
• }
• Loge_if (mcameraservice = 0, "No cameraservice !? ");
• Return mcameraservice;
• }

The most important one is binder = Sm-> getservice (string16 ("media. camera "); this call is used to get a name named" media. camera "service, the type of the returned value of this call is ibinder, which is converted to the type icameraservice according to the implementation.
The implementation of a function connect is as follows:

• SP camera: connect ()
• {
• Sp c = new camera ();
• Const sp & cs = getcameraservice ();
• If (CS! = 0 ){
• C-> mcamera = cs-> connect (C );
• }
• If (c-> mcamera! = 0 ){
• C-> mcamera-> asbinder ()-> linktodeath (C );
• C-> mstatus = no_error;
• }
• Return C;
• }

Connect calls getcameraservice to get an icameraservice, and then obtains an icamera pointer through CS-> connect (C) of icameraservice. Call connect to get a camera pointer. Normally, the camera member mcamera has been initialized.
A specific function startpreview is as follows:

• Status_t camera: startpreview ()
• {
• Return mcamera-> startpreview ();
• }

These operations can be performed directly on mcamera, which is an icamera pointer.
The implementation of other functions is similar to that of setdatasource.
Some other files in libmedia. So have the same names as header files. They are:
Frameworks/base/libs/UI/icameraclient. cpp
Frameworks/base/libs/UI/icamera. cpp
Frameworks/base/libs/UI/icameraservice. cpp
In this case, although the bncameraclient and bncameraservice classes implement the ontransact () function, this class cannot be instantiated because pure virtual functions are not implemented.
Bncameraclient in icameraclient. cpp is not implemented elsewhere, while bncameraservice in icameraservice. cpp is inherited and implemented elsewhere. The successor implements the specific functions of the camera service.

3.4 camera service libcameraservice. So

Frameworks/base/camera/libcameraservice/is used to implement a camera service, which inherits the specific implementation of icameraservice.
Android. in the MK file, use the macro use_camera_stub to determine whether to use the real camera. If the macro is true, use camerahardwarestub. CPP and fakecamera. CPP constructs a false camera. If false, use cameraservice. CPP constructs an actual camera service.
Cameraservice. cpp inherits the implementation of bncameraservice. In this class, the class client is defined. cameraservice: client inherits bncamera. During the operation, the cameraservice: connect () function is used to obtain a cameraservice: client. during use, the camera function is mainly implemented by calling the interface of this class, because cameraservice: the client itself inherits the bncamera class, while the bncamera class inherits icamera, this class can be used as icamera.
The results of the cameraservice and cameraservice: client classes are as follows:

• Class cameraservice: Public bncameraservice
• {
• Class client: Public bncamera {};
• Wpmclient;
• }

In cameraservice, a static function instantiate () is used to initialize a camera service. The heat is as follows:

• Void cameraservice: instantiate (){
• Defaultservicemanager ()-> addservice (
• String16 ("media. Camera"), new cameraservice ());
• }

In fact, the cameraservice: instantiate () function registers a service named "media. Camera", which corresponds to the name called in camera. cpp.
The entire operating mechanism of camera is that icameraservice APIs can be called in camera. cpp. bpcameraservice is actually called, while bpcameraservice implements communication between two processes through the Binder Mechanism and bncameraservice. Bpcameraservice is implemented here as cameraservice. Therefore, camera. although CPP runs in another process, calling the icameraservice interface is like calling it directly. You can get an icamera pointer from connect (). The implementation of a true pointer is actually
Cameraservice: client.
The specific implementation of these camera functions is implemented by cameraservice: The constructor of the client is as follows:

• Cameraservice: Client: client (const sp & cameraservice,
• Const sp & cameraclient ):
• Mcameraservice (cameraservice), mcameraclient (cameraclient), mhardware (0)
• {
• Mhardware = opencamerahardware ();
• Mhasframecallback = false;
• }

In the constructor, call opencamerahardware () to obtain a pointer of the camerahardwareinterface type and use it as a member of mhardware. In the future, all operations on the actual camera are performed on this pointer. This is a simple direct call relationship.
In fact, the real camera function has been implemented by implementing the camerahardwareinterface class. In this library, camerahardwarestub. H and camerahardwarestub. the CPP file defines the interface of a pile module and is used without the camera hardware. For example, the file used in the simulator is camerahardwarestub. CPP and its dependent file fakecamera. CPP.
The structure of the camerahardwarestub class is as follows:

• Class camerahardwarestub: Public camerahardwareinterface {
• Class previewthread: Public thread {
• };
• };

In camerahardwarestub, a thread class previewthread is included. This thread is used to process preview, that is, to refresh the content of the viewfinder. The actual camera hardware interface can be implemented by calling the v4l2 capture driver. At the same time, a JPEG encoding program is required to encode the data retrieved from the driver into a jpeg file.