Android Camera Development Those pits

Copyright notice: This article by Swandre original article, reprint please indicate source:
Article original link: https://www.qcloud.com/community/article/168

Source: Tengyun https://www.qcloud.com/community

I recently developed an Android camera-related requirement that allows users to quickly take photos of a specific size (1:1 or 3:4) using a mobile camera and support stickers-related operations on the photos taken. Since I had not been in touch with the development of the Android camera, it took a lot of time and effort to put a lot of holes in the whole development process. This article summarizes the development of Android camera knowledge, process, and easy to encounter the pit, hoping to help future friends may contact the development of Android camera quickly, save time, less detours.

One of the two ways to develop camera apps in Android

Android system provides two ways to use the phone camera resources to achieve the shooting function, one is directly through the intent call system camera components, this method is quick and convenient, suitable for direct access to photos of the scene, such as uploading albums, Weibo, Friends circle send photos and so on. The other is to use the camera API to customize a custom camera, which is suitable for scenes that require a custom camera interface or the development of special camera functions, such as cropping, filtering, adding stickers, emoticons, location tags, etc. to the photo. This article is mainly about how to use the camera API to customize a custom camera in this direction.

Two. Key class parsing in camera API

Capturing using the camera API involves several key classes and interfaces:

Camera: The most important class for managing and manipulating camera resources. It provides a complete camera bottom interface, support camera resource switching, set preview/capture size, set aperture, exposure, focus and other related parameters, get preview/capture frame data and other functions, the main methods are the following:

• Open (): Gets the camera instance.
• Setpreviewdisplay (Surfaceholder): Binds surface that draws a preview image. Surface is a handle to the original image buffer (raw buffer) of the screen window, which allows you to get the canvas on the screen to do the work of drawing the view on the screen. With Surfaceholder, you can connect the camera to your surface, and when camera and surface are connected, the preview frame data from the camera can be displayed on the screen through your surface.
• Setprameters set camera parameters, including front and rear camera, flash mode, focus mode, preview and photo size.
• Startpreview (): Start the preview and display the preview frame data from the camera's underlying hardware on the bound surface.
• Stoppreview (): Stop previewing, turn off CAMRA of the underlying frame data, and draw on surface.
• Release (): Releasing the camera instance
• Takepicture (camera.shuttercallback shutter, camera.picturecallback raw, Camera.picturecallback jpeg): This is the main way to achieve camera photography , which contains three callback parameters. Shutter is the callback when the shutter is pressed, Raw is the callback that gets the original data of the camera, and the JPEG is the callback that gets the image data compressed into JPG format.

Surfaceview: The class used to draw the camera preview image, providing the user with a real-time preview image. Common view and derived classes share the same surface, and all drawing must be done in the UI thread. While Surfaceview is a special view, it doesn't share surface with other ordinary view, but internally holds a separate surface,surfaceview responsible for managing the surface's format, size, and display location. Because the UI thread also handles other interaction logic at the same time, the update speed and frame rate for the view is not guaranteed, and Surfaceview can be drawn in a separate thread because it holds a separate surface, thus providing a higher frame rate. The preview image of a custom camera is more suitable for display with Surfaceview because of the high requirement for update speed and frame rate.

Surfaceholder:surfaceholder is an abstract interface that controls surface, which controls the size and formatting of your surface, modifies the pixels on your surface, monitors changes in your surface, and more. The typical application of Surfaceholder is used in Surfaceview. Surfaceview obtains Surfaceholder instances through the Getholder () method, which manages the state of the monitor surface.

Surfaceholder.callback Interface: The interface responsible for monitoring surface state changes, there are three methods:

• Surfacecreated (Surfaceholder Holder): called immediately after surface has been created. When developing a custom camera, you can call Camera.open (), Camera.setpreviewdisplay () by overloading the function to get camera resources, connect to camera and surface, and so on.
• Surfacechanged (surfaceholder holder, int format, int width, int height): Called when surface has a format or size change. When developing a custom camera, you can call Camera.startpreview by overloading this function to turn on the camera preview so that the camera preview frame data can be passed to your surface to display a preview image in real time.
• Surfacedestroyed (Surfaceholder holder): Called before surface is destroyed. When developing a custom camera, you can call Camera.stoppreview (), Camera.release () to stop the camera preview and release the camera resources by overloading the function. Three. Customizing the camera development process customizing a custom camera app, you typically need to complete the following steps, as shown in flowchart 1:
• Detects and accesses the camera resource to check if the phone has a camera resource and, if present, requests access to the camera resource.
• The Create preview class creates a capture preview class that inherits from Surfaceview and implements the Surfaceholder interface. This class is able to display a real-time preview image of the camera.
• Build a preview layout with the capture preview class, you can create a layout file that blends the preview screen with the Designed user interface controls.
• Set up a photo listener to bind the listener to the user interface control so that it responds to user actions (such as pressing a button) to begin the photo process.
• Take a photo and save the file to convert the captured image into a bitmap file, and the final output is saved in a variety of commonly used format images.
• Release camera resource The camera is a shared resource and must be carefully managed for its life cycle. When the camera is finished, the application must release it correctly to avoid conflicts when other programs access it.
  
  Figure 1 Customizing the process for a custom camera
  Corresponding to code writing can be divided into three steps:
  The first step: Add the permissions used by camera-related features in Androidmanifest.xml, specifically stating the following:
  
  Step two: Write the camera operation function class Cameraoperationhelper. Using singleton mode to manage camera resources, encapsulate direct call of camera API, and provide callback interface for UI interaction with custom camera activity, its functions are as follows, mainly create \ release camera, connect \ start \ Close Preview interface, take pictures, autofocus, switch front and rear camera, Switch flash mode and so on, the specific implementation can refer to the official API documentation.
  
  
  The third step: Write custom camera activity, mainly custom camera interface, implement UI interactive logic, such as button click event Processing, icon resource switch, lens size switch animation. Here you need to declare a Surfaceview object to display the camera preview screen in real time. Manage the connection between the screen surface and the camera resource with Surfaceholder and its callback interface, and the camera previews the display/close of the image.
  Four. Some pits encountered during the development process I'll talk about some of the pits I stepped on when I developed my custom camera: 1. When activity is set to portrait, the Surfaceview preview image is reversed 90 degrees.

Before explaining this question, introduce the concept of several directions on the Android phone:
Screen orientation: In the Android system, the upper-left corner of the screen is the coordinate system's origin (0,0) coordinates. The origin extends to the right of the x-axis, and the origin extends downward to the y-axis positive direction.
Camera Sensor Direction: The image data of the phone camera is from the camera hardware image sensor, the sensor is fixed to the phone after a default framing direction, as shown in 2, the coordinates of the origin is located in the mobile phone in the upper left corner, that is, the screen application of the X-direction of the same. In other words, the X-direction of the screen applied to the vertical screen is 90 degrees.

Figure 2 Camera Sensor orientation
Camera Preview direction: Since the phone screen can be rotated 360 degrees, in order to ensure that the user can see the "correct" preview screen regardless of how to rotate the phone (this "correct" means that the screen displayed in the UI preview interface with the eyes of the human eye to see the picture is consistent), The underlying Android system rotates the data collected by the image sensor according to the direction of the current phone's screen, before it is sent to the display system, so that the preview screen is always "correct". The camera preview direction can be set by Setdisplayorientation () in the camera API. By default, this value is 0, consistent with the image sensor. So for a landscape app, the preview image won't be reversed 90 degrees because the screen orientation and preview orientation are the same. However, for portrait applications, the screen orientation and preview direction are vertical, so there is a 90-degree reversal. In order to get the correct preview, the camera's preview direction must be rotated 90 through the API, keeping it in line with the screen orientation, as shown in 3.

Figure 3 Camera Preview Direction
(The red arrow is the preview direction, the blue direction is the screen orientation)
Camera Orientation: When clicking the Take photo button, the photos taken are stored directly on the SDcard by the data captured by the image sensor, so the camera orientation is consistent with the sensor orientation.

2. Surfaceview preview image, take photo stretch deformation

Before you explain the problem, let's say a few dimensions related to the camera.
Surfaceview Size: The size of the view that is used to display the camera preview image in a custom camera app, which is the size of the screen when it fills full screen. The preview image shown here is Surfaceview the phone preview image.

Previewsize: The size of the preview frame data provided by the camera hardware. The preview frame data is passed to the Surfaceview, enabling the display of the preview image. The preview image corresponding to the preview frame data is called the camera preview image.
Picturesize: The size of the shot frame data provided by the camera hardware. Capturing frame data can generate bitmap files that are eventually saved as images in. jpg or. png formats. The image of the captured frame data here is called the camera shot image. Figure 4 illustrates the relationship between the above images and photos. The phone preview image is directly available to the user to see the image, it is generated by the camera preview image, take photos of the data from the camera to take pictures.

Figure 4 The relationship between several images
The following is the three kinds of tensile deformation I encountered during the development process:
1, the mobile Phone preview screen objects are stretched and deformed.
2. The object in the photograph is stretched and deformed.
3, click to take photos instantly, the phone preview screen will pause, the image is stretched deformation, and then preview the image after the restoration of normal.

The reason for phenomenon 1 is that the Surfaceview and previewsize have inconsistent length-to-width ratios. Because the image of the phone preview view is scaled by the camera preview image according to the size of the surfaceview, it inevitably causes the image to deform when the aspect ratio is inconsistent. The latter two phenomena are caused by the inconsistency between the previewsize and Picturesize's length-width ratio, and the relevant data are found, which is related to the underlying realization of some mobile phone camera hardware. In order to avoid the above several deformation phenomenon, in the development of the best Surfaceview, Previewsize, picturesize three dimensions to ensure the length and width of the same ratio. The implementation can get all the preview and capture dimensions supported by the camera hardware first through camera.getsupportedpreviewsizes () and camera.getsupportedpicturesizes (), It then filters out the size of the Surfaceview with the same length-to-width ratio and the appropriate size, and updates the settings by Camera.setprameters. Note: The size of the phone camera hardware supported on the market is generally 4:3 or 16:9 of the mainstream, so the Surfaceview size can not be too wonderful, preferably set to such a aspect ratio.

3. Various crash

The reason for the top two crash is that the camera hardware must be connected to surface before it is focused and photographed, and the camera preview is turned on and surface has received preview data. If the camera has not been executed yet. Setpreviewdisplay or the camera is not called. This run-time exception occurs before Startpreview is called Camera.autofocus or camera.takepicture. In the code that corresponds to the custom camera, be aware that before you perform camera.autofocus or camera.takepicture in the photo button event response, be sure to verify that the camera has a preview Surfaceview set up and that it has a preview enabled. Here's a way to tell the preview state: Camera.setpreviewcallback is the callback function that previews the frame data, which is called when Surfaceview receives the camera's preview frame data, so it can set whether to allow the focus and the camera's flag bit.

One more thing to note, Camera.takepicture () In the execution of the process will be executed Camera.stoppreview to capture frame data, performance as a preview screen stuck, and if the user clicked the button, that is, call Camera.takepicture, will also appear above the crash, so in the development, you may also need to block the camera button continuous click.
The third crash is related to the clipping of the image, because to support 1:1 or 4:3 size lens, so the preview view will need to be cropped, because it is a vertical screen application, so the clipping area of the coordinate system with the camera sensor direction is a 90-degree angle, in the clipping is, the X-direction on the screen, Corresponds to the height orientation on the shot image, while the Y direction on the screen corresponds to the width direction on the shot image. Therefore, we must pay attention to the transformation of coordinate system and the protection of cross-border.

4. Mirror effect of the front-facing camera

Android Camera hardware has a special setting, that is, for the front-facing camera, in the display preview view with a mirror-like effect, showing the image of the camera image. The photos are still imaged using a camera. See here, you may have some doubts, you might as well try your own Android phone on the front camera, compare the preview image and take photos of the difference. This is because the underlying camera does a horizontal rollover when passing the front camera preview data and flips the X-direction mirror 180 degrees. This change in the direction of the previous vertical screen preview will also affect, originally for the rear camera rotated 90 degrees can make the preview view correct, and the front camera, if also rotated 90 degrees, see the preview image is upside down (because x direction flipped 180 degrees), so must be rotated 180 degrees, To display correctly, 5 shows that you can combine the previous camera preview direction with the understanding.


Figure 5 Preview direction of front-facing camera
In addition, since the shooting image does not do a horizontal flip, so for the front camera shot out of the photos, the user will find that the preview is seen with the left and right flip. This can affect the user experience to some extent. To solve this problem, you can add a horizontal flip matrix transform to the image captured by the front-facing camera when generating the bitmap file.

5. Release of camera resources under lock screen

In order to save battery power, do not waste the camera resources, in the development of the custom camera, if the preview image is no longer required to display, such as pressing the home keyboard to switch backstage or lock screen, this time should close the preview and release the camera resources. Referring to the official API documentation, when Surfaceview becomes visible, it creates surface and triggers the surfacecreated callback function in the Surfaceholder.callback interface. When Surfaceview becomes invisible, it destroys surface and triggers the surfacedestroyed callback function. We can handle camera related actions in the corresponding callback function, such as connecting surface, turning preview on/off. As for the release of camera resources, it can be performed in Acticity OnPause. Accordingly, to re-restore the preview image, you can place the camera resource request and initialization in the Acticity Onresume, and then connect and turn on the preview by creating the Surfaceview.

However, during the development process, it is found that the program can run normally when you press the home button to cut back scenes. For a lock screen scenario, crash occurs when the camera resource is re-requested, saying that the camera resource access failed. So what's the reason? I added debug log in the code, checked the execution order of the code, the result is as follows:
The execution process when you press the home key on the custom camera page:

• Press the home key to run the program
• The order of activity calls is Onpause->onstop
• Surfaceview called the Surfacedestroyed method
• And then cut back into the program.
• The order of activity calls is Onrestart->onstart->onresume
• Surfaceview called the Surfacecreated->surfacechanged method
• For the lock screen, the execution process is:
• Activity only calls the OnPause method
• Activity calls the Onresume method after unlocking
• All the methods of Surfaceholder.callback in Surfaceview are not executed.

The problem was found, due to the lock screen, the callback callback method is not executed, causing the camera and preview connection has not been disconnected, the camera resources are released, so the system reported crash when re-requesting the camera resources. According to the above document, it is assumed that the lock screen system does not change the visibility of the surfaceview, so I try to onpause and Onresume by manually setting the Surfaceview visibile properties, the results found that the callback function can be triggered normally. Because the user should not be able to see the surfaceview in the background or lock screen, this method of manually changing the visibility of the surfaceview does not affect the user's experience.

Android Camera Development Those pits