---basic knowledge outline of live video technology in Android

First, preface

Recently, all kinds of video Live app are everywhere, all kinds of PA screen, of course, we also need to experience, about the video live software is not introduced here, in the people who are not technology, live is a trend, is a kind of entertainment, but as a high-tech, we have to look, more importantly, learning technology, In fact, Android video technology Nothing to say, because the information on the Internet a lot, but before most of the video technology has appeared in the broadcast, is the mainstream video player, the most important one of the technology is the video codec, this will be in the next article in detail about the processing technology of video. But now live technology is in the previous video technology has another requirement is video recording, now recording a lot of thanks to the hardware camera. But besides this technology, there are other things we can do to solve this problem with mobile devices. This follow-up will also say how to use the device to record video.

ii. Summary of Knowledge

From the beginning of this article we will introduce about the video processing in Android, here mainly include: Android camera technology, recording video, video player and other knowledge points, this article is to introduce the general knowledge points, for the subsequent chapters of knowledge points to do the groundwork, In fact, in the previous study of Android is the most afraid of the picture and video processing technology, because these technologies have a basic requirement is the byte operation, consider a lot of word throttling processing, this in Android has a class Bytebuffer, this class will run through all of our subsequent chapters of the Knowledge Point, This class is also the midpoint of our next article, which will be indispensable in the video streaming process.

Let's take a look at one of the processing outline plots for video in Android:

This picture is too big, if you can not see clearly enough to download, and then zoom in on the view. Here are some of the points of knowledge that are covered in this diagram:

III. Structure of knowledge

First, Video coding

There are two ways to encode video in Android, mainly two core classes, one is Mediacodec and Mediarecorder, what is the difference between these two classes? It's good to understand that they can encode the video, but the only difference is that the MEDIACODEC is more biased toward the native, while the mediarecorder is biased toward the upper package.

1, Mediacodec class

Mediacodec can handle specific video streams, mainly with these methods:

Getinputbuffers: Gets the input stream queue that needs to encode the data, and returns a Bytebuffer array

Queueinputbuffer: Input into queue

Dequeueinputbuffer: Fetching data from the input stream queue for encoding operations

Getoutputbuffers: Gets the data output stream queue after the codec, and returns a Bytebuffer array

Dequeueoutputbuffer: Extracting data from the output queue after the encoding operation

Releaseoutputbuffer: Processing complete, releasing Bytebuffer data

Look like:

See here:

The video stream has an input queue, and an output queue, respectively, corresponding to the Getinputbuffers and Getoutputbuffers methods to obtain this queue, Then for the input stream this end has two methods one is Queueinputbuffers is the video into the queue, Dequeueinputbuffer is to extract data from the input stream queue to encode and decode operations, In the output side there is a Dequeueoutputbuffer method to obtain video data from the output queue, the Releaseoutputbuffers method will be processed output video stream data Bytebuffer put back into the video stream output queue, reuse again. This allows the video stream input and output to correspond to a bytebuffer queue, these bytebuffer can be reused, after processing the data and then put back.

So here to see the Mediacodec class processing video can be exposed to video streaming data, here for example, if we have some special needs, such as video overlay technology, add subtitles and so on can be processed here. At the same time Mediacodec has a method: Createinputsurface can set the video source input surface type, but also can set the video output surface type by configure method.

2, Mediarecorder class

Mediarecorder This class relative to the mediacodec simple, because he encapsulated very good, directly is a few interfaces to complete video recording, such as video encoding format, video Preservation road, video source, etc., the use of simple, but there is a problem is not access to video streaming data, Can't handle the video data of the native. This is the biggest difference between him and Mediacodec, he can't finish the video overlay technology.

Attention:

About Mediarecorder This class, in fact, he and Android in a command is the opposite, that is: adb screenrecord. This class has one more place to pay attention to is that he has a method: Setvideosource, you can set the video source, the code follow-up article will be introduced, mainly on two sources: one is from the device cameras camera, one is from the surface, About surface This class is described later.

Attention:

Now the video encoding format is H264, the H264 format description is as follows:

H.264,mpeg-4,mpeg-2 such as these are compression algorithm, after all, bandwidth is limited, in order to obtain better image transmission and display effect, and constantly try to get rid of some information, conversion of some information, and so on, this is the compression algorithm to do things. The biggest advantage is the high data compression ratio, in the same image quality conditions, the compression ratio of H. MPEG-2 is more than twice times, is the MPEG-4 1.5~2 times. For example, if the original file size of 88GB, using MPEG-2 compression standard compression into 3.5GB, compression ratio of 25:1, and the compression standard of H. 102:1 compression to 879MB, from 88GB to 879mb,h.264 compression ratio to achieve an alarming amount of up to a total of up to a total value of up to a total of. Why is the compression ratio so high in H. Low bit rates play an important role, compared with MPEG-2 and MPEG-4 ASP compression technology, the compression technology will greatly save users ' download time and data traffic charges. It is particularly noteworthy that the high compression ratio of H + + also has the quality and smoothness of the image. Write so much, for instance, for example, Mobile TV, we receive the image signal is generally the format of the picture, the mobile device received, it needs to decode into the original YUV stream, and then converted to an RGB stream, a frame of the RGB data on the video memory to display the image. Although the transmission is fast, it is to increase the cost of decoding equipment, but overall it is certainly worth it. Now the graphics card on the PC has to be integrated with the hardware decoding, and it is said that Apple's newest product ipad also has this hard decoding. and YUV to RGB conversion, a lot of arm chips have.

Second, video data source

Here's the video data source is the video encoder needs to encode the video source, in the mobile device, we get to know two places to get the video, one is from the webcam camera, one is from the device screen (desktop).

There are two classes to be introduced here: one is the webcam camera, the other is Android5.0 's new screen recording class Mediaprojection and Virtualdisplay.

1, Camear class

Camera this class in the current live and beauty cameras and other apps is important, he is a mobile device to take video and image information An important channel, his usage is very simple, divided into front camera and rear camera, can set the direction, size and other parameters, most importantly, he also needs a preview interface, He generally previews there are two methods: Setpreviewdisplay and Setpreviewtexture, the first method is to set the Surfaceholder type, the second method is to set the type of surfacetexture about both types, I'll talk about it later. But there is a doubt that the data from the camera will be previewed, but what do we do with the camera's data? Here you need a callback interface for camera: Previewcallback, this interface has a callback method: Onpreviewframe (byte[] data ...), see this method we all know this is the video data captured by the camera every frame of data , we can get every frame of data here and then process it, like now the beauty camera, is here to get a frame of data, and then do the filter effect, and then produce a picture. Of course, it is also possible to record whitening videos here. So here we can get to the data source of the video.

Then the above Mediacodec class can use the Getinputbuffer class to get the video stream input queue, we can get the data in this callback method, and then passed into this queue, for encoding operations, but it is important to note that the data format needs to do a conversion, which is described later. The Mediarecorder class can also set the video source directly through the Setvideosource method.

2. Mediaprojection class and Virtualdisplay class

These two classes are mainly Android5.0 added an API, is specifically used to record device video, but in the process of using the authority to authorize, if not authorized or very dangerous, if malicious software in the background secretly recording device screen video, know what you did, it is very dangerous. You need to get the Virtualdiaplay class by mediaprojection This class, and you need to pass in an important parameter, the surface class that is recording the screen video preview.

So here we can connect with the video encoder above, Mediacodec has a Createinputsurface method to set the video source type of surface, And Mediarecoder This class is also available through the Setvideosource method to set the surface type of video input source, where you want to implement the device screen recording video can be done by the above two video encoding class and then save.

Third, video data format

We saw two kinds of video sources, one from the camera, one from the screen, but the two data sources have their own format, so we need to explain the data format and the conversion between them. Our usual contact is generally argb color space, a for transparency, RGB is a three-color, but in the process of video processing, especially when recording mobile device video has a color space: YUV

It is also a color space, why to appear YUV, mainly for two reasons, one is to make color signal compatible with black and white machine, another reason is to reduce the transmission of bandwidth. YUV, Y is the luminance, U and V is the color, in short, it is the RGB signal processing, according to the brightness of people more sensitive, increase the brightness of the signal, reduce the color of the signal, so as to "deceive" the eyes of the means to save space. There are many formats for YUV, but the 422 and 420 formats are common. In the general technical development, commonly used or YCBCR, which is a 420 format, also known as I420, note that the YV12 data arrangement is exactly the opposite.

Y,u,v there is a ratio between them, this ratio is not unique, for example, the number of y,u,v three components is 4:1:1. That is, a pair of Uvs is shared per four pixels. If it is a 30*40 frame, then there are 1200 y components, each with 300 U and 300 v components. There are a total of 1200*1.5 so many values.

1, N21/YV12

This format is generally the device's webcam camera data collected, which is what we said above Onpreviewframe (byte[] data ...) Each frame of data, in fact, is N21 or YV12 format, the specific format, can be set. So here, for example, we want to get a frame of data processing, we must remember the format of conversion, such as to save a picture here, then you need to convert NV21 to RGB format, or directly use the system class Yuvimage, produce a picture.

2, yuv420p (I420)/yuv420sp (N12)

YUV420 has a packaged format (Packed), along with a flat format (Planar), that is, Y, U, v are stored separately, each component occupies a single place, where Y is width*height, and U, v together account for half of Y, the format of each pixel accounted for 12 bits. According to the Order of U, V, 2 kinds of format, u before V is yuv420p, also called i420,v before u, called YV12 (yv means y followed by v,12 to represent 12bit). In addition, there is a half-plane format (Semi-planar), that is, Y occupies a single place, but then u, V and next to each other, according to the Order of U, V, there are 2, u before V is called NV12, in the domestic as many people call it YUV420SP format; v before u is called NV21. This format seems slightly more popular than NV16.

This format is generally recorded on the screen video source format, is the above Mediaprojection class, so we mentioned above one will record device screen video and then save, we need to convert the camera's N21/YV12 format to encoder recognition yuv420p/ The YUV420SP format.

IV. Video Preview screen

1, Surfaceview class

This class, we may use in the development of the application is very little, in the development of the game will use some, we in the development of the application to create special animation only need to inherit the view class, and then start drawing in OnDraw, this class can also do the drawing function, see the video source needs a preview function, Need a surface, in fact Surfaceview is the view+surface combination, surface is the image of the data layer, and view is just a presentation layer, in the middle there is a surfaceholder as a chain then, their relationship is as follows:

You can get a surface class by Surfaceholder's Getsurface method.

So here we are in the use of Surfaceview as a video preview interface, is actually to get to surface or surfaceholder can be, For example, the camera preview interface can be set by the camera's Setpreviewdisplay method surfaceholder type, screen recording interface can be passed through the Virtualdisplay class parameters of an input surface type.

2, Textureview class

Textureview is introduced in 4.0 (API level 14). It can project the content stream directly into the view and can be used to implement features such as Live Preview. Unlike Surfaceview, it does not create Windows individually in the WMS, but as a normal view in the view hierachy, so it can be moved, rotated, scaled, animated, and so on with other normal view. It is important to note that the Textureview must be in a hardware-accelerated window. The content stream data it displays can be from the app process or the remote process. As you can see from the class diagram, Textureview inherits from view, which is managed and drawn in view hierachy like any other view. Textureview overloads the Draw () method, which mainly updates the image data received in the surfacetexture as a texture to the corresponding hardwarelayer. Surfacetexture.onframeavailablelistener is used to inform the Textureview content stream that a new image has arrived. The Surfacetexturelistener interface is used to let Textureview users know that surfacetexture is ready so that surfacetexture can be handed to the appropriate content source. Surface is the producer interface implementation class for Bufferqueue, allowing the producer to provide graphic buffer for bufferqueue inside the surfacetexture through its software or hardware rendering interface.

This class is actually similar to Surfaceview, but he is not dependent on surface and surfacholder, but surfacetexture, about surfacetexture It has many advantages:

surfacetexture is a new class that was added from Android3.0 (API 11). This class is similar to Surfaceview, where you can get the image stream from camera preview or video decode. However, unlike Surfaceview, Surfacetexture does not need to be displayed after it receives the image stream. Anyone who has done Android camera development knows that a problem with headaches is that the preview image stream read from the camera must be output to a visible Surfaceview, A copy of the image frame data is then obtained through the Camera.previewcallback onpreviewframe (byte[] data, camera camera) function. There is a problem, such as wanting to hide the preview image of the camera or to do some processing of each frame and then display it on the phone display. Then there is no way to do before the Android3.0, or you need to use some small tricks, such as the use of other controls to block Surfaceview, note that the original camera image stream Surfaceview is still there, that is, the blocked Surfaceview is still connected Receive the image from the camera, and always follow a certain frame rate to refresh, which is CPU-consuming, and if some parameters are not properly set, the hidden Surfaceview may be exposed, so these tips are not a good way. However, with the surfacetexture, it is much better, because Surfacetexture does not need to be displayed on the screen, so we can use Surfacetexture to receive images from the camera stream, The image frame is then processed from the Surfacetexture, and then sent to another surfaceview for display.

And surfacetexture can easily get video timestamp data, do not need our manual to calculate, at the same time he has a powerful function is to combine with render, and render is to talk about the core of Glsurfaceview, is the OpenGL technology , for subsequent image and video filter processing, this plays a huge role, because it does not directly display the image stream processing, but instead of the GL external texture, so it can be used for image streaming data two processing (such as camera filters, desktop effects, etc.). such as camera preview data, become a texture can be handed to glsurfaceview direct display, or through surfacetexture to Textureview as a hardware accelerator layer in view heirachy display. First, surfacetexture obtains frame data from the image stream (from Camera preview, video decode, GL plot scene, etc.), and when called Updateteximage (), updates the surfacetexture corresponding GL texture object based on the most recent image in the content stream , for the camera data source can be set by the Setpreviewtexture method of the Surfacetexture type, recording the screen data source can be set without a portal.

3, Glsurfaceview class

Glsurfaceview started with Android 1.5 (API level 3) as a supplement to Surfaceview. It can be seen as a typical usage pattern of surfaceview. On the basis of Surfaceview, it joins the management of EGL and comes with a rendering thread. It also defines the render interface that the user needs to implement, providing the flexibility to change the specific render behavior with strategy pattern. As a Glsurfaceview client, you only need to set the implementation class of the renderer that implements the render function to Glsurfaceview.

One of the features of Glsurfaceview here is that it is not the system that helps us to draw the preview screen, but rather we need to get the data ourselves and render it ourselves, and there will be a separate GL thread to refresh the data.

Iv. Summary of the process

The above describes the approximate function of all classes and several video source data collection format, the following summarizes:

First, two encoders Mediacodec and Mediarecorder

Mediacodec can set the input surface type and configure method settings to output surface type by Createinputsurface method

Mediarecorder can set the video source through the Setvideosource method, two kinds: one is the camera, one is the recording screen

The difference between these two encoders is that MEDIACODEC can handle detailed video streaming information, but the Mediarecorder package is too good to handle.

Second, two kinds of video source camera and Mediaprojection

The camera data source provides a callback method in a callback interface: Onpreviewframe (byte[] data ...) You can get the data for each frame of the video

The screen data source class Virtualdiaplay provides a set entry type for entering surface type

Third, video source format and video encoding data format

Camera capture video data format is N21 and YV12, but the Encoder MEDIACODEC processing data format is y420p and Y420SP, so here need to do a data format conversion, also if you want to capture every frame of the camera image to do the processing, You also need to convert the N21 format into RGB format.

IV. Video Preview view

1 "Here is the main surfaceview type, his main and surfaceholder,surface associated, the camera provides the Setpreviewdisplay method to set the Surfaceholder type. The camera can preview the data through the Surfaceview, recording the screen virtualdisplay can provide a set surface portal, so the recording screen can also be surfaceview for data preview.

2 "There is the Textureview type, which corresponds to the Surfacetexture class, and the camera provides a Setpreviewtexture method to set the Surfacetexture type, But the virtualdisplay of the recording screen is not, so the camera can preview the data through Textureview, but the recording screen is not available.

3 "Finally is the Glsurfaceview type, he is inherited Surfaceview, on the basis of the addition of OpenGL technology, used to deal with video data and image data. But Glsurfaceview and the previous two preview view are different, he needs to get the data to render a preview, the approximate process is as follows:

The glsurfaceview->setrender->onsurfacecreated callback method constructs a Surfacetexture object and then sets it to the camera preview, Surfacetexture callback method onframeavailable to know that a frame of data is ready->requestrender notify render to draw data, Call Surfacetexture's Updateteximage method in the render callback method Ondrawframe to get a frame of data, and then start using GL for drawing previews.

v. Use of the scene

First scenario: Save video data from camera to local
The first approach:
Camera->setpreviewcallback->onpreviewframe-> get no frame data (N21), conversion data format for y420sp-> to Mediacodec-> Encoding generates H264 format for video streams saved to local
The second approach:
Set Mediarecorder video source for camera, but this process can not get the camera's no frame data, do not handle.

Second scenario: Record screen video data saved to local
mediaprojection->virtualdisplay-> Settings Input surface (Mediarecorder is obtained by Getsurface method, Here you need to set the Mediarecorder video source to surface format)

The third scenario: capturing data from the camera to do each frame data acquisition image
The first method: Use the Yuvimage class to turn the N21/YV12 format into a bitmap data
The second method: get each frame of data, convert the N21/YV12 data format into ARGB8888 format data, and then produce the picture
The third method: Call the Camera callback interface Picturecallback in the callback method Onpicturetaken (byte[] data, final camera camera) directly obtained data is the picture format data

Fourth scenario: Get a picture when recording the screen (screen function)
Use imagereader->getsurface-> to set the Surface->imagereader in the Virtualdisplay class as input to get an image image

Fifth scene: Filter effects on captured videos and images

We pass the camera callback method Onpreviewframe (byte[] data ...) To get every frame of data in the video stream, and then data processing with powerful OpenGL technology, similar to the beauty camera function

The sixth scene: scanning two-dimensional code technology analysis

Camera can be used to obtain each frame of data, and then the identification of two-dimensional code.

Vi. Summary

Here we introduce the general knowledge structure of the video stream, two data sources, two encoders, data formats, three preview view, but we see in the preview view of OpenGL technology, yes, this is what we need to introduce later, How to use OpenGL to do the video filter effect, so that each live is so white, followed by a detailed combination of cases to introduce each knowledge point.

More content: Click here

Follow the public number, the latest Android technology real-time push

---basic knowledge outline of live video technology in Android