Android Audio Development (5): Codec for audio data

The first four articles, respectively, introduced the basic knowledge of audio development, how to capture a frame of audio, how to play a frame of audio, how to store and parse WAV file, it is suggested that interested small partners to read, this article focuses on how to encode and decode a frame of audio data.

1. Android's official Mediacodec API

First, let's look at the official Android audio codec API, the Mediacodec class, which was introduced in the Andorid 4.1 (API 16) version and therefore works only on Android 4.1 + phones.

1.1 Mediacodec Basic Introduction

(1) provides a set of access to the Android underlying multimedia module interface, mainly audio and video codec interface

(2) The Android underlying multimedia module uses the OpenMAX framework, and any Android underlying codec module implementation must follow the OpenMAX standard. Google official default provides a series of software codecs: including: Omx.google.h264.encoder,omx.google.h264.encoder, OMX.google.aac.encoder, OMX.google.aac.decoder, and so on, and hardware codec function, it needs to be done by the chip manufacturers in accordance with the OpenMAX framework standard, so, the general use of different chip models of mobile phones, hardware codec implementation and performance is different

(3) The Android application layer is unified by the Mediacodec API to provide a variety of audio and video codec functions, by parameter configuration to determine which codec algorithm, whether to use hardware codec acceleration and so on

1.2 Mediacodec Core Principle

I am not going to detail how each function of the Mediacodec API is used, and the sample code can be viewed and learned in the resource links given later in this article.

Here I'm going to focus on the core workings of Mediacodec, because only by figuring this out, you'll understand why the interface provided by the Mediacodec API looks like this.

The basic processes used by MEDIACODEC are:

-Createencoderbytype/createdecoderbytype-configure-start-while (1) {-Dequeueinputbuffer-queueinputbuffer- dequeueoutputbuffer-releaseoutputbuffer}-Stop-release

As you can see, the operation of the buffer queue is one of its most central parts, the buffer queue for Mediacodec, as follows:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/7E/7B/wKioL1cCSojSz4HdAAD0xxQxvwg230.png "title=" Image.png "style=" float:left; "alt=" Wkiol1ccsojsz4hdaad0xxqxvwg230.png "/>

The MEDIACODEC architecture employs 2 buffer queues to process data asynchronously, and the client and Mediacodec modules described below are working in parallel (note: The client here means "developer, user of the API"):

(1) Client request from input buffer queue empty buffer [Dequeueinputbuffer]

(2) The Client copies the data that needs to be decoded to empty buffer and then puts it into the input buffers queue [Queueinputbuffer]

(3) Mediacodec module from input buffer queue to take a frame of data for codec processing

(4) After the codec processing is finished, mediacodec the original data buffer to empty and put back into the input buffer queue, putting the encoded data into the output buffer queue

(5) Client requests the codec buffer from the output buffer queue [Dequeueoutputbuffer]

(6) The Client renders/plays the decoded buffer

(7) After rendering/playback is complete, the Client then puts the buffer back into the output buffer queue [Releaseoutputbuffer]

Mediacodec in architecture, in fact, a "ring buffer" based on the "producer-consumer" mode, it designed 2 based on the IDX serial number of "ring buffer" , note that is 2, one at the input side, one at the output end.

I once shared a Linux C code on Github, named: "Rw_queue", this is a simplified version of the ring buffer, we are interested to see, address: https://github.com/Jhuster/clib/tree/master/ Rw_queue

The total of the IDX-based ring buffers is as follows, in the figure, WP stands for "write pointer", pointing to "empty buffer", and the RP stands for "read pointer", pointing to "filled buffer":

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/7E/7B/wKioL1cCStbQJjLDAAB7AoEE3KU663.png "title=" Rw_ Queue.png "alt=" Wkiol1ccstbqjjldaab7aoee3ku663.png "/>

"Producers" and "consumers" are actually sharing this buffer queue, the "producer" is responsible for removing unused buffer from the queue, filling in the data, and then put back into the queue, "consumer" is responsible for removing the fill buffer after the data, processing, processing, after the end of the buffer marked "empty", Go back to the queue for the "producer" to continue populating the data.

On the input side, "Client" is the ring buffer "producer", "Mediaocodec module" is "consumer".

On the output side, the "Mediaocodec module" is the ring buffer "producer", and "Client" becomes "consumer".

This is the core of the principle of work, in fact, not complicated, we calm down, we will soon be able to understand the mystery.

1.3 Reference Resources

About the example code of MEDIACODEC, online actually also many, I directly give some personal feel good link, interested small partners can go to study.

(1) Android Official document: "Mediacodec"

(2) "Android Mediacodec Stuff"

(3) "Hwencoderexperiments"

(4) Some open source player Android source code, such as VLC, Ijkplayer

2. Third-party audio codec library

Although the official MEDIACODEC API supports hardware codec acceleration, but the problem and limitations are many, on the one hand, only in Android 4.1 or more models can be used, on the other hand, due to a wide range of Android phones, manufacturers of the underlying source changes are different, resulting Mediacodec API in the actual use, will encounter many pits, there are many compatibility issues, so we can also consider the use of third-party codec library.

Here, I simply recommend a few third-party audio codec libraries (can be ported to the Android platform), you can go directly to the official website or the project page to learn more about the details.

(1) Opus Codec library

Like opus, the low bit rate opus triumph had the obvious advantage of he AAC, I used it to achieve an Android LAN VoIP network Telephony Application: "Flying Pigeon Phone", the effect is very good.

Opus Official website address: https://www.opus-codec.org

Note: Android 5.0 is now officially supported in opus format, and the list of multimedia formats supported by Android can be viewed on Android developer Guide: "Supported Media Formats"

(2) Speex codec library

The old audio processing library is not only codec, but also includes audio denoising, echo cancellation, mute detection and other functions, official website address: http://www.speex.org

(3) FFmpeg

The famous ffmpeg must not be missed, official website: https://www.ffmpeg.org

(4) Android AAC Encoder

A lightweight Android AAC encoding library: Https://github.com/timsu/android-aac-enc

(5) Opencore-amr-android

AMR codec library extracted from Opencore, address: https://github.com/kevinho/opencore-amr-android

(6) Ilbc-android

ILBC is the famous audio codec module for the WebRTC project, Ilbc-android is a personal project from which ILBC modules are ported to the Android platform, address: https://github.com/lukeweber/iLBC-Android

About the third-party codec library is briefly introduced here, the last three is a personal project, I have not used, sincerely thank these authors for their selfless dedication, in addition, more third-party library welcome message or letter to add.

3. Summary

about how to encode and decode audio data on the Android platform simply introduced here, the article is not clear about the place to welcome messages or letters [email protected] exchange, or follow my Sina Weibo @ Lu _ June or the public number @Jhuster to obtain the latest articles and information.

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/7E/7B/wKioL1cCUUuAWhLvAACb8XAe6Uo758.jpg "title=" Weixin _jhuster.jpg "alt=" Wkiol1ccuuuawhlvaacb8xae6uo758.jpg "/>

This article is from the "jhuster column" blog, be sure to keep this source http://ticktick.blog.51cto.com/823160/1760191

Android Audio Development (5): Codec for audio data