Android audio development (5): audio data encoding/Decoding

Android audio development (5): audio data encoding/Decoding

The previous four articles introduced the basic knowledge required for audio development, how to collect a frame of audio, how to play a frame of audio, how to store and Parse Files in wav format, we recommend that you read the audio data first. This article focuses on how to encode and decode the audio data of a frame.

1. Android official MediaCodec API

First, let's take a look at the Android audio codec API, called MediaCodec, which was introduced in Andorid 4.1 (API 16, therefore, it can only work on Android 4.1 and later mobile phones.

1.1 Introduction to MediaCodec

(1) provides an interface for accessing the underlying Android multimedia module, mainly the audio/video encoding/decoding interface.

(2) The Android underlying multimedia module adopts the OpenMax framework. The implementation of any Android underlying Codec Module must follow the OpenMax standard. Google provides a series of software codecs by default, including OMX. google. h264.encoder, OMX. google. h264.encoder, OMX. google. aac. encoder, OMX. google. aac. decoder and so on. However, the hardware codec function must be implemented by chip manufacturers according to the OpenMax framework standards. Therefore, the hardware codec implementation and performance of mobile phones of different chip models are generally different.

(3) The Android Application layer uses MediaCodec API to provide various audio/video coding/decoding functions. The parameter configuration determines which encoding/decoding algorithm is used and whether hardware encoding/decoding acceleration is used.

1.2 core principles of MediaCodec

I am not going to detail how to use each function of the MediaCodec API. You can view and learn the sample code in the resource link provided later.

Here I want to focus on the core working principles of MediaCodec, because only by figuring out this, you will understand why the interfaces provided by MediaCodec APIs look like this.

The basic process for MediaCodec is as follows:

-createEncoderByType/createDecoderByType-configure-start-while(1){-dequeueInputBuffer-queueInputBuffer-dequeueOutputBuffer-releaseOutputBuffer}-stop-release

We can see that the operation of the Buffer queue is one of its core parts. The Buffer Queue of MediaCodec is as follows:

The MediaCodec architecture uses two buffer queues to process data asynchronously. The Client and MediaCodec modules described below work in parallel (note: the Client here refers to "developers, API user "):

(1) The Client applies for empty buffer [dequeueInputBuffer] from the input buffer queue.

(2) The Client copies the data to be decoded to the empty buffer and then puts it into the input buffer queue [queueInputBuffer]

(3) The MediaCodec module extracts one frame of data from the input buffer queue for encoding and decoding.

(4) After the encoding/decoding process is complete, MediaCodec sets the original data buffer to empty and puts it back into the input buffer queue. The decoded data is put into the output buffer queue.

(5) The Client requests the buffer after codec from the output buffer queue [dequeueOutputBuffer]

(6) The Client renders and plays the buffer after encoding/decoding.

(7) after rendering/playback is complete, the Client puts the buffer back into the output buffer queue [releaseOutputBuffer]

In architecture, MediaCodec uses a "producer-consumer" mode based on the "Ring buffer". It designs two "Ring buffers" based on the idx serial number. Note, there are two, one on the input side and the other on the output side.

I once shared a Linux C code on Github, called: "rw_queue", is the simplified version of this ring buffer, you are interested to see, address: https://github.com/Jhuster/clib/tree/master/rw_queue

In the figure below, wp represents the "Write Pointer" and points to "empty buffer", while rp represents the "read Pointer ", pointing to "filled buffer ":

The "producer" and "consumer" actually share this Buffer Queue. The "producer" is responsible for extracting unused Buffer from the queue, filling in data, and then putting it back into the queue, the "consumer" is responsible for extracting and processing the Buffer after the data is filled. After the processing, the Buffer is marked as "null ", return to the queue for "producer" to continue filling in data.

On the input side, "Client" is the "producer" of the circular buffer, and "MediaoCodec module" is "consumer ".

On the output side, the "MediaoCodec module" is the "producer" of this circular buffer, while "Client" is changed to "consumer ".

This is the core working principle. In fact, it is not complicated. You will soon be able to understand the secrets.

1.3 reference resources

There are actually a lot of examples of MediaCodec code on the Internet. I will directly give some links that I personally think are good. Interested friends can study them.

(1) Android official document: MediaCodec

(2) Android MediaCodec stuff

(3) HWEncoderExperiments

(4) Some open-source player Android source code, such as VLC and ijkplayer

2. Third-party audio codec Library

Although the official MediaCodec API supports hardware codec acceleration, there are still many problems and limitations. On the one hand, it can only be used on Android 4.1 and later models. On the other hand, due to the wide variety of Android mobile phones, the changes made by the vendor to the underlying source code are different. As a result, in actual use of the MediaCodec API, there may be many pitfalls and compatibility issues. Therefore, we can also consider using a third-party codec library.

Here, I recommend several third-party audio codec libraries (which can be transplanted to the Android platform). You can go to the official website or the project homepage to learn more.

(1) opus codec Library

I like opus very much. At a low bit rate, opus wins the he aac, which once had obvious advantages. I used it to implement an Android LAN VoIP network phone application: "Feige phone ", good results.

Opus official website address: https://www.opus-codec.org

Note: currently, Android 5.0 supports the opus format. For details about the multimedia Formats Supported by Android, see Android developer guide: Supported Media Formats.

(2) Speex codec Library

Old audio processing library, not only codec, but also provides audio Noise Removal, Echo elimination, mute detection and other functions, official site address: http://www.speex.org

(3) ffmpeg

The famous ffmpeg must not be missed, Official Website: https://www.ffmpeg.org

(4) Android AAC Encoder

A lightweight Android aac Library: https://github.com/timsu/android-aac-enc

(5) opencore-amr-android

Amr codec library extracted from opencore, address: https://github.com/kevinho/opencore-amr-android

(6) iLBC-Android

ILBC is the famous WebRTC project Audio Codec Module, iLBC-Android is from the extraction iLBC module transplanted to the Android platform personal project, address: https://github.com/lukeweber/iLBC-Android

I will briefly introduce the third-party codec library. The last three are personal projects that I have never used. I sincerely thank these authors for their selfless dedication. In addition, for more third-party libraries, you are welcome to leave a message or send a letter to supplement them.