"WEBRTC Audio preprocessing unit APM's overall compilation and use-Android"

Objective

Before writing the article "separate compilation using WEBRTC audio processing module-Android", I have been trying to compile the WEBRTC audio processing engine Voe the whole to use for his project, but limited to the poor technology, time is tight, so did not succeed. The AECM, AGC, NS, and VAD modules in the engine were compiled separately to make it work. Although it can achieve a certain effect, but always not ideal. 5 months later, Bill needed to optimize the previous project, so it was cruel to use the entire audio processing module ...

Body

However, this optimization is still not able to use the entire Voe, because Voe not only contains audio preprocessing, it integrates the audio encoding module, transmission module into the engine, and Bill's project needs to use the existing encoding, transport layer, so using the entire Voe to me is redundant and not operational. Despair, aside from Voe, Bill found the module--APM (Audio preprocessing module) next to the Voe level-a pure audio preprocessing unit that incorporates all of the previous modules.

Step 1-Download Google WebRTC source

Google WEBRTC Development Progress is still considerable, this article will be WEBRTC's latest trunk revision 5125 as an example to explain. Please use SVN to sync the following directories yourself (as for the sync method, please Google yourself):

http://webrtc.googlecode.com/svn/trunk/

Step 2-Extract the resources needed to compile APM

The overall compilation of APM requires the following resources under the WEBRTC source directory:

1) Common_audio Entire Directory

2) Modules directory (does not include the video section)

3) system_wrappers Entire Directory

4) Common_types.h in the WebRTC source root directory | Common.h | Typedefs.h three header files.

5) android-webrtc.mk file located in the WebRTC home directory.

Step 3-Compiling APM basic steps and some key points in eclipse

For the compilation and use of JNI in Eclipse, see the previous article, and don't repeat it here.

This section is only explained in accordance with your own JNI directory organizational structure and can be adapted to your needs.

The JNI organizational structure in Eclipse is as follows:

All folders and header files in the Step-2 are located under the WEBRTC subdirectory. The ANDROID-WEBRTC.MK is located in the JNI root directory.

Let's step through the decomposition:

Step 3.1

First we need to describe and set up the entire Android project, open the Application.mk file under the JNI root directory, and edit as follows:

123456789101112

# Copyright (c) Billhoo. All rights Reserved. # Copyright (c), the WebRTC project authors. All rights Reserved. # # use of this source code are governed by a Bsd-style license # that can is found in the LICENSE file in the root of the source # tree. An additional intellectual property rights grant can is found # in the file Patents.&nbs P All contributing project authors may # being found in the authors file in the root of the S Ource tree. app_stl: = gnustl_static app_cppflags: =-frtti-fexceptions app_abi: = Armeabi armeabi-v7a app_platform: = android-9

The official notes of App_stl are as follows:

1234567891011

APP_STL    By default, the NDK build system provides C++ headers for the minimal    C++ runtime library (/system/lib/libstdc++.so) provided by the Android    system.    However, the NDK comes with alternative C++ implementations that you can    use or link to in your own applications. Define APP_STL to select one of    them. Examples are:       APP_STL := stlport_static    --> static STLport library       APP_STL := stlport_shared    --> shared STLport library       APP_STL := system            --> default C++ runtime library    For more information on the subject, please read docs/CPLUSPLUS-SUPPORT.html

Because the NDK uses the minimal C + + runtime library to compile the project by default, the source code for STL containers such as Std::map cannot be compiled WebRTC. Therefore, we need to set our own C + + Runtime library gnustl_static for this project.

Step 3.2

To open and edit the Android.mk file under the JNI root directory, this file simply tells the NDK to call the Android.mk file in all subdirectories:

123456789

# Copyright (c) 2013 BillHoo. All Rights Reserved.# Copyright (c) 2012 The WebRTC project authors. All Rights Reserved.## Use of this source code is governed by a BSD-style license# that can be found in the LICENSE file in the root of the source# tree. An additional intellectual property rights grant can be found# in the file PATENTS.  All contributing project authors may# be found in the AUTHORS file in the root of the source tree.include $(call all-subdir-makefiles)

Step 3.3

Ready to work, you can start to compile the entire APM unit, first open the JNI/WEBRTC directory, the new Android.mk file is as follows:

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465

# Copyright (c) 2013 BillHoo. All Rights Reserved.# Copyright (c) 2012 The WebRTC project authors. All Rights Reserved.## Use of this source code is governed by a BSD-style license# that can be found in the LICENSE file in the root of the source# tree. An additional intellectual property rights grant can be found# in the file PATENTS.  All contributing project authors may# be found in the AUTHORS file in the root of the source tree.#MY_WEBRTC_ROOT_PATH := $(call my-dir)## voiceinclude $(MY_WEBRTC_ROOT_PATH)/common_audio/signal_processing/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/common_audio/vad/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/aec/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/aecm/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/agc/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/ns/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/utility/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/utility/source/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/modules/audio_device/Android.mkinclude $(MY_WEBRTC_ROOT_PATH)/system_wrappers/source/Android.mk## build .soLOCAL_PATH := $(call my-dir)include $(CLEAR_VARS)LOCAL_ARM_MODE := armLOCAL_MODULE := liblu_audio_preprocessingLOCAL_MODULE_TAGS := optionalLOCAL_WHOLE_STATIC_LIBRARIES := \    libwebrtc_spl \    libwebrtc_apm \    libwebrtc_apm_utility \    libwebrtc_vad \    libwebrtc_ns \    libwebrtc_agc \    libwebrtc_aec \    libwebrtc_aecm \    libwebrtc_system_wrappers \    libwebrtc_audio_device \    libwebrtc_utility## Add Neon libraries.ifeq ($(WEBRTC_BUILD_NEON_LIBS),true)LOCAL_WHOLE_STATIC_LIBRARIES += \    libwebrtc_aecm_neon \    libwebrtc_ns_neon \    libwebrtc_spl_neonendifLOCAL_STATIC_LIBRARIES := \    libprotobuf-cpp-2.3.0-liteLOCAL_SHARED_LIBRARIES := \    libcutils \    libdl \    libstlportLOCAL_PRELINK_MODULE := false##TODO(billhoo) find a properway to do this.LOCAL_LDLIBS += $(NDK_ROOT)/sources/cxx-stl/gnu-libstdc++/4.6/libs/armeabi/libgnustl_static.aLOCAL_LDLIBS += -lOpenSLESifndef NDK_ROOTinclude external/stlport/libstlport.mkendifinclude $(BUILD_SHARED_LIBRARY)

A few things to note:

1) At compile time if you are not prompted to find: /.. /.. /android.mk files and other errors, please check and correct your relative path.

2) the GNU Static Library link path on line 60th is for the NDK version r8d, if the reader version does not match, please find the path of the LIBGNUSTL_STATIC.A static library to replace it yourself.

3) This example does not intend to compile the WebRTC test project, use the Eclipse search file feature, locate the-dwebrtc_audioproc_debug_dump in the Android.mk file, and comment out.

Step 3.4

Everything is ready, we can start to compile APM, but in the compilation process will certainly have a lot of small problems (such as the relative path is not correct, can't find the symbol of such a function, etc.), these questions are left to the reader to Google, so solve, Bill will not repeat.

Step 4-Considerations for using APM in Android apps

After the above steps, the reader will be able to get libwebrtc_audio_preprocessing.so this dynamic link library. All we need to do is write our own JNI wrapper function to provide the APM interface to the Android application layer. The specifics of Bill's previous article have been described in detail. It is important to note that the order in which the Android class loads the two libraries is sensitive if the reader intends to refer to the compiled APM library in its own dynamic library.

Assuming that the reader encapsulates its own JNI interface as a separate library libmy_jni_wrapper.so, and that the library references libwebrtc_audio_preprocessing.so, the following sequence should be used when loading the two libraries:

12345

static{    // Ordering of loading these shared libraries is significant.    System.loadLibrary("webrtc_audio_preprocessing");    System.loadLibrary("my_jni_wrapper");}

If the sequential write is reversed, the runtime will get an exception that is not found in the Webrtc_audio_preprocessing library symbol.

Summarize

The whole compilation work now looks very simple, but requires a lot of patience and search, but the results are quite satisfactory, APM out of the effect than the previous use of the individual audio modules are much better. However, for the effects of jitter and other factors, APM will be powerless. Maybe Bill should take the time to look at Neteq, jitter buffer and other modules. How to use them, how to integrate into their own projects, then will know.

"WEBRTC Audio preprocessing unit APM's overall compilation and use-Android"