ORB_SLAM2 porting process on Android

ORB_SLAM2 porting process on Android

I have never had time to write a blog. I recently took some time to write some experience on ORB_SLAM2's porting process on Android.

After writing the script, I clicked a link. In an instant, 1/3 of the workload was gone, and I was drunk when I finished it late at night...

Start of Text

This blog explains how to port ORB_SLAM2 on the Android platform. It describes how to set up the basic Android environment, how to configure the NDK environment, how to port the Android environment, and how to port ORB.

Android platform construction and NDK environment Configuration

System: Windows 7 32bit
IDE: Eclipse Luna
Environment tools: ADT24.0.2, Android SDK, NDK r10b

PS: it is not recommended to use the Eclipse version integrated with the ADT environment, because various inexplicable errors may be reported during NDK compilation. The JDK environment is installed by default.

Components: Baidu Network Disk (including ADT and NDK. The SDK is too large. Please download it yourself)

Steps:

Install Eclipse. Ensure that Eclipse is 32-bit or 64-bit. Download the corresponding version for installation. As shown above. Download ADT and do not need to decompress it. Open the installed Eclipse and click Help> Install new software on the menu bar. The following page is displayed:
Click Add in the upper right corner:

Enter a name in the first column, such as ADT. Click Archive in the second column, select the ADT package you downloaded, and click OK. In the list that appears, select all and deselect the "Contact all update sites…" check box ..... Click next to start the installation of the ADT, which takes about 10 minutes. You may also need the accept protocol or something, which is skipped here. After the installation is complete, the system prompts that Eclipse is successfully installed after being restarted. Download the Android SDK and decompress it (the path must not contain Chinese characters ). In Eclipse, select Window> preferences. Select Android on the Left bar, enter the root directory of the decompressed folder in SDK Location, and click apply and OK. The Android environment configuration is complete. This is what you should have on the menu bar.
Update the SDK. Click Window> Android SDK manager. The following page is displayed:

A list may not be loaded due to a wall problem. You need to use a proxy or image. Of course, if you have a ladder, you can. Here we will introduce some images:
G.cn: 80. In SDK Manager, Click tools> options:

Follow the settings. Other image sources include the image station of Beijing Chemical University. Click Packages-> reload to get the corresponding image. Some options are selected by default. Generally, you can select an sdk tool, an sdk build-platform-tools, an sdk build tools, and an Android version (generally more than 4.0 ). Click Install to download the corresponding SDK from the image. The process is long and short, depending on the network.
After the preceding steps are completed, the Android development environment is configured. NDK environment configuration: Download The ndk from the Baidu Network Disk above, decompress it (the rule is the same as the sdk), open Eclipse, click Window-> Preferences, select Android-> NDK, enter NDK Location in the right interface.
Android porting Basics

NDK is an integrated toolkit for Calling C ++ code in Android. The core is the JNI (Java Native Interface) technology. It is skipped here. Let's talk about the basic steps of NDK development:
1. Write Java code: Define a class in Java, for example, NDKHelper, which defines several java methods and only needs to be declared without implementation, as shown below:

Public class NDKHelper {// NDK example method 1 public static native void ndkOne (int a, long B); // NDK example method 2 public static native int ndkTwo (String, string B );}

Native identifier indicates that the function will be implemented using C ++ code.
Right-click the project and choose Android Tools> Add native support. The following page is displayed:

The name is the name of the library to be generated. You can enter the name as needed and modify it. Click "OK" to add C ++ compilation support to your project. A hammer is added to the menu bar:

This is the shortcut key used to compile C ++. The jni directory and obj directory will be created under your project directory. The jni directory is used to store things related to C ++ code, obj stores the middleware generated during C ++ compilation. The generated library is written to the libs folder.
The following file is generated in the jni Folder:. Cpp, an Android. mk, where. Cpp is automatically generated and used to write C ++, while Android. mk is similar to CMakeList in C ++. It is used to specify the file to be compiled and the name of the module generated by compilation. It is the simplest Android. the mk file is as follows:

LOCAL_PATH := $(call my-dir)include $(CLEAR_VARS)LOCAL_MODULE    := NDKTestLOCAL_SRC_FILES := NDKTest.cppinclude $(BUILD_SHARED_LIBRARY)

LOCAL_PATH: = $ (call my-dir) indicates that the current directory is included.
Include $ (CLEAR_VARS) indicates clearing all non-system variables and some system variables;
LOCAL_MODULE: = NDKTest indicates the name of the currently generated module. The libNDKTest. so file will be generated eventually.
LOCAL_SRC_FILES: = NDKTest. cpp indicates the cpp file to be compiled;
Include $ (BUILD_SHARED_LIBRARY) indicates to generate a shared library. To generate a static library, change it to BUILD_STATIC_LIBRARY.

Other basic commands:
Local_c_shortdes: = indicates adding a header file to the compiling environment.
LOCAL_LDLIBS: = indicates adding the system static library
LOCAL_SHARED_LIBRARIES: = indicates adding a shared library.
For other commands, see the API documentation.

The compilation conditions are specified here. If you need to specify the compiler version and the compilation target platform, you need to create the Application. mk file in the jni directory. The basic statements are as follows:

APP_STL := gnustl_staticAPP_CPPFLAGS := -frtti -fexceptions NDK_TOOLCHAIN_VERSION := 4.8APP_ABI :=armeabi-v7a

APP_STL: = indicates that stl library is used, APP_CPPFLAGS indicates some CPP compilation parameters, NDK_TOOLCHAIN_VERSION indicates the compiler version used by NDK, APP_ABI indicates the compiling target platform, and multiple platforms can be specified, platforms are separated by spaces, or all is specified for full-platform compilation (armeabi, armeabi-v7a, mips, x86 ). For other commands, check the API.

Next, write the corresponding C ++ file.
Open eclipse, click Project-> build Project (if build automatically is selected, it will be automatically compiled), open the command line, cd to the bin-> classes folder in your Project folder, run the following command:

javah com.example.ndktest.NDKHelper

Press enter to generate the corresponding header file in your classes folder. HERE com. example. ndktest is your package name, And NDKHelper is the class name of your NDK function.
The generated header file is as follows:

/* DO NOT EDIT THIS FILE - it is machine generated */#include 
  
   /* Header for class com_example_ndktest_NDKHelper */#ifndef _Included_com_example_ndktest_NDKHelper#define _Included_com_example_ndktest_NDKHelper#ifdef __cplusplusextern "C" {#endif/* * Class:     com_example_ndktest_NDKHelper * Method:    ndkOne * Signature: (IJ)V */JNIEXPORT void JNICALL Java_com_example_ndktest_NDKHelper_ndkOne  (JNIEnv *, jclass, jint, jlong);/* * Class:     com_example_ndktest_NDKHelper * Method:    ndkTwo * Signature: (Ljava/lang/String;Ljava/lang/String;)I */JNIEXPORT jint JNICALL Java_com_example_ndktest_NDKHelper_ndkTwo  (JNIEnv *, jclass, jstring, jstring);#ifdef __cplusplus}#endif#endif
  

For other functions, we need to pay attention to the two function declarations in the middle:

JNIEXPORT void JNICALL Java_com_example_ndktest_NDKHelper_ndkOne(JNIEnv *, jclass, jint, jlong);

This function is the C ++ version corresponding to the ndkOne function in the NDKHelper class, where JNIEXPORT and JNICALL are fixed fields, and void is the function return value, the function name is composed of Java field + package name + class name + function name. The parameters include the JNI system parameters JNIEnv and jclass. The others are the corresponding parameters in the NDKHelper class, ndk will parse and link the function to implement the connection between java and C ++.
Copy the generated. h header file to the jni directory, create the corresponding cpp file, include the header file, and implement the corresponding function. The implementation process depends on the function.
After completing these steps, you need to modify your Android. mk file and include the newly created cpp and H files.
Then, click the starting hammer or right-click the project and choose RunAs> Android Application. The C ++ part starts compilation, the compilation process can be seen in the Console Window below Eclipse (if there is no Console Window, click Window-> Show Views, and select Console OK ).
After compilation, the corresponding inventory will be generated and placed in the libs directory. You can start to call the ndkOne and ndkTwo functions just defined in Java to implement specific functions.

This is the basis for NDK. For more details, download the ndk samples officially provided by Android.

ORB_SLAM2 porting

Do not want to know the transplant process of shoes can directly download my Github source code: https://github.com/FangGet/ORB_SLAM2_Android directly follow the steps to proceed.
Porting process
First look at the directory:

It can be divided into ORB and ThirdParty, among which ThirdParty includes boost clapack DBow2 g2o eigen3.
Clapack and eigen come from a github open source Library: https://github.com/simonlynen/android_libs here integrates some typical C ++ library ndk version, download to use. G2o and DBoW2 are from the github address of ORB_SLAM2 Original Author. Boost is the self-compiled lib. Here we only introduce the library configuration of clapack and opencv.
Clapack Configuration
Copy the clapack directory from the aforementioned open-source Library to the corresponding directory of Thirdparty. clapack already contains the compilation process of the extremely sub-directories in the current directory. We have Android in the jni directory. add the following content to the mk file:

include $(CLEAR_VARS)MAINDIR:= $(LOCAL_PATH)include $(MAINDIR)/Thirdparty/clapack/Android.mkLOCAL_PATH := $(MAINDIR)include $(CLEAR_VARS)MAINDIR:= $(LOCAL_PATH)LOCAL_MODULE:= lapackLOCAL_SHORT_COMMANDS := trueLOCAL_STATIC_LIBRARIES := tmglib clapack blas f2cLOCAL_EXPORT_C_INCLUDES := $(LOCAL_C_INCLUDES)LOCAL_PATH := $(MAINDIR)include $(BUILD_SHARED_LIBRARY)

The basic commands here have been mentioned before, and only the following content is added:

LOCAL_SHORT_COMMANDS is a parameter set to Prevent Windows from limiting the length of G ++ compilation commands. This parameter will slow down the entire compilation process, so please use it with caution; LOCAL_EXPORT_C_INCLUDES indicates that the header file of the current library is EXPORT to the system, so that the calling Process <> can be implemented in the program code. If this parameter is not set, the library may not be referenced in the cpp file; LOCAL_STATIC_LIBRARIES: = tmglib clapack blas f2c refers to some dependent modules compiled in the lapack subdirectory.
Here we will compile a library project named lapack, which can be referenced by ORB as a dependency.
OpenCV Compilation
Opencv4Android is a tool set released on the opencv official website to support Android. It can be downloaded from the opencv official website. The directory structure is as follows:

The sdk is the core part? Http://www.bkjia.com/kf/ware/vc/ "target =" _ blank "class =" keylink "> drawing/a1xLG + drawing/drawing + 7Ev7XEzay8tsS/paper" here write picture description "src =" http://www.bkjia.com/uploads/allimg/160417/041415Nb-11.png "title = ""\"/>
Then add the following code to the Android. mk file in the jni directory to reference OpenCV:

OPENCV_LIB_TYPE:=STATICifeq ("$(wildcard $(OPENCV_MK_PATH))","")  #try to load OpenCV.mk from default install location  include E:/ORB_SLAM2/OpenCV-2.4.9-android-sdk/sdk/native/jni/OpenCV.mkelse  include $(OPENCV_MK_PATH)  endif 

Here, opencv. mk is an absolute address. In fact, the relative address is also acceptable. The above reference will compile opencv and introduce it to the current working module. The basic call of the opencv library is completed here. For convenience, you can also compile opencv itself into a library project and open it to other modules for reference.

The compilation process of other libraries is similar to that of the above project. The main steps are as follows:

Copy the current library to a specific jni directory. create a new module in mk and name the module. The header file LOCAL_C_INCLUDE is imported into the cpp file LOCAL_SRC_FILES is imported into the library. LOCAL_LDLIBS/LOCAL_SHARED_LIBRARIES/LOCAL_STATIC_LIBRARIES introduces the dependent library;

ORB_SLAM2 Compilation
Here we will compile the ORB_SLAM2 source file into a library for calling. The compilation process is the same as above. Note that due to pangolin compilation problems, I split the pangolin part of the source file, commented out the corresponding part of the code, and introduced opengl es for map and pose painting. At the same time, in order to complete the callback of the feature detection image, I changed the returned value of TrackMonocular in System. cc to Mat.
After the above process is completed, our C ++ compilation is basically completed. The last and most important step is to implement C ++ for the native method defined in Java, in JAVA, I defined the following native functions:

/*** Initialize the SLAM System in jni * @ param VOCPath * @ param calibrationPath */public static native void initSystemWithParameters (String VOCPath, String calibrationPath ); /*** start function of ORB System in Dataset mode * @ param curTimeStamp * @ param data * @ param w * @ param h * @ return */public static native int [] startCurrentORB (double curTimeStamp, int [] data, int w, int h ); /*** start * @ param curTimeStamp * @ param addr * @ param w * @ param h * @ return */public native static int [] startCurrentORBForCamera (double curTimeStamp, long addr, int w, int h);/*** Opengl es initialization */public native static void glesInit (); /*** opengl es plotting update */public native static void glesRender (); /*** prevents the effect of opengl es window resize * @ param width * @ param height */public native static void glesResize (int width, int height );

The corresponding C ++ code is:

/** Class: required * Method: initSystemWithParameters * Signature: (Ljava/lang/String;) V */JNIEXPORT void JNICALL encode (JNIEnv * env, jclass cls, jstring VOCPath, jstring calibrationPath) {const char * calChar = env-> values (calibrationPath, JNI_FALSE); const char * vocChar = env-> values (VOCPath, JNI_FALSE ); // use your string std: string voc_string (vocChar); std: string cal_string (calChar); env-> GetJavaVM (& jvm); jvm-> AttachCurrentThread (& env, NULL); s = new ORB_SLAM2: System (voc_string, cal_string, ORB_SLAM2: System: MONOCULAR, true); env-> ReleaseStringUTFChars (calibrationPath, calChar ); env-> ReleaseStringUTFChars (VOCPath, vocChar); init_end = true;}/** Class: Using * Method: startCurrentORB * Signature: (DDD [I) [I */JNIEXPORT jintArray JNICALL encode (JNIEnv * env, jclass cls, jdouble curTimeStamp, jintArray buf, jint w, jint h) {jint * cbuf; cbuf = env-> GetIntArrayElements (buf, false); if (cbuf = NULL) {return 0;} int size = w * h; cv: Mat myimg (h, w, CV_8UC4, (unsigned char *) cbuf); cv: Mat ima = s-> TrackMonocular (myimg, curTimeStamp); jintArray resultArray = env-> NewIntArray (ima. rows * ima. cols); jint * resultPtr; resultPtr = env-> GetIntArrayElements (resultArray, false); for (int I = 0; I <ima. rows; I ++) for (int j = 0; j <ima. cols; j ++) {int R = ima. at <Vec3b> (I, j) [0]; int G = ima. at <Vec3b> (I, j) [1]; int B = ima. at <Vec3b> (I, j) [2]; resultPtr [I * ima. cols + j] = 0xff000000 + (R <16) + (G <8) + B;} env-> ReleaseIntArrayElements (resultArray, resultPtr, 0 ); env-> ReleaseIntArrayElements (buf, cbuf, 0); return resultArray;}/** Class: orb_slam2_android_nativefunc_OrbNdkHelper * Method: glesInit * Signature :() V */JNIEXPORT void JNICALL mask (JNIEnv * env, jclass cls) {// enable glShadeModel (GL_SMOOTH) for shadow smoothing; // glClearColor (1.0f, 1.0f, 1.0f, 1.0f, 0.0f); // set the deep cache glClearDepthf (1.0f); // enable the deep test GL_DEPTH_TEST); // glDepthFunc (GL_LEQUAL) Type of the deep test ); // notify the system to modify glHint (GL_PERSPECTIVE_CORRECTION_HINT, GL_NICEST) for perspective;}/** Class: callback * Method: glesRender * Signature :() V */JNIEXPORT void JNICALL invoke (JNIEnv * env, jclass cls) {glClear (GL_COLOR_BUFFER_BIT | callback); glMatrixMode (GL_MODELVIEW); glLoadIdentity (); if (init_end) s-> drawGL ();}/** Class: Drawing * Method: glesResize * Signature: (II) V */JNIEXPORT void JNICALL evaluate (JNIEnv * env, jclass cls, jint width, jint height) {// position, length, and width of the final display image to the screen glViewport (, width, height); // specify the glMatrixMode (GL_PROJECTION) matrix ); // set the current matrix to the glLoadIdentity () matrix specified by glMatrixMode; glOrthof (-2, 2,-2, 2,-2, 2);}/** Class: using * Method: readShaderFile * Signature: (Landroid/content/res/AssetManager;) V */JNIEXPORT jintArray JNICALL labels (JNIEnv * env, jclass cls, jdouble timestamp, jlong addr, jint w, jint h) {const cv: Mat * im = (cv: Mat *) addr; cv: Mat ima = s-> TrackMonocular (* im, timestamp); jintArray resultArray = env-> NewIntArray (ima. rows * ima. cols); jint * resultPtr; resultPtr = env-> GetIntArrayElements (resultArray, false); for (int I = 0; I <ima. rows; I ++) for (int j = 0; j <ima. cols; j ++) {int R = ima. at <Vec3b> (I, j) [0]; int G = ima. at <Vec3b> (I, j) [1]; int B = ima. at <Vec3b> (I, j) [2]; resultPtr [I * ima. cols + j] = 0xff000000 + (R <16) + (G <8) + B;} env-> ReleaseIntArrayElements (resultArray, resultPtr, 0); return resultArray ;}

The difference between the start method in Dataset and Camera mode is explained here. In fact, image parameters are transmitted in different ways. In DataSet mode, we use ImageView to display images and Bitmap to read images in files, data of non-basic types cannot be accepted by the jni interface. Therefore, we need to use the getPixels method of Bitmap to convert it into intp [type data for transmission, in jni, the data type corresponding to int [] is jintArray. After obtaining the data, we can convert jintArray to Mat for subsequent processing; in Camera mode, we use cvCameraView in opencv android sdk to directly call cameras and display images. Inputfram in its onCameraFrame (CvCameraViewFrame inputFrame) can be converted to Mat-type data through the rgba () method, while Mat-type data is not recognized by jni, therefore, we need to use the getNativeObjAddr method of Mat to obtain the long pointer of Mat data and pass it to jni for processing.

End

After the above steps are completed, we will get the final generated sdk. The layout file and the corresponding activity file of Android are omitted here. After obtaining the final generated apk, if we want to test the Camera mode, we need to first install the corresponding types of opencv manager in the apk folder in opencv4Android to the mobile phone and open it in advance before using it, otherwise, the system will prompt that the support library of opencv cannot be found. If you only need to test the Dataset mode, the above steps are not required.

The level is limited. If any error occurs, please do not correct it. Thank you.