Full stack scheme of speech recognition project based on react-native0.55.4

The API ability verification scheme of mobile side is different from PC side! Not the same!! Not the same!!!

Even if the need to useAPIall exist, not necessarily can be used, this and PC side is very different, the domestic mobile phone system although are based onAndroid, but almost will go through the major manufacturers of customization, the function and the originalAndroidsystem is not exactly the same, in the investigation of technical solutions must be confirmed with Justdemorun the function and don't ask me how to know.

A. Mobile direct access to Web apps?

PC-basedWeb APIspeech recognition program can refer to the "recorder.js+ Baidu speech recognition" Full stack scheme technical details of the article.

1.Web APIThe calling multimedia acquisition interface requires a specific domain

Web APIMultimedia interface is the implementation of WebRTC technology in the PC side, because the multimedia acquisition involves user privacy, so in the browser-side call this interface needs to be in a secure domain to be transferred, secure domain refers to the following three categories:

• file:///Local area
• http://localhostLocal Web server
• https://Security Domain

The first two categories are generally used for desktop applications and local debugging, the actual site on-line deployment needs to be deployed in ahttpsway, how to deployhttpsand apply for free CA certificate Online There are many articles to explain, this article will not repeat.

2. The mobile browser almost does not support the WebRTCinterface directly

After the PC-side Web application is deployed in ahttpsway that does not evoke the recording interface authorization when accessed directly from the mobile browser, thenavigator.getUserMedia( )method returns anpermissionDeniederror, either by editing the Android6.0 manifest.xmlor by adding or Android6.0 above through the dynamic access to obtainRECORD_AUDIOaccess to the site can be normal access, the relevantWeb APIinterface is also present, but even if the user authorization can not be adjusted to the recording function. I tested the UC browser , Baidu Mobile browser and Android6.0 (API23) comes with the browser ,Android8.0 (API26) comes with the browser , the result is not supported.

Two. Programme research and new pits

O (￣▽￣) d since the mobile side of the direct access to the Web application can not be transferred to the recording interface, at least not compatible with many systems and models, if not consider the direct native development of Android, only hope that the hybrid solution.

2.1 WebView

• Solution :

  Place a WebView component in a single page full screen in one app, and then load a Web app that is deployed in HTTPS mode.

• reason :

  If the mobile browser cannot support it, it can only be sent in the hopeWebView.WebViewis the android underlying component for loading Web pages, the Android4.4 version has replaced the built-in browser engine with chromium, the chrome kernel, from Can I use The support of the query above Android5.0 is theWebViewWebRTCmethod that supports the interfacegetUserMedia( ).

• Test Results :

  The application compilation target version isAPI23, in the supportAPI23(Android6.0) of the virtual machine and the real machine test, can not be transferred through theWebAPIinterface microphone for recording. In a supportedAPI26(Android8.0) version of a virtual machine, functionality is achievable. Finally, in the remarks on the statistical information of the method support in Can I Use, it is found that thegetUserMedia( )known issue is stated in the following:

Simply put, this method isAndroid webview,iOSand Basic,PWAnot used. It is suggested that some of the less commonly used APIs may be used in future development to take a complete look at the relevant information.

• Conclusion:

  Android8.0 Support, Android support is poor, not recommended.

2.2 Crosswalk

• Solution :

  Official website: https://crosswalk-project.org/

  Withcrosswalk, when the app is packaged,webviewreplace the kernelxwalkwith the chromium-based browser kernel developed by crosswalk to extend nativewebviewcapabilities.

• reason :

  Since nativewebviewfunctionality is castrated, this small black technology can be used to package a more powerful browser kernel with its own application, the author 3 years ago in thecordova2.0-3.0 version of the popular era of the use of this technology, the benefits are indeed can expandwebviewthe ability undoubtedly, The downside is that the app project will directly increase the volume of the 80-90MB, of course, through several versions of the iteration, now crosswalk can generate a different package for the phone kernel type, app volume increment of about 20Mb, basically belong to the acceptable range.

• Test Results :

  Unfortunately this project was discontinued a year ago, and the final version of the official scaffolding tool is also unable to initialize the new project, the indirect use of the way divided into two kinds, first, downloadcrosswalkthe package, manually in theandroidproject to replace the nativeWebView, It is difficult for hybrid developers and is not controllable with hybrid technology compatibility; another scenario is described in the next section.

• Conclusion:

  Not recommended, it's better to look at a reliable hybrid solution.

2.3 Cordova/ionic

• Solution :

• Official website: https://cordova.apache.org

  codovais a popular hybrid solution that has now been upgraded to version 8.0.0, which is itself a solution for packaging web apps as apps.cordovaThe basic principle is toUIput the general layer of operations and functions in theWebViewimplementation, need to call mobile device hardware or native interface, are implemented by addingcordovaplug-ins, eachcordovaversion will be across the support of severalAndroidversions, such as thecordova7.0.0new The official documentation is supported by Android from 4.4 to 8.1 versions, which I think is ideal for small hybrid development teams to use.

• reason :

• It is worth mentioning thatcordovahas a very popular mobile development xxxionic, has now been iterative to the4.0stage, the author of this technology has a special feeling, the same yearionicin thealphaversion, the author is in use, it is based on Cordova+angular This technology portfolio, with a fresh and highly-designed UI component, is worth trying. In addition, is the plug-incordovacrosswalk, can be directly in the form of plug-in, in thecordovaproject packagingcrosswalk, the relevant needs of readers can take a try, especially when the team does not have the Android developer and no special designers,ionicThe application of the production will certainly make others treat you differently.

• Test Results :

• I have been in the usecordovaof 3.3 when the integrationcrosswalk, but also throughcordovathe plug-in successfully called the bottomGPS,摄像头and some other native components, was to adapt to the Android4.4 version.cordova7.0.0scaffolding tested in the domestic can be used, the new project either by the command line or import into the Android studio to develop can be packaged as a corresponding project, the official document has a very detailed call to various low-level interface instructions, the Internet also hascordova7.0.0+crosswalkTechnical stickers for the solution.

  As a result of the technical protocol specified in the technical stack for the sake of not halfway to replace the solution, so this time has not been tested.

• Conclusion:

• Consider trying as a total solution.

2.4 react-native

• Solution :

• Official website: https://reactnative.cn

  This is the author of this use of the program, because the Web-basedReacttechnology stack to complete, in order not to increase the learning cost of the team's small partners, the mobile side ofReact-Nativethe choice of the scheme. This can be done either in a mixed-development manner or in a singleWebViewway (it has been verified that this scenario cannot be supportedWebRTC).

  Many people may have heard theAirbnbannouncement last year that they were no longer using itReact-Nativeas a mobile solution and explained it in detail, and many people were preaching that it was coolReact-Native. In fact, it is veryAirbnbclear in the statement that itReact-Nativeis a very good hybrid solution, and the problem is that when the performance and user experience are optimized to a certain extent, there is too much manpower invested in the maintenance and development of hybrid technology, and the front-end personnel of the entire project not only have a web front-end, There are also high-levelAndroidandIOSpersonnel to ensure the promotion of the hybrid project, they believe that such human costs compared to the original development is much higher, so replaced the scheme. Do you understand me? Therefore, as a software technology than foreign countries do not know how many years of the celestial yards, considering the actual project needs, despite the confidence of the bold use of the good, the follow-the-way really no value.

• reason :

• Popular hybrid solutions, and web front end of one of the three carriagesReact, the syntax and components of the structure of a high similarity, community active and the surrounding ecology is better.

• Test Results :

• React-nativeHas been released0.57.3, but tested0.55.4in the domestic version of the normal new project (usingreact-native init XXXthe command created by the project), the0.56release of the large version of the two versions are in the initial packaging times error, command line prompts link to a known issue, But unfortunately, after doing not packaged successfully, the0.57default android-sdk is, that isAPI27, Android8.1, for inexperienced developers (such as myself), too new version is not recommended, unless your project is running on the specified machine.

  React-nativeComponents are also encapsulatedWebView, but unfortunately, the way the Web app is loaded directly is not testedgetUserMedia( ), so it can end up being implemented in a mixed-development scenario (but, in retrospect, compared toWebViewcalling hardware interfaces by In fact, this way of implementation is more logical.

• Conclusion:

• Hybrid developers who do not have the ability to master multilingual mixed development are advised to choose popular solutions as much as possible, for the simple reason that all front-end projects have pits, but hot items can be consulted for a problem.

WebRTCTechnical recordings are relatednavigator.getUserMedia,navigator.mediaDevices.getUserMediaandAudioContextThis is true of the above scenarios, but the fact is that none of them can bewebviewrecorded in the microphone.

Of courseWebRTC, as an independent standard and technology, but also can be integrated into the Android project, but from the front-end developer's point of view this road is a bit off, clinging toWebRTCor the team has a native developer of the small partners can study.

Three. The overall architecture of the react-native solution

Basically as long as the use of ready-made components, plus the right amount of customization, as far as possible without some artifice, the smoothness of the product is basically not to distinguish whether it is hybrid development or native development, of course, with the author of the project volume is not very large also has a certain relationship.

Four. Using the plug-in checklist

• react-native-audio

  Address: Https://github.com/jsierles/react-native-audio

  Call the microphone to collect audio.

• rn-fetch-blob

  Address: Https://github.com/joltup/rn-fetch-blob

  Sending large-volume binary data or files directly from the native layer through the native thread in Rn can cause performance problems when requests are made from the Web via Bridge objects.

• MulterModule

  Address: Https://github.com/expressjs/multer

  ExpressServer-side middleware for receiving large volumes of binary data or files sent by clients.

• FFmpegTools

  Address: http://ffmpeg.org/

  Multimedia format Conversion Library. The format of the mobile phone-side acquisition code cannot be directly recognized by Baidu's speech recognition interface, and it needs to be re-coded first.node.jsdeveloperschild_processcan invoke command line execution directly from the code through the module.

• docxtemplaterModule

  Address: https://docxtemplater.readthedocs.io/en/latest/

  node.jsModule speech recognition results need to be generated in the backgrounddocxformat of the file (Word document), you can use this module, using the method and template rendering engine is basically consistent.

Five. RN development details and the pits encountered

1. Real-Time debugging, you need to shake the phone, in the configuration menu, fill in the intranet ip+ port number , or you will be directly red screen error.
2. Real-Time debugging, you need to open the app'shover boxpermission in the settings, otherwise the white screen will not show anything.
3. WebRTCis not compatible withAndroid WebView,IOSbuilt-in browser is not supported.
4. React-native-audiowhen recording, each time you callStop, to start recording again, you must first call theAudiorecorder.preparerecordingatpath ()method reinitialization, or red screen error. The
5. WebViewcomponent must be set toref={(WebView) =>{ This.webview = WebView}}, otherwiseonMessageproperty cannot be heard fromwebviewload Web page throughWindow.postmessage the message that was sent by the. The
6. touchablehighlightcomponent must first set the callback function for theonpressproperty (which can be an empty function), otherwise touch the color response propertyUnderlaycolorcannot take effect. The
7. Modalcomponent can have only one in a custom component (if more than one must be instantiated by criteria), or even if thevisible ofModalcomponents is not displayed Theproperty is set tofalse, and its instance method overlaps with anotherModalcomponent, which may occur when the firstModalinterface is displayed. A secondModalmethod with the same name was executed.

react-native0.55.4-based speech recognition project full stack scheme