Full stack scheme of speech recognition project based on react-native0.55.4

Source: Internet
Author: User

Tags: other support 5.0 ble EXPRESSJS FFFFFF Plugin images error

The API ability verification scheme of mobile side is different from PC side! Not the same!! Not the same!!!

Even if the need to use API all exist, not necessarily can be used, this and PC side is very different, the domestic mobile phone system although are based on Android , but almost will go through the major manufacturers of customization, the function and the original Android system is not exactly the same, in the investigation of technical solutions must be confirmed with Just demo run the function and don't ask me how to know.

A. Mobile direct access to Web apps?

PC-based Web API speech recognition program can refer to the "recorder.js+ Baidu speech recognition" Full stack scheme technical details of the article.

1. Web API The calling multimedia acquisition interface requires a specific domain

Web APIMultimedia interface is the implementation of WebRTC technology in the PC side, because the multimedia acquisition involves user privacy, so in the browser-side call this interface needs to be in a secure domain to be transferred, secure domain refers to the following three categories:

    • file:///Local area
    • http://localhostLocal Web server
    • https://Security Domain

The first two categories are generally used for desktop applications and local debugging, the actual site on-line deployment needs to be deployed in a https way, how to deploy https and apply for free CA certificate Online There are many articles to explain, this article will not repeat.

2. The mobile browser almost does not support the WebRTC interface directly

After the PC-side Web application is deployed in a https way that does not evoke the recording interface authorization when accessed directly from the mobile browser, the navigator.getUserMedia( ) method returns an permissionDenied error, either by editing the Android6.0 manifest.xml or by adding or Android6.0 above through the dynamic access to obtain RECORD_AUDIO access to the site can be normal access, the relevant Web API interface is also present, but even if the user authorization can not be adjusted to the recording function. I tested the UC browser , Baidu Mobile browser and Android6.0 (API23) comes with the browser ,Android8.0 (API26) comes with the browser , the result is not supported.

Two. Programme research and new pits

O ( ̄▽ ̄) d since the mobile side of the direct access to the Web application can not be transferred to the recording interface, at least not compatible with many systems and models, if not consider the direct native development of Android, only hope that the hybrid solution.

2.1 WebView
    • Solution :

      Place a WebView component in a single page full screen in one app, and then load a Web app that is deployed in HTTPS mode.

    • reason :

      If the mobile browser cannot support it, it can only be sent in the hope WebView . WebViewis the android underlying component for loading Web pages, the Android4.4 version has replaced the built-in browser engine with chromium, the chrome kernel, from Can I use The support of the query above Android5.0 is the WebView WebRTC method that supports the interface getUserMedia( ) .

    • Test Results :

      The application compilation target version is API23 , in the support API23 (Android6.0) of the virtual machine and the real machine test, can not be transferred through the WebAPI interface microphone for recording. In a supported API26 (Android8.0) version of a virtual machine, functionality is achievable. Finally, in the remarks on the statistical information of the method support in Can I Use, it is found that the getUserMedia( ) known issue is stated in the following:

Simply put, this method is Android webview , iOS and Basic, PWA not used. It is suggested that some of the less commonly used APIs may be used in future development to take a complete look at the relevant information.

    • Conclusion:

      Android8.0 Support, Android support is poor, not recommended.

2.2 Crosswalk
    • Solution :

      Official website: https://crosswalk-project.org/

      With crosswalk , when the app is packaged, webview replace the kernel xwalk with the chromium-based browser kernel developed by crosswalk to extend native webview capabilities.

    • reason :

      Since native webview functionality is castrated, this small black technology can be used to package a more powerful browser kernel with its own application, the author 3 years ago in the cordova 2.0-3.0 version of the popular era of the use of this technology, the benefits are indeed can expand webview the ability undoubtedly, The downside is that the app project will directly increase the volume of the 80-90MB, of course, through several versions of the iteration, now crosswalk can generate a different package for the phone kernel type, app volume increment of about 20Mb, basically belong to the acceptable range.

    • Test Results :

      Unfortunately this project was discontinued a year ago, and the final version of the official scaffolding tool is also unable to initialize the new project, the indirect use of the way divided into two kinds, first, download crosswalk the package, manually in the android project to replace the native WebView , It is difficult for hybrid developers and is not controllable with hybrid technology compatibility; another scenario is described in the next section.

    • Conclusion:

      Not recommended, it's better to look at a reliable hybrid solution.

2.3 Cordova/ionic

  • Solution :

    Official website: https://cordova.apache.org

    codovais a popular hybrid solution that has now been upgraded to version 8.0.0, which is itself a solution for packaging web apps as apps. cordovaThe basic principle is to UI put the general layer of operations and functions in the WebView implementation, need to call mobile device hardware or native interface, are implemented by adding cordova plug-ins, each cordova version will be across the support of several Android versions, such as the cordova7.0.0 new The official documentation is supported by Android from 4.4 to 8.1 versions, which I think is ideal for small hybrid development teams to use.

  • reason :

    It is worth mentioning that cordova has a very popular mobile development xxx ionic , has now been iterative to the 4.0 stage, the author of this technology has a special feeling, the same year ionic in the alpha version, the author is in use, it is based on Cordova+angular This technology portfolio, with a fresh and highly-designed UI component, is worth trying. In addition, is the plug-in cordova crosswalk , can be directly in the form of plug-in, in the cordova project packaging crosswalk , the relevant needs of readers can take a try, especially when the team does not have the Android developer and no special designers, ionic The application of the production will certainly make others treat you differently.

  • Test Results :

    I have been in the use cordova of 3.3 when the integration crosswalk , but also through cordova the plug-in successfully called the bottom GPS , 摄像头 and some other native components, was to adapt to the Android4.4 version. cordova7.0.0scaffolding tested in the domestic can be used, the new project either by the command line or import into the Android studio to develop can be packaged as a corresponding project, the official document has a very detailed call to various low-level interface instructions, the Internet also has cordova7.0.0+crosswalk Technical stickers for the solution.

    As a result of the technical protocol specified in the technical stack for the sake of not halfway to replace the solution, so this time has not been tested.

  • Conclusion:

    Consider trying as a total solution.

2.4 react-native

  • Solution :

    Official website: https://reactnative.cn

    This is the author of this use of the program, because the Web-based React technology stack to complete, in order not to increase the learning cost of the team's small partners, the mobile side of React-Native the choice of the scheme. This can be done either in a mixed-development manner or in a single WebView way (it has been verified that this scenario cannot be supported WebRTC ).

    Many people may have heard the Airbnb announcement last year that they were no longer using it React-Native as a mobile solution and explained it in detail, and many people were preaching that it was cool React-Native . In fact, it is very Airbnb clear in the statement that it React-Native is a very good hybrid solution, and the problem is that when the performance and user experience are optimized to a certain extent, there is too much manpower invested in the maintenance and development of hybrid technology, and the front-end personnel of the entire project not only have a web front-end, There are also high-level Android and IOS personnel to ensure the promotion of the hybrid project, they believe that such human costs compared to the original development is much higher, so replaced the scheme. Do you understand me? Therefore, as a software technology than foreign countries do not know how many years of the celestial yards, considering the actual project needs, despite the confidence of the bold use of the good, the follow-the-way really no value.

  • reason :

    Popular hybrid solutions, and web front end of one of the three carriages React , the syntax and components of the structure of a high similarity, community active and the surrounding ecology is better.

  • Test Results :

    React-nativeHas been released 0.57.3 , but tested 0.55.4 in the domestic version of the normal new project (using react-native init XXX the command created by the project), the 0.56 release of the large version of the two versions are in the initial packaging times error, command line prompts link to a known issue, But unfortunately, after doing not packaged successfully, the 0.57 default android-sdk is, that is API27 , Android8.1, for inexperienced developers (such as myself), too new version is not recommended, unless your project is running on the specified machine.

    React-nativeComponents are also encapsulated WebView , but unfortunately, the way the Web app is loaded directly is not tested getUserMedia( ) , so it can end up being implemented in a mixed-development scenario (but, in retrospect, compared to WebView calling hardware interfaces by In fact, this way of implementation is more logical.

  • Conclusion:

    Hybrid developers who do not have the ability to master multilingual mixed development are advised to choose popular solutions as much as possible, for the simple reason that all front-end projects have pits, but hot items can be consulted for a problem.

WebRTCTechnical recordings are related navigator.getUserMedia , navigator.mediaDevices.getUserMedia and AudioContext This is true of the above scenarios, but the fact is that none of them can be webview recorded in the microphone.

Of course WebRTC , as an independent standard and technology, but also can be integrated into the Android project, but from the front-end developer's point of view this road is a bit off, clinging to WebRTC or the team has a native developer of the small partners can study.

Three. The overall architecture of the react-native solution

Basically as long as the use of ready-made components, plus the right amount of customization, as far as possible without some artifice, the smoothness of the product is basically not to distinguish whether it is hybrid development or native development, of course, with the author of the project volume is not very large also has a certain relationship.

Four. Using the plug-in checklist
    • react-native-audio

      Address: Https://github.com/jsierles/react-native-audio

      Call the microphone to collect audio.

    • rn-fetch-blob

      Address: Https://github.com/joltup/rn-fetch-blob

      Sending large-volume binary data or files directly from the native layer through the native thread in Rn can cause performance problems when requests are made from the Web via Bridge objects.

    • MulterModule

      Address: Https://github.com/expressjs/multer

      ExpressServer-side middleware for receiving large volumes of binary data or files sent by clients.

    • FFmpegTools

      Address: http://ffmpeg.org/

      Multimedia format Conversion Library. The format of the mobile phone-side acquisition code cannot be directly recognized by Baidu's speech recognition interface, and it needs to be re-coded first. node.jsdevelopers child_process can invoke command line execution directly from the code through the module.

    • docxtemplaterModule

      Address: https://docxtemplater.readthedocs.io/en/latest/

      node.jsModule speech recognition results need to be generated in the background docx format of the file (Word document), you can use this module, using the method and template rendering engine is basically consistent.

Five. RN development details and the pits encountered
    1. Real-Time debugging, you need to shake the phone, in the configuration menu, fill in the intranet ip+ port number , or you will be directly red screen error.
    2. Real-Time debugging, you need to open the app's hover box permission in the settings, otherwise the white screen will not show anything.
    3. WebRTC is not compatible with Android WebView , IOS built-in browser is not supported.
    4. React-native-audio when recording, each time you call Stop , to start recording again, you must first call the Audiorecorder.preparerecordingatpath () method reinitialization, or red screen error. The
    5. WebView component must be set to ref={(WebView) =>{ This.webview = WebView}} , otherwise onMessage property cannot be heard from webview load Web page through Window.postmessage the message that was sent by the . The
    6. touchablehighlight component must first set the callback function for the onpress property (which can be an empty function), otherwise touch the color response property Underlaycolor cannot take effect. The
    7. Modal component can have only one in a custom component (if more than one must be instantiated by criteria), or even if the visible of Modal components is not displayed The property is set to false , and its instance method overlaps with another Modal component, which may occur when the first Modal interface is displayed. A second Modal method with the same name was executed.

react-native0.55.4-based speech recognition project full stack scheme

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

Tags Index: