javascript--Next Generation Internet of things full stack development

Li Zhizhou, PhD, Institute of Micro Systems and information technology, Chinese Academy of Sciences, early entrepreneur of the Internet of Things, launched the Open source IoT project Openfpgaduino, currently engaged in network security development based on big data and machine learning in internationally renowned investment banks.
This article is "Programmer" magazine original article, Unauthorized, do not reprint
Follow the public number "CSDN IoT development" to learn more about IoT information and dry goods

Jeff Atwood has proposed that "any application that can be implemented in JavaScript will eventually be implemented in JavaScript" and his admiration for JavaScript is self-evident. Today, when the Internet of Things is hot, JavaScript supports HTTP and JSON, supports functional programming, and provides interactive environments that are suitable for all-stack development of the Internet of things. This article provides an explanation of each of the IoT processes that JavaScript can participate in, and summarizes the open source projects.

IoT Node JavaScript Development

In the network, the HTTP protocol and JSON data format, especially the RESTful API, are undoubtedly dominant, and various cloud services and data transmissions are based on these protocols. JavaScript provides the best support for HTTP and JSON, and when the IoT system is developed with JavaScript, it has a natural connection to the Internet. The cloud services and cloud resources, including cloud storage, cloud computing and other resources can be easily invoked, Just like you access a variety of cloud services on your phone. The rise of the microservices architecture on the server side allows every IoT node written by JavaScript to serve as a micro service in a large system, providing its own services through the RESTful API interface.

In design mode, the event-driven programming model, such as JavaScript callbacks and event loops, is ideal for the internet of things. In the context of IoT, the environment is constantly changing, and IoT nodes are constantly responding to changes in the environment, in other words, IoT systems are often I/O intensive systems, and callbacks and event loops efficiently perform intensive I/O operations, While event-responsive programming is highly efficient in memory usage compared to multi-process and multithreaded programming, which is what IoT systems need, often IoT systems are resource-constrained systems with very limited memory and CPU frequencies. In the IoT node development, the interrupt response mode is often used to check the presence of the interrupt signal in the CPU by the hardware called the interrupt controller, and to control the CPU to execute a specific program fragment after the interrupt occurs, which is consistent with the JavaScript callback and is easy to use JavaScript Callback mechanism to implement a hardware interrupt response.

The deployment characteristics of IoT nodes determine that the cost of recovery and maintenance is very high and even unacceptable. The IoT node needs to constantly respond to new environments and applications, so remote deployment and updating of IoT nodes in development is a very important feature. JavaScript is a language that is deployed from the server side to the client, and naturally has the properties of remote deployment on the Web, as simple as downloading JavaScript scripts and running in a browser. The hot deployment of JavaScript is also a hot research area, and through thermal deployment, IoT nodes can remotely add new features and fix bugs remotely while running.

With the benefits of using JavaScript for IoT development, it's also important to talk about the differences and pitfalls of javascript in Internet of things applications versus WEB applications. The first problem is real-time sex. As an embedded system of the IoT gateway, the first is the real-time problems, such as the control of various motor, some voltage signal acquisition is real-time requirements, need to complete the required work within the specified time. Because JavaScript's memory management uses a garbage collection mechanism, there must be a situation where the application is interrupted due to garbage collection, and may even cause real-time requirements to be compromised in some cases. So when designing JavaScript IoT real-time programs, you need to consider the impact of time on the real-time nature of memory garbage. Of course, you can also adopt another design pattern, that is, the real-time task is written in the C language of the separate thread, using JavaScript single-threaded features, this design pattern does not occupy an excessive amount of resources in the embedded system, after all, JavaScript is a language embedded in the browser. On the other hand, since IoT gateways are devices that require long-term operation, it is necessary to have higher reliability and stability relative to the Web side, so unit testing and integration testing are necessary, and code analysis tools are required to ensure that the code does not have any leaks of memory and file handles. Even in the garbage collection environment, memory leaks exist, and in embedded systems due to limited memory, it is particularly vulnerable to leaks. Fortunately, the development of server-side JavaScript has been a long time, you can learn from the server development of many tools.

JavaScript IoT node Development Open Source project summary

Openfpgaduino (https://github.com/OpenFPGAduino/) is an open source IoT hardware and software development platform based on node. JS and FPGA, similar to the rapid prototyping platform such as Arduino. Based on node. JS, network interactivity is provided, including network Ides for development and Linux software development systems similar to Raspberry Pi for easy installation of various Linux generic software, and programmable compatible Arduino peripheral interfaces via FPGAs for use with rich IoT sensors To connect.

Iotjs (HTTPS://GITHUB.COM/SAMSUNG/IOTJS) is a Samsung-developed embedded JavaScript execution environment that enables complete network and JAVASCRIPT support on embedded systems with hundreds of KB of memory without relying on the operating system. The JavaScript interpretation engine uses

Jerryscript (Https://github.com/jerryscript-project/jerryscript), the event loop takes Libtuv. It can be said to be the embedded version of node. JS and its API is compatible with node. js.

Esprunio (Https://github.com/espruino/Espruino) is an embedded JavaScript interpretation engine developed for micro-control, which provides a minimal configuration to execute JavaScript on a microcontroller with memory down to 8KB Language.

Cyclon (Https://github.com/hybridgroup/cylon) is a project that uses JavaScript to do robot control, supports major development boards on the market, and provides drivers for a variety of peripherals without having to write a C Language can develop and assemble a robot.

Internet of things JavaScript big Data processing

Now that big data technology is in the ascendant, the most basic idea in these technologies is MapReduce, which splits different operations into maps and reduce, and then maximizes the parallel execution of these maps and reduce tasks in the cluster. One of the most important things in implementing MapReduce pattern programming is the support for functional programming or LAMBDA programming, where all MAP and Reduce operations accept functional programming as parameters. The success of big data has driven the revival of functional programming, with most modern programming languages including new versions of C + + and Java supporting functional programming. JavaScript supports functional programming on the day it is born, and its callback function is a functional programming pattern. Naturally, it is not difficult to support MAP and Reduce, so it is easy and efficient to use JavaScript for IoT Big Data analysis and processing. JSON data format because of simple format, easy to handle the characteristics of a lot of favor, is widely used in HTTP network transmission, the existing Internet infrastructure and cloud services are also in JSON format. With these facilities as data sources, it is necessary to use JavaScript to handle big data, because JSON is a snippet of JavaScript code, essentially all the interfaces based on RESTful API design are almost javascript-friendly, and of course, for efficiency, sometimes Use the binary form of JSON BSON.

Because of the symmetry of the Internet of things, the nodes of a physical network can be not only the data collector, but also the initiator of the data, using JavaScript to deal with big data of the IoT, then each IoT data node can initiate the processing of big data and take advantage of the background big Data environment to make decisions and respond to changes in the environment. This makes use of JavaScript to directly link the IoT front-end and back-end data processing, realizes the LAMBDA architecture of real-time processing and response of big data, and simultaneously completes the unification of batch processing and real-time processing architecture. In the future, we can use JavaScript to design a processing framework based on node. JS, and use the physical network nodes as big data nodes to unify the processing and transmission of data in the Internet of things.

Machine learning has to be mentioned as the hottest big data-processing tool now. Due to the dominance of JavaScript on the browser side, there are many attempts to machine learning using JavaScript, and with JavaScript's machine learning Library, it is very easy to run some machine learning programs on the browser, as well as to run these machine learning libraries node. js makes the IoT nodes also capable of machine learning and data processing. Of course, the direct control of deep machine learning clusters using JavaScript is also a good choice, the success of JavaScript in data visualization, help it can easily visualize the process and results of machine learning, convenient remote control machine learning cluster.

JavaScript IoT Big Data Open Source project

Eclairjs (Https://github.com/EclairJS/eclairjs-node) is a front end of the Spark Big Data processing platform based on node. JS, and through Eclairjs, JavaScript can be used To manipulate the memory-processing-based Big data analytics platform Spark, through the built-in scheduler of Spark, implements the algorithm that schedules resources across the cluster to maximize parallelism to execute JavaScript. At present, Spark is the most mainstream and the fastest growing big data processing platform.

Skale (https://github.com/skale-me/skale-engine) is a completely JavaScript-based distributed data processing system that uses node. js as the execution environment, with faster performance than Spark.

MQTT is a lightweight communication protocol designed for the internet of Things, which is based on TCP/IP and is suitable for low-bandwidth, unreliable environments. The cost of the protocol is very small, supporting the publish-subscribe model, which is a very efficient communication protocol. Mqtt.js (Https://github.com/mqttjs/MQTT.js) is an MQTT communication module for node. js that enables node. js to send and receive MQTT data.

The Kafka is a large data distributed messaging bus that provides persistent capabilities for massive messages, providing consistent data throughput and high availability by directly persisting the received data to the hard disk. The Kafka Rest (https://github.com/confluentinc/kafka-rest) module provides Restful API support for Kafka message bus, which is easy to use, whether using a browser or node. js JavaScript to send and receive messages.

Convnetjs (HTTPS://GITHUB.COM/KARPATHY/CONVNETJS) is a neural network machine learning library written entirely in JavaScript that can be run in a browser or in node. js. Convnetjs provides a number of interesting examples, in the browser implementation of the neural network learning process of visual animation and interactive mode, to understand the neural network machine learning is very good help.

As the most important and most popular project in the field of deep neural network learning, Google's TensorFlow is a milestone in deep learning of open source. TensorFlow's open source makes using deep learning to do data processing is no longer a difficult task. Node-tensorflow (Https://github.com/node-tensorflow/node-tensorflow) is a NodeJS API that uses Google's Open source machine learning Library TensorFlow.

Internet of things JavaScript data show

On the browser side, JavaScript has long dominated the visualization of data, and as the only programming language that can now run on all browsers, using JavaScript not only gives users an interactive environment, but also makes very nice charts and animations. including two-dimensional chart curves and three-dimensional virtual space, can be implemented using JavaScript control browser, now many browser-based VR environment can also be developed through JavaScript. The use of JavaScript to do the internet of things data display is very suitable.

In today's Internet world, JavaScript is already a full-stack web development language. In particular, after the rise of HTML5, JavaScript is no longer confined to the browser side, but extends its tentacles to all aspects of the network, including the IoT side, mobile phones and traditional PC-side. HTML5 is a browser-designed, well-developed JavaScript cross-platform feature that truly develops a JavaScript code that runs across platforms from IOS, Android to PC-side Windows, MacOS, and Linux on your phone. While JavaScript still does not match the performance of native development applications at the moment, the development costs and responsiveness are unmatched by other development approaches, using only a single JavaScript for the development of applications. Especially in the IoT development environment, many applications are lightweight and do not have a particularly high performance requirement, but they need to be tailored and adapted to different IoT environments, and JavaScript is the right way to do such data presentation work. According to the symmetry of the network, and even, it is possible to use JavaScript to achieve data display on the IoT node, as we use the mobile phone for data display and control.

JavaScript data show summary of open source projects

D3 (HTTPS://GITHUB.COM/D3/D3) is an SVG-based vector diagram manipulation tool. The D3 is the underlying standard for two-dimensional visualization before HTML5 occurs by using JAVASCRIPT,D3 to present the data in vector form and providing the ability to scale arbitrarily. Often many charting and data visualization tools use D3 as the underlying driver engine, providing a user-friendly API interface on top of it.

Three.js (https://github.com/mrdoob/three.js/) is a WebGL-based Javascript3d drawing tool. With WebGL extensions, the JavaScript language enables direct drive of the underlying graphics for 3D plotting. The Three.js also provides VR support on 3D images, enabling the easy use of JavaScript for VR data presentation.

Ionic (Https://github.com/driftyco/ionic) is a AngularJS-based, mobile-side, cross-platform development environment that enables the use of JavaScript HTML CSS in a single unified framework, while completing the development of IOS, Android, Windows Phone program, and automatically adapt to different UI for each environment.

Electron (Https://github.com/electron/electron) is a cross-platform desktop development environment based on node. JS and Chrome, which is packaged by the browser kernel and the backend node. JS service. Implements a full-stack development environment that uses JAVASCRIPT to deploy programs to Windows, MacOS, and Linux platforms without any modification.

NW (Https://github.com/nwjs/nw.js) is a lightweight JavaScript cross-platform desktop development environment that implements a variety of API functions that directly invoke node. js in HTML code, providing a smooth, natural desktop application development environment Hides the details of client server programming.

The Internet of Things JavaScript example tutorial

To better understand JavaScript IoT development, the following is an example of the IoT system designed by the author, which introduces the design of the IoT system in detail. All design materials are open source, readers can control the source pages and documents. (Http://openfpgaduino.github.io)

Hardware design of the minimum system of Internet of things Gateway

For the embedded system used in the Internet of things, the hardware design of the system is an important part, usually dealing with different applications, the embedded hardware system design will be different, each system will be adjusted according to the corresponding application scenario. But the core system of the Internet of Things is usually a design framework that can be applied to a wide class of systems that need to be tuned often to peripheral devices.

Figure 1

1 is the design of the IoT Gateway minimum system, in the system design thinking first to ensure the universality of the core system, the use of ARM processor. Because it is a network joint point, the design requires the gateway to directly connect the IP network and provide full-featured network services, so the choice of a full Linux system to run the ARM9 processor, running the full Linux is also able to run node. JS and complete JavaScript requirements. Due to the use of standard Linux, the Linux kernel provides some real-time patches, but because the programming model is complex and not suitable for integration with node. js, FPGAs are added to the smallest core system. FPGA can complete the real-time response at the hardware level, solve the real-time response requirements of the system, secondly, because FPGA is a configurable system, can be designed and configured according to the application needs, so the introduction of FPGA also greatly improves the system's adaptability, and provides the basis for flexible adaptation of various kinds of IoT applications.

Software design of Internet of things gateway system

The software design of IoT Gateway is based on the architecture of hardware design, maximizing the function of hardware, the design process is as follows:

1. First run the full Linux system on the ARM9 processor, which has a complete network service and can install the required packages as needed via the Apt-get command.
2. On the basis of the Linux system, run NODE.JS+MONGODB to provide all network services.
3. Run an HTTP server in node. js to provide a variety of local services based on Restful APIs, including control of various peripherals and I/O.
4. Running the HTTP server, based on the RESTful API, implements the HTML5-based cloud IDE development environment. Convenient technical geeks Customize the capabilities of the gateway, including JavaScript IDE, Blockly IDE based on graphics programming, WEB design tools that can configure FPGA module functionality and PIN definition, and, of course, the lower C language IDE.

With these features, users can write a simple program to control any device connected to the IoT gateway and implement their own smart internet of things according to their needs.
Within the board, the Linux kernel-driven user space model enables node. JS to directly access the FPGA memory space, control FPGA internal logic, and ultimately control the functionality of various IoT peripherals connected to the FPGA. At the same time, it realizes the site reconfiguration of node. js to FPGA, so that the FPGA configuration file can be downloaded directly from the network, and the whole core system will be changed on site.

Software design of Internet of things gateway system

Design of Internet of things cloud building system

3 shows the cloud Building system of the IoT gateway, which shows that the IoT cloud build system consists of github,docker build and seven Qiniu storage. The idea of a cloud-building system comes from continuous integration, where the concept of continuous integration is commonplace, and each time a commit to a code change triggers a continuous process of compiling the test. Simulation software continuous integration, in the cloud to build a corresponding to the IoT gateway of the continuous integration environment, the choice of the cloud because the embedded system itself, the ability and processing speed is limited, not suitable for continuous integration of the build environment. In the cloud build system, use GitHub to manage the underlying version of the code, use the Docker Hub's container build mechanism to build the application and FPGA programs, and finally use the seven Qiniu as the distribution platform for the build results. All Boards drive cloud-compiled environments through RESTful APIs and access seven Qiniu for program configuration upgrades. With this cloud-based continuous integration system, not only can the application be deployed quickly, but also open to the cloud development environment that enables users to customize the development, users can easily realize the custom design of FPGA by using this cloud environment, and realize a set of FPGA design environment in the cloud.

Cloud Architecture for IoT gateways

Design of big Data processing system for Internet of Things

4 is shown in the Internet of things big Data processing system Design framework diagram, IoT Gateway collected IoT data by the System of node. JS is sent to the Big Data message bus Kafka HTTP proxy, each Kafka HTTP proxy can undertake thousands of such IoT gateways , and proxy will also send aggregated data to the Kafka cluster for further summarization, and then within the Kafka cluster to implement the ETL process of data, including data outlier filtering, deduplication, data format conversion. Finally, the Kafka cluster is used to send data to different downstream consumers, which is sent to the Big data storage System S3 or HDFS for permanent preservation, providing data for future offline analysis systems such as Hadoop or Spark, and then the data is sent to the ElasticSearch for indexing. The data will be searched and queried quickly, and the data can be visualized by using Kibana to provide the health monitoring of the whole data stream; The final data is sent to the real-time online large data processing system Spark streaming for data online analysis, machine learning and other tasks, to achieve data analysis and response online.

Design structure diagram of large data processing system

Reference documents

• https://blog.jscrambler.com/javascript-the-perfect-language-for-the-internet-of-things-iot/
• http://fex.baidu.com/blog/2015/05/nodejs-hot-swapping/
• Https://cnodejs.org/topic/5659a9e0c4fa25cb27cc3c23
• Http://openfpgaduino.github.io/docs/index.html
• http://blog.csdn.net/brucesea/article/details/45937875
• https://www.burakkanber.com/blog/machine-learning-in-other-languages-introduction/

javascript--Next Generation Internet of things full stack development