Getting to know Tensorflow.js
The purpose of Tensorflow.js is to help front-end engineers and lower the threshold for enhanced learning. In addition, let everyone give full play to their creativity and implement
machine learning in front-end applications.
It itself is an all-round platform. It is not a simple platform to call the server, but a platform completely implemented by JavaScript. It allows you to convert your existing models from Python and execute them in the browser. The existing model can also be retrained through transfer learning or reinforcement learning. In addition, it allows you to create models. If you are familiar with Tensorflow.js, it has a corresponding API. You can create models and train them through JavaScript. Generally, Node.js is more suitable, so we also support Node.js. .
Why do you not understand this thing? Maybe you don't know what its application scenarios are, and everyone is generally concerned about big data and other big issues. I personally think that there are many small problems that can be solved with it, such as cool interaction methods that can be used to achieve it.
Tensorflow.js application scenarios
The application scenarios of Tensorflow.js are:
1. Augmented reality AR
For example, the functions of Douyin are implemented on the end, and they can also be implemented with JavaScript.
2. Interaction based on gestures or body
Now we use fingers for mobile phones and mice for websites. Through it, you can add new ways of interaction, such as mouth shape, eyes, and gestures.
3. Voice recognition
After using it, speech recognition can be injected into the website, and some new experiences and new interaction methods can be added. I will give some examples later. In any case, the Web is a text-based environment, and text-oriented models are very suitable in this interactive environment, such as semantic analysis and intelligent conversation. These models can be directly executed in the App. Finally, through analysis, the performance of the device itself or the network situation can be used to optimize the web page.
Why are these scenarios implemented with
machine learning on the end instead of the server? The advantages are:
1. Protect user privacy
2. Reduce response delay and reduce traffic
It does not require a network. If the network is not good or the traffic is limited, it can help users reduce traffic.
3. Improve user experience
Some functions cannot be achieved in real time. 5G may be possible in the future, but it cannot be achieved now.
4. Reduce server costs
By putting ML on the end, the dependency on the server is reduced. If the developer builds the service by himself, the cost will be high, which can reduce the cost of the server.
The purpose of Tensorflow.js is to simplify these problems and provide front-end engineers with all the functions that can be achieved. Our goal is to provide many existing models for everyone to use. And these models are open source, you can use them as you like.
Five models
Tensorflow.js provides five types of models: image classification, object recognition, gesture recognition, voice command recognition, and text classification. You can install it directly with NPM, or you can install it directly through JavaScript and put it on the web page.
1. Gesture recognition
The gesture recognition model can be run in a small program. It provides the coordinates of the human's five senses and the coordinates of the limbs and joints. It has a variety of different precision and speed optimized versions. You can run on different devices by installing different versions.
2. Character segmentation
It can help you do background blur, or classification of various parts of a person, such as head, torso, and limbs. You can also do a lot of interesting things, and have different accuracy for different devices.
3. High-resolution face tracking
This is a model we are about to launch called FaceMesh, which has about 400 key points on a person's face, and it provides a three-dimensional face model.
4. Voice command recognition
We once asked a patient with frostbite to interact with a machine and press each button with a laser. The interaction speed was very slow. The patient with frostbite could not speak, but he could make some sounds. His voice had different meanings. We passed this The model makes a personalized interaction for them. Let them easily use different voices, such as "I want a pizza", which increases vocabulary through constant training.
5. Text model
In the text model, support such as tone analysis, toxic language detection, and intelligent reply. We hope to make some useful and usable models for everyone.
6. Transfer learning
In the example, my colleague did 3 different actions. He started training the model in real time by taking pictures of the video. The training speed is very fast, about a few milliseconds or tens of milliseconds. Now he is using very high accuracy Recognize 3 different actions, one is to wave your hand, the second is to shake your head, and the third is to put your hand next to your mouth.
7. Support multiple platforms
Tensorflow.js supports multiple platforms: wireless (WeChat applet, React Native), server (Node.js); desktop (Electron), browser (Google Chrome, Firefox, Safari).
Let me give you an example. On the Google search engine, there is a picture of its logo. We turn this picture into a game. This is to commemorate Bach in the eighteenth century. His music is very rhythmic. So we made a model: the player can compose a song in one part, and finally a complete four-part concerto is formed through the machine model.
This model is CoCoNet. Players have been online for 78 years and have shared more than 50 million creations. It is probably the largest user-generated symphony library. The technology it uses is Tensorflow.js and Google's TPU.
Some users have mobile phones that are not fast, so they need to go to the server. If the performance of the phone is good, it can be executed on the phone. It can also save 50% of server costs, because it makes full use of the user's GPU, and it is conceivable that if it is a very large project, how much can be saved.
