Deep understanding of the development of Kinect for Windows

---- Microsoft build2012 Conference: Kinect for Windows Programming deep dive

This weekend, I attended the two-day Microsoft build 2012 conference. It was really a big event with many topics on it. They were the latest developments in Microsoft related technologies and products. net, WPF, office, and Kinect related development, the above speech video and PPT can be downloaded, I personally think that the evolution. net. new versions and features released at previous developer conferences since net, which were announced at the Developer Conference by Mr. Bill cover more than 10 years ago. NET strategy speech video, do not feel the sigh of time is to kill. What's new for developers in office 2013 and SharePoint 2013 introduces a new office development method officeapp different from the previous ones using VBA, sharedaddin, and vsto, building apps for office and SharePoint 2013 using the web technologies you know and love, Part 1. Part2 describes how to use a variety of familiar technologies such as HTML, JavaScript, and C # To build an officeapp. There are also many introductions on Windows Phone 8, wpf4.5, and other topics. We strongly recommend that you go and have a look.

At the Conference, there were three articles about the development of Kinect. The first one was that the super-natural interaction PPT presented by Microsoft Research Institute was bigger than the presentation video, reaching a rare 997 M. Why is it so big, because it has embedded a 40-minute video, haha, this speech mainly demonstrates the various research activities of Microsoft Research Institute, such as man-machine interaction and virtual reality, which are very cutting-edge, there are also some parts related to the Kinect. If you are interested, you can check them out. The second is the Kinect design considerations. a speech on the design of the Kinect application should be considered, such as the design of the interactive mode, however, the content should be similar to that of the Kinect Human Interface guideline. You can view the content in the Kinect developer toolkit. The third speech is the Kinect for Windows Programming deep dive. In order to gain a deep understanding of the development of the Kinect for Windows system, this speech is relatively related to the development of the Kinect system, the speech was written at the level of 300-advanced. I personally think the content of this speech is actually very simple. It is just a data source that the Kinect SDK can obtain, as well as future trends, we will give a brief introduction. In fact, there are not many things in the Kinect SDK. The real thing is that various pattern recognition algorithms, such as deep data, infrared data, and various object recognition. With this data, in particular, the SDK version 1.6 provides the ability to obtain raw infrared data, which can be combined with deep data to make a lot of amazing applications. I will share with you the main content of this speech. It is also a good introduction to the Development of Kinect.

Outline 1

The speech is divided into five parts. The first part introduces the development method and running environment of the Kinect SDK, and the second part is code writing to explain some color and depth in the Kinect SDK, the third part describes the data streams directly generated by sensors, including color, depth, infrared, voice, and accelerator data sources, the fourth part describes the bone, speech, and facial recognition data sources that can be directly recognized after the original data stream is processed by the Kinect SDK, the last part describes a series of applications that can be implemented by Kinect in the future. Now let's take a look at the first part.

2. Application scenarios and development environment of the Kinect

First, we will introduce the application scenarios of Kinect. There are three main applications. The first is the natural human-computer interaction interface. For example, you can use your fingers to manipulate the display content on a large screen, such as the one in the dispatch report. This is often seen in high-tech or sci-fi movies. There are also some applications that have been used in foreign countries, such as the Kinect virtual fitting room, the demonstration advertisement of the Kinect auto show, the image operation of the Kinect operating room, the slide operation of the Kinect, and the picture browsing of the Kinect, these natural human-computer interaction interface applications bring about a new user experience. The second type is natural environment recognition, such as 3D scanning and reconstruction of Objects Based on the depth data generated by the Kinect, and automatic obstacle avoidance using the Kinect for robot navigation. The third type is natural user recognition, such as using Kinect for pose recognition and face recognition. For example, some familiar somatosensory games such as the xbox360 dance conference, sports meeting, and adventure all use some data of the Kinect to identify user movements and participate in interactions in the game. There are also some augmented reality applications that take advantage of the above three features, such as the use of Kinect to turn the screen into a touch screen, and can perform various operations on the above, and so on.

 

To develop the above application, first understand the development environment, as shown in this slide:

Kinect provides both C ++ and C ++ hosting (. net). If you use C ++ for development, you need to install the speech Runtime (V11), Kinect for Windows runtime and driver, if you use C # and VB. net, Microsoft. kinect. DLL and mirosoft. speech. DLL. net encapsulation, regardless of the development, you need to install the driver, all these are included in the Kinect SDK installation package, the installation method you can refer to the previous article. Kinect supports Windows 7, Windows Embedded standard, and Windows 8 operating systems. The latest SDK version 1.6 also supports Windows systems in virtual machines. However, Windows systems that are not too old can be used. It is recommended that you use. Net 4.0/4.5 in the previous version. If you are using. NET development, you need to install. NET Framework. IDE can use Visual Studio 2010/2012.

Three-code demo

This part mainly demonstrates the functions of Kinect by writing code. I will not talk about the code here.

The first demonstration is to display color images, which is relatively simple. Then, the depth value corresponding to a specific point is displayed based on the deep image data.

You can add a label by clicking the mouse on the graph. The value in the label is the depth value of the point. This function should be better implemented, it mainly shows how to use color image data and deep data. If you are interested, you should be able to implement this function after reading this article.

Then demonstrate the skeleton tracing function. In the figure, the position of the head and hands is tracked, and the depth value of the two hands is displayed. The tracking ID is displayed on the head, kinect can track 6 targets to be selected at the same time, but only two targets are active. Each trail is assigned a tracking ID.

This function mainly demonstrates how to use Skeleton data. If you are interested, you can implement this function after reading this article.

In addition to the various types of data that can be provided by 1.5 and earlier, 1.6 of sdks provide infrared image data, and you can use the data to calibrate color image data, this allows you to recognize characters in the dark. In fact, Kinect bone recognition is mainly implemented through deep data, while deep data is generated through infrared emission reception. The SDK of version 1.6 provides us with the ability to directly access and operate infrared data.

The fourth demo demonstrates the face recognition function of Kinect. the facial recognition function is introduced in the 1.5 SDK. Facial recognition can recognize up to one hundred facial features, the demo is as follows:

The above features include instances and source code in the Kinect developer toolkit. You can download and view them. If you are getting started, you can also read this article to briefly introduce how to use facetracking.

In addition to these features, Kinect also has powerful speech recognition functions that have not been demonstrated in this speech. However, in the video of the super-natural interaction speech, you can see that the function is powerful, kinect speech recognition not only recognizes sound, but also recognizes the Source Direction of sound. It also has powerful noise reduction and enhancement functions. You can read this article to learn more about this, this section briefly introduces the speech recognition function of Kinect.

Iv. Kinect data sources and Applications

There are two types of data sources involved in Kinect development:

One is the raw data sources generated by sensors, such as color image data sources, deep image data sources, voice data sources, infrared image data sources, and accelerator data sources, the accelerator data source is added to the 1.6 SDK. The following figure shows the various data sources:

The second type of data source is the data sources that can be directly used for recognition identified by some algorithms in the SDK, including: Bone tracing, speech recognition, and facial recognition data sources.

We can see that the bone tracing data source is implemented through a series of algorithms after the deep image data source, and the speech recognition is implemented through the speech data source through a series of algorithms, facial recognition integrates color images, deep images, and bone tracing as a series of data sources and is implemented through a series of algorithms. These functions are implemented through the SDK.

With these data sources, you can perform various applications:

A typical application is green screen. This function is used in recording TV programs, such as weather forecast programs. It separates tasks from the background, then you can change the background at will. This feature uses image data and deep data. You can refer to this article for a simple example. The second application is to generate a dot matrix cloud, that is, to generate the location information of each point through the deep image data of Kinect and then generate a three-dimensional model. For example:

On the left side of the figure is a three-dimensional modeling image produced through the deep image data of the Kinect.

The third application is the magic mirror function. I think it should be a number of applications similar to the mirror effect or photo booth in iOS, mainly using color image data, facial recognition, and bone recognition. There are also functions such as virtual fitting rooms. The last application scenario is various Nui interactive interfaces.

5. Outlook

In addition to the above typical data sources, we also show the next plan, such as the Kinect fusion data source that further processes the depth of field data.

The following is an example of the Demonstration: in the figure, a teapot is placed on the right table, and three-dimensional modeling is performed on the teapot using the Kinect. After removing the real teapot on the right, you can perform various operations on the virtual teapot, these are the next version or future sdks that can help us or simplify our implementation of these features to prepare the added features.

At the end, I looked forward to the work to be done in the next step:

We can see that the vision of Kinect is to enable computers to see, hear, and better understand people and environments around them.

The version of the Kinect SDK was released very quickly. Since the release of the Kinect sensor for Windows somatosensory instrument and the official version of the Kinect for Windows SDK February this year in 1.0; in 1.5, SDK 1.5 and developer toolkit and debugging tool Kinect studio 1.5 were released, and developer toolkit 1.5.1 and developer toolkit 1.5.2 were released soon; on April 9, October 9, the SDK and developer toolkit of version 1.6 were released, and the official release of the Kinect sensor for Windows somatosensory instruments was announced on the Chinese mainland. Each version adds some new features. I believe that the next version of the SDK will also have new features.

I believe that after reading this speech, we should be able to feel the changes brought about by Kinect. We also believe that there will be more and more examples of developing and applying Kinect to daily life.

Finally, I hope you will be free to go to channel9 to watch Real-person speeches. There are also many interesting keynote speeches on this page. All the presentations can be viewed online and the videos can be downloaded, if you are not sure about whether or not you want to watch the speech, you can check the following online asshole. Of course, all the other slides except the abnormal PPT of more than 900 MB can be viewed online, in this way, you can quickly understand the speech content and whether you are interested in the content.