[Translation] Getting started with the development of the Kinect for Windows SDK (2): Basic knowledge

Document directory

• 1.1 discover connected Kinect Devices
• 1.2 open Sensor
• 1.3 Stop Sensor

The previous article introduced the environment configuration for the development of Kinect. This article and the next article will introduce the basic knowledge of the Development of Kinect, and lay a solid foundation for the in-depth study of the Windows SDK.

Each Kinect application has some basic elements. The application must detect and discover links to the Kinect sensor on the device. Before using these sensors, Initialization is required. Once the initialization is successful, data can be generated and our program can process the data. Finally, when the application is closed, these sensors must be properly released.

The first part of this article will introduce how to test how to initialize several release sensors, which is a very basic topic, but is very important for applications developed based on Kinect. After initialization, various sensors of the Kinect system can generate data. Our program can read these data streams. The Data Stream Generated by Kinect is similar to the IO Data Stream under the system. Io namespace.

The second part describes the basics of data streams in detail and shows how to use colorimagestream from the Kinect to obtain the data produced by the color camera. Data streams can generate pixel-based data so that color images can be produced from cameras or basic photos. This data can be processed in a variety of interesting ways.

This article is the basic part of the whole development of the Kinect SDK. After learning about this, it is helpful to be familiar with other parts of the SDK.

 

1. Kinect sensor

The kinectsensor object is the first object to be used for applications developed based on Kinect. This object directly represents the hardware device of Kinect. The kinectsensor object is the source of the data we want to obtain, including color image data, depth of field data, and bone tracing data. This article will introduce colorimagestream in detail, and the subsequent articles will discuss depthimagestream and skeletonstream in detail.

The most common way to get data from kinectsensor is to listen to a series of events of this object. Each data stream has a corresponding event. When the data stream type is available, the change time is triggered. Each data stream is in frame units. For example, colorimagestream triggers the colorframeready event when new data is obtained. When discussing specific sensor data streams, we will discuss these events in detail.

Each data stream (color, depth, skeleton) is displayed in different coordinate systems in the form of data points. We can clearly see this in the subsequent discussions. It is a common operation to convert the point data in a data stream to another data stream. Later in this article, we will discuss how to convert and why this conversion is necessary. Kinectsensor objects have some column-based methods to convert data streams to data lattice. They are mapdepthtocolorimagepoint, mapdepthtoskeletonpoint, and mapskeletonpointtodepth. Before obtaining the Kinect data, we must first find the connected Kinect device. It is easy to find the Kinect device, but you also need to pay attention to it.

 

1.1 discover connected Kinect Devices

The kinectobject object does not have a public constructor and cannot be directly created by an application. Instead, this object is created when the SDK detects a connected Kinect device. When a Kinect device is connected to a computer, the application should be notified or reminded. The kinectseneor object has a static attribute kinectsensors, which is a kinectsensorcollection set that inherits from readonlycollection. The readonlycollection set is very simple. It has only one indexer and one event called statuschanged.

Use the indexer in the set to obtain the kinectsensor object. The number of elements in the set is the number of Kinect devices. That is to say, a computer can connect multiple Kinect devices to obtain data from different directions. Applications can use multiple Kinect devices to obtain multi-faceted data. The number of Kinect devices is limited only by the computer configuration. Since each Kinect device transmits data via USB, a USB cable is required for each Kinect device to connect to the computer. In addition, more Kinect devices require more CPU and memory consumption.

You can use a simple traversal set to find the Kinect device. However, none of the devices in the kinectsensor set can be directly used. Therefore, the kinectsensor object has a status attribute, which is an enumeration type, identifies the status of the current Kinect device. The following table lists the sensor statuses and their meanings:

The kinectsensor object can be initialized only when the device is in the connected state. The sensor status may change throughout the application lifecycle, which means that the application we develop must monitor the connection status of the device, in addition, you can take appropriate measures to improve the user experience when the device connection status changes. For example, if the USB cable connecting to the Kinect is pulled out from the computer, the sensor connection status changes to disconnected. In general, the application should be suspended in this case, the user is prompted to insert the Kinect device to the computer. The application should not assume that the Kinect device is available at the beginning, nor that it will be connected to the computer throughout the entire program.

Next, we will first create a WPF Application to demonstrate how to discover and obtain the state of the Kinect sensor. First, create a WPF project and add Microsoft. Kinect. dll. Write the following code in mainwindows. XAML. CS:

Public partial class mainwindow: window {// Private kinectsensor object private kinectsensor Kinect; Public kinectsensor Kinect {get {return this. kinect;} set {// if the sensor with a value assignment is different from the current one if (this. kinect! = Value) {// if the current sensing object is not null if (this. Kinect! = NULL) {// uninitailize current object this. Kinect = NULL;} // If the input object is not empty and the status is connection status if (value! = NULL & value. status = kinectstatus. connected) {This. kinect = value ;}}} public mainwindow () {initializecomponent (); this. loaded + = (S, e) => discoverkinectsensor (); this. unloaded + = (S, e) => This. kinect = NULL;} private void discoverkinectsensor () {kinectsensor. kinectsensors. statuschanged + = kinectsensors_statuschanged; this. kinect = kinectsensor. kinectsensors. firstordefault (x => X. status = kinectstatus. connected);} private void kinectsensors_statuschanged (Object sender, statuschangedeventargs e) {Switch (E. status) {Case kinectstatus. connected: If (this. kinect = NULL) This. kinect = E. sensor; break; Case kinectstatus. disconnected: If (this. kinect = E. sensor) {This. kinect = NULL; this. kinect = kinectsensor. kinectsensors. firstordefault (x => X. status = kinectstatus. connected); If (this. kinect = NULL) {// todo: the notification is used for Kinect unplugging} break; // todo: processing status in other cases }}}

 

The code comment above is very detailed. First, we define a private variable Kinect. The application should define a private variable to store references to the obtained kincectsensor object, when the application does not need kinectsensor to generate data, you can use this local variable to release the reference to the kinectsensor object to release resources. We also define a Kinect attribute to wrap this private variable. The attribute is used to ensure that the kinectsensor object can be initialized and reversed in the correct way. In the set method, we can see that the value assignment is performed only when the group flight of the objects to be assigned is too connected, an invalidoperationexception is thrown when any sensor object that is not in the connected state is copied to the kinectsensor object.

There are two anonymous methods in the constructor, one for listening to loaded events and the other for listening to unloaded events. When uninstalling the tool, you should leave the Kinect attribute empty. In the loaded event of the window, the program tries to call a connected sensor through the discoverkinectsensor method. Register these two events in the form loaded and unloaded events to initialize and release the Kinect object. If the application does not find the Kinect object, the user will be notified.

The discoverkinectsensor method has only two lines of code. The first line of code registers the statuschanged event, and the second line of code queries the first sensor object in the connected state in the set through lambda expressions, and copy the object to the Kinect property. The Set Method of the Kinect attribute ensures that all values can be assigned a valid Kinect object.

In the statuschanged event, it is worth noting that when the status is kinectsensor. when connected, the IF statement limits the application to have only one Kinect sensor, and ignores other Kinect sensors that may be connected to the computer.

The code above demonstrates the most streamlined code used to discover and reference the Kinect device. With the complexity of applications, more code may be required to ensure thread security and allow the Garbage Collector to release resources in time to prevent memory leakage.

 

1.2 open Sensor

Once a sensor is discovered, it must be initialized before the application can use the sensor. Sensor initialization involves three steps. First, the application must set the data stream to be used and set its status to available. Each data stream type has an enable method, which can initialize the data stream. Each data stream is completely different. You need to set columns before using them. In some cases, these settings are processed in the Enable method. In the following sections, we will discuss how to initialize the colorimagestream data stream. In future articles, we will also discuss how to initialize the depthimagestream data stream and skeletonstream data stream.

After initialization, the next step is to determine how the application uses the generated data stream. The most common method is to use some column events of the Kinect object. Each data stream has a corresponding event. They are: colorimagestream corresponds to the colorframeready event, depthimagestream corresponds to the depthframeready event, and skeletonstream object corresponds to the skeletonframeready event. And the allframesready event. The corresponding events can only be used after the corresponding data stream is enabled. The allframesready event can be used when any data stream is enabled.

Finally, after the application calls the start method of the kinectsensor object, the frame-ready event is triggered to generate data.

 

1.3 Stop Sensor

Once the sensor is turned on, you can use the stop method of the kinectsensor object to stop. In this way, all data generation will stop. Therefore, when listening to The frameready event, you must first check whether the sensor is not null.

Kinectsensor objects and data streams all use system resources. Applications must be able to release these resources reasonably when they do not need to use kinectsensor objects. In this case, the program will not only stop the flyer, but also log off the frameready event. Note: Do not call the dispose method of the kinectsensor object. This will prevent the application from getting the sensor again. The application must start or unplug the Kinect and then insert it to obtain and use the object again.

 

2. Color Image Data Stream

Kinect has two types of cameras: Near-infrared cameras and common video cameras. A video camera provides a color image similar to a general camera. This type of data stream is the easiest to use and set in the data stream of the third party. So I will introduce him as an example of the Kinect data stream.

There are also three data streams Using Kinect. First, data streams must be available. Once the data stream is available, the application can read data from the data volume and process and display the data. Once a new data frame is available, the two steps will continue. The following code shows how to initialize the colorimage object.

Public kinectsensor Kinect {get {return this. Kinect;} set {// if the sensor with a value assignment is different from the current one if (this. Kinect! = Value) {// if the current sensing object is not null if (this. Kinect! = NULL ){Uninitializekinectsensor (this. Kinect );// Uninitailize current object this. Kinect = NULL;} // If the input object is not empty and the status is connection status if (value! = NULL & value. Status = kinectstatus. Connected) {This. Kinect = value;Initializekinectsensor (this. Kinect );}}} Private void initializekinectsensor (kinectsensor) {If (kinectsensor! = NULL) {kinectsensor. colorstream. enable (); kinectsensor. colorframeready + = new eventhandler <colorimageframereadyeventargs> (kinectsensor_colorframeready); kinectsensor. start () ;}} private void uninitializekinectsensor (kinectsensor) {If (kinectsensor! = NULL) {kinectsensor. Stop (); kinectsensor. colorframeready-= new eventhandler <colorimageframereadyeventargs> (kinectsensor_colorframeready );}}

The above code modifies the previous Kinect attribute, bold as the modified part. Two methods are called to initialize and release the kinectsensor and colorimagestream objects. The initializekinectsensor object calls the Enable method of colorimagestream, registers the colorframeready event, and calls the start method. Once the sensor is enabled, the frameready event will be triggered when the new data frame avenue is enabled. The trigger frequency of this event is 30 times per second.

Before implementing the kinect_colorframeready method, we first add some space in the XAML form to display the obtained data. The Code is as follows:

<Window x:Class="KinectApplicationFoundation.MainWindow"        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"        Title="ColorImageStreamFromKinect" Height="350" Width="525">    <Grid>        <Image x:Name="ColorImageElement"></Image>    </Grid></Window>

 

Then, in the kinect_colorframeready method, we first extract the frame data by opening or getting a frame. The opencolorimageframe attribute of the colorimageframereadyeventargs object returns the current colorimageframe object. This object implements the idisposable interface. So we can hold this object in the using statement. before extracting pixel data, we need to use a byte array to save the obtained data. The specific size of the data and sequence returned by the pixeldatalength object of the frameobject object. Call the copypixeldatato method to fill in the pixel data and then display the data to the image control. The Code is as follows:

void kinectSensor_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs e){    using (ColorImageFrame frame = e.OpenColorImageFrame())    {        if (frame != null)        {            byte[] pixelData = new byte[frame.PixelDataLength];            frame.CopyPixelDataTo(pixelData);            ColorImageElement.Source = BitmapImage.Create(frame.Width, frame.Height, 96, 96,                                                         PixelFormats.Bgr32, null, pixelData,                                                         frame.Width * frame.BytesPerPixel);        }    }}

 

Run the program to obtain the video information obtained from the Kinect, as shown in figure. This is a photo of my room obtained from the Kinect color camera. This is no different from a normal video, but it is generated from the Kinect video camera.

 

3. Conclusion

This article briefly introduces the basic objects encountered during the development of Kinect, the discovery of physical devices of Kinect, the initialization of kinectsensor objects, the opening of kinectsensor objects, and how to obtain data streams, the colorimagestream object is used as an example to show how to obtain and display data from the Kinect.

Because the color camera of Kinect generates 30 sets of colorimageframe per second by default, the above application will generate 30 bitmap objects, and these objects will soon become garbage waiting for the garbage collector to collect after initialization, when the collected data volume is large, it will affect the performance. Due to space limitations, the next article will introduce how to improve this and discuss two programming modes for obtaining the data produced by the Kinect sensor: Event-based and round-robin. Click here to download the sample code in this article. I hope the above content will help you understand the Kinect SDK.