Kinect for Windows SDK development entry (18): Kinect interaction Interactive Control

In July March this year, the SDK version 1.7 was released. The biggest difference between the SDK version and the previous version was the addition of Kinect interactions and Kinect fusion. Kinect interactions provides some new controls with pose recognition, such as the push-to-press button and grip-to-Pan list control. It also supports interaction between multiple users and two people at the same time, these newly added controls can be easily integrated into applications, greatly simplifying the development and Debugging Processes.

The second feature added in the 1.7 SDK is the Kinect fusion feature mentioned at build 2012 last year. It enables us to quickly create 3D models of objects and use 3D printing and modeling, industrial design and virtual reality.

In the Kinect developer toolkit, the following demos are provided: controlbasic-WPF, Kinect interactions, and Kinect fusion. The last two demos have high requirements on the screen resolution of computers and graphics cards, running Kinect interactions requires a resolution of 1920*1080, while Kinect fusion requires a good GPU for real-time rendering.

Based on the above reasons, This article briefly shows the new controls and interaction methods provided by the Kinect interactions.

I Establish necessary environment

Before creating a project, download the latest 1.7 SDK and developer toolkit from the official website. First open Visual Studio to create a simple WPF desktop application, and then add Microsoft. kinect. DLL, Kinect. toolkit. DLL, Kinect. toolkit. controls. DLL and Kinect. toolkit. interaction. DLL reference. These DLL files are generally under the installation directory, and are under c: \ Program Files \ microsoft sdks \ Kinect \ on my machine.

Use kinectsensorchooserControls initialize KinectSensor

In Microsoft. kinect. toolkit. in the controls namespace, the first control used may be kinectsensorchooserui, which is used to indicate the working status of the current Kinect, prompting the user whether the Kinect sensor works normally, for example, whether it is disconnected, whether the wrong USB interface is inserted or not.

To add this control, first Add the following namespace in the main form:

xmlns:k="http://schemas.microsoft.com/kinect/2013"

Then, add the kinectsensorchooserui control to the main form. The Code is as follows:

<Grid>    <k:KinectSensorChooserUI HorizontalAlignment="Center" VerticalAlignment="Top" Name="sensorChooserUi" /></Grid>

In the CS code, we need to initialize the kinectsensorchooser control object:

private KinectSensorChooser sensorChooser;

Then, register the onload event in the constructor of the main form:

public MainWindow(){    InitializeComponent();    Loaded += OnLoaded;}

How to create an onloaded delegate:

private void OnLoaded(object sender, RoutedEventArgs e){    this.sensorChooser = new KinectSensorChooser();    this.sensorChooser.KinectChanged += SensorChooserOnKinectChanged;    this.sensorChooserUi.KinectSensorChooser = this.sensorChooser;    this.sensorChooser.Start();}

The sensorchooseronkinectchanged event will be triggered if the sensor status changes, such as off or Initialization is complete. For ease of demonstration, the message "MessageBox" is displayed:

private void SensorChooserOnKinectChanged(object sender, KinectChangedEventArgs args){    MessageBox.Show(args.NewSensor == null ? "No Kinect" : args.NewSensor.Status.ToString()); }

Run the system now. If you have not connected the Kinect to your computer, you will see:

In this case, sensorchooser. Kinect = NULL. If you move the mouse over this small box, the system will provide more information and further operation prompts:

 

These sdks have been encapsulated for you. It may take some time to write them by yourself. The help connection above will be linked to the k4w official website.

If you connect the USB of the Kinect sensor to your computer, if the power cord of the Kinect sensor is not connected, the following prompt will appear:

Now plug in the power supply. Now the kinectsensorchooserui will switch to the initialization mode, move the mouse over it, and the system will prompt that initialization is in progress:

Wait a moment. After the initialization is complete, you can see the Kinect icon and the status changes to initialization successful. Then, the callback method is executed and the MessageBox dialog box is displayed. After the initialization is complete, the icon disappears from the interface until the state of the Kinect changes again:

This icon is used in many places in the demo of the Kinect developer kit and the Kinect studio. If you want to add the widget to your application with a slight transformation, is it very user-friendly and easy.

The kinectsensorchooserui control will try to give the user a prompt based on different states, switch the USB port, and check whether the power cord of the Kinect is connected.

Create a KinectBasic interaction environment

Now that we have connected to Kinect, we can start some basic interaction settings. Now complete the sensorchooseronkinectchanged Event code:

private void SensorChooserOnKinectChanged(object sender, KinectChangedEventArgs args){    bool error = false;    if (args.OldSensor != null)    {        try        {            args.OldSensor.DepthStream.Range = DepthRange.Default;            args.OldSensor.SkeletonStream.EnableTrackingInNearRange = false;            args.OldSensor.DepthStream.Disable();            args.OldSensor.SkeletonStream.Disable();        }        catch (InvalidOperationException)        {            // KinectSensor might enter an invalid state while enabling/disabling streams or stream features.            // E.g.: sensor might be abruptly unplugged.            error = true;        }    }    if (args.NewSensor != null)    {        try        {            args.NewSensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30);            args.NewSensor.SkeletonStream.Enable();            try            {                args.NewSensor.DepthStream.Range = DepthRange.Near;                args.NewSensor.SkeletonStream.EnableTrackingInNearRange = true;                args.NewSensor.SkeletonStream.TrackingMode = SkeletonTrackingMode.Seated;            }            catch (InvalidOperationException)            {                // Non Kinect for Windows devices do not support Near mode, so reset back to default mode.                args.NewSensor.DepthStream.Range = DepthRange.Default;                args.NewSensor.SkeletonStream.EnableTrackingInNearRange = false;            }        }        catch (InvalidOperationException)        {            error = true;            // KinectSensor might enter an invalid state while enabling/disabling streams or stream features.            // E.g.: sensor might be abruptly unplugged.        }    }}

The above code can deal with the old Kinect sensor unplugging, connecting to the new sensor and enabling the deep and skeleton data stream scenario. In development, we may connect multiple sensors at the same time. In try code, we try to enable close-to-scene mode. If the Xbox sensor is used by the user, it is not supported and then switched to normal mode. In general, enable close-to-scene mode to facilitate debugging and close-to-use.

2. Kinect Interactive controls

Kinect RegionWidget

To use the Kinect interaction, you must add a kinectregion container class on the interface. Kinectregion is a key element in WPF that uses the Kinect interaction for interaction. It is a container for other Kinect interaction controls. Kinectregion is also responsible for displaying and moving hand icons, that is, displaying users' hand bones on the interface. There can be multiple kinectregion on the main interface of the application, but each kinectregion cannot be nested. Each kinectregion can have its own Kinect sensor object.

Now, add a kinectregion object to the main form:

<k:KinectRegion Name="kinectRegion"></k:KinectRegion>

Then, set the kinectsensor attribute of kinectregion to the kinectsensor we obtained:

if (!error)    kinectRegion.KinectSensor = args.NewSensor;

If you run the program, an error is prompted:

This error is common and kinectinteraction170_32.dll cannot be found. Where can I find it? Find the installation path of the k4w SDK and copy the 32-bit or 64-bit DLL to the output path of the project based on the current operating system version.

 

Running the program again may require 10-20 s of events for initialization and recognition. If everything is normal, the hand-shaped cursor will be displayed on the application:

Here is a tip for demonstration, because the demo must be a certain distance from the Kinect, so that the keyboard's printscreen key cannot be reached, therefore, we recommend that you use Kinect studio to record the data obtained by Kinect, and then play the data to debug the Kinect application. This greatly improves the debugging efficiency, you do not have to stand in front of the Kinect every time. I first recorded the action and then made screenshots or screenshots to make animations, which is much more convenient.

Kinectuserviewer Control

Sometimes in applications, we want to be able to display users' own images in order to give some feedback in the Kinect. toolkit. there is a kinectuserviewer control in controls. You can use this control to display the depth image of a character. to use it, you need to define a kinectuserviewer object on the main interface:

<k:KinectUserViewer VerticalAlignment="Top" HorizontalAlignment="Center"  k:KinectRegion.KinectRegion="{Binding ElementName=kinectRegion}" Height="100" />

Kinectuserviewer displays the deep image data of a person in the Kinect. If you cannot see the image, it indicates that the current Kinect has not tracked you. In this case, you should try to move it to let the Kinect track you. Run the program and use the data just recorded by Kinect studio. Then, you can see your own deep image of the person in the field of view.

The previous control is used to show the current state, hand-shaped location, and deep image of a person in the Kinect. All of these can give you some global feedback, indicating that the current system is running normally, now we need to use some basic controls to truly interact with applications.

Kinecttilebutton

The first thing to introduce is the kinecttilebutton control, which is also the simplest one. It is a bit like the metro patch in Win8. It is actually a common button control, however, we only use the Kinect to simulate the mouse behavior by hand. Now we add a kinecttilebutton to play on the interface.

<k:KinectTileButton Label="Press me!" Click="ButtonOnClick"></k:KinectTileButton>

Then, write some tips in the click event:

private void ButtonOnClick(object sender, RoutedEventArgs e){    MessageBox.Show("You clicked me!");}

When our hands do not touch the control, it is blue by default. When the hand moves to the button, the button will become larger, and a shadow will appear around the hand icon to display the current state of being suspended. Then we push the palm in the direction of the Kinect, and we will find that the hand icon will generate an animation based on the distance from the Kinect, indicating that we are generating a "Press" button action. When the depth threshold is reached, it indicates that the user has pressed the button, the hand icon will change color, and the click event will be triggered.

 

Kinectcirclebutton

Kinectcirclebutton is similar to kinecttilebutton, except that the button is in a circular shape, which can be seen from the name. To add kinectcirclebutton to the existing interface layout, we need to adjust it slightly. Kinectregion is a content control, which indicates that there can only be one child control. However, this child control can be a control that contains other containers, such as grid or stackpanel. Now we need to include the previous kinecttilebutton into a grid control, move the kinecttilebutton to the left, and then add a kinectcirclebutton control to the right.

<k:KinectCircleButton Label="Circle" HorizontalAlignment="Right" Height="200" VerticalAlignment="Top" Click="ButtonOnClick" >Hi</k:KinectCircleButton>

Repeat the above actions to see similar animation effects:

It should be noted that the two controls can also be operated with the mouse.

Kinectscrollviewer

Before using the k4w 1.7 SDK, it is difficult to use the Kinect browsing list and select it. However, the kinectscrollviewer control released in the latest 1.7 SDK greatly simplifies this operation. Now, we place a kinectscrollviewer at the bottom of the form. Add the following code to the grid in kinectregion:

<k:KinectScrollViewer VerticalScrollBarVisibility="Disabled" HorizontalScrollBarVisibility="Auto" VerticalAlignment="Bottom">    <StackPanel Orientation="Horizontal" Name="scrollContent" /></k:KinectScrollViewer>

This control is like the standard scrollviewer in WPF. There is a stackpanell used to place the scrolling content. These two controls can be combined to make the content scroll horizontally. Now some kinectcirclebuttons are placed in the stackpanel.

for (int i = 1; i < 20; i++){    var button = new KinectCircleButton    {        Content = i,        Height = 200    };    int i1 = i;    button.Click +=        (o, args) => MessageBox.Show("You clicked button #" + i1);    scrollContent.Children.Add(button);}

Run the program now, as shown in the animation. At the beginning, you can see the kinectcirclebutton below. When your hand moves to the rolling area, the color changes. Then, close your finger and hold your fist, the gesture icon changes, indicating that you are currently capturing the kinectscrollviewer, moving your fist, and the kinectscrollviewer control will paste the gesture icon to the control, just as our fingers operate on the touch screen, then the button will move with the gesture. After positioning, open your fist and stop moving. At this time, press your hand forward toward the desired button, the click event will be triggered like the previous control.

 

To put it off the question, the above animation recording is a little large, with more than 5 MB, which leads to the following error 500 Error always prompted when I publish it to the blog Garden in Windows Live write:

Some of them say that the picture is too big, so I downloaded a software for editing GIF animation named Ulead GIF animation or, deleted some repeated frames, and reduced the size. This error does not occur when you release the product again.

3. Conclusion

This article briefly introduces the functions and related controls of the Kinect interactions introduced in the Kinect for Windows 1.7 SDK, these controls with the Kinect property greatly simplify the difficulty of adding the Kinect feature to the application, allowing us to create a cool Kinect application faster and better through the Kinect. Click here to download the code in this article. Due to the size relationship, the two DLL kinectinteraction170_32.dll or kinectinteraction170_64.dll are not in the code. You can install the SDK and copy it to the corresponding directory to run it, I hope this article will help you understand the control related to Kinect interactions in SDK 1.7.