Control PowerPoint playback Using Kinect

With Kinect, we can use gestures to control the playback of slides without having to give a speech and press the keyboard or hold a PPT controller, all we need to do is to gently roll to the right or to the left to control the slide to forward or backward, so cool. Although it may be a bit strange to do this during the presentation, it is also a good way to control the slides.

It is very easy to implement the control of the slide playback by using the Kinect to capture human movements, and then send an event to the system by clicking forward and backward based on the recognized action, so that the slide can be switched. The core function here is gesture recognition. Before development, we need to define what kind of gesture is to switch the slides forward or backward. Gesture and pose recognition are detailed in articles 9th, 10th, and 11th in my step-by-step development of Kinect. This article only discusses the main ideas and key code parts.

 

1. Implement PPT control through pose Recognition

 

Pose recognition is based on the relative positional relationship between the shut-down node and the shut-down node. It is relatively easy to judge by using a certain frame of bone shut-down node data. Gesture Recognition is complicated by judging the actions in a continuous period of time. However, there is no difference between the two for specific purposes. Just like common algorithms, they are not as complex as possible. Some methods are very simple and efficient.

In the control PPT Playback command, we set that if the distance between the right-hand customs node on the X axis is greater than 0.45 than that of the head customs node, the user tries to click the right button on the keyboard. If the head node is located at a position greater than 0.45 on the X axis than the left-hand node, the user tries to click the left button on the keyboard. 0.45 this value was obtained through repeated tests, which are common in the Development of Kinect. The key code is as follows:

private void ProcessForwardBackGesture(Joint head, Joint rightHand, Joint leftHand){    if (rightHand.Position.X > head.Position.X + 0.45)    {        if (!isBackGestureActive && !isForwardGestureActive)        {            isForwardGestureActive = true;            System.Windows.Forms.SendKeys.SendWait("{Right}");        }    }    else    {        isForwardGestureActive = false;    }    if (leftHand.Position.X < head.Position.X - 0.45)    {        if (!isBackGestureActive && !isForwardGestureActive)        {            isBackGestureActive = true;            System.Windows.Forms.SendKeys.SendWait("{Left}");        }    }    else    {        isBackGestureActive = false;    }}

In the code above, when the user is judged to be waving to the right, the system is executed. windows. forms. sendkeys. sendwait ("{right}") statement to send and click the right button on the keyboard. When this method is executed, the PowerPoint program is in the active state, in this way, the PPT will click the event on the right keyboard. It should be noted that the isbackgestureactive and isforwardgestureactive Boolean bits in the method can prevent users from sending system messages when they are in a certain action. windows. forms. sendkeys. sendwait ("{XX }").

The above method can be put in the sensor_skeletonframeready event. First, you can get the data of the header and the left-hand and right-hand nodes, and then call this method.

void sensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){    using (var skeletonFrame = e.OpenSkeletonFrame())    {        if (skeletonFrame == null)            return;        if (skeletons == null ||            skeletons.Length != skeletonFrame.SkeletonArrayLength)        {            skeletons = new Skeleton[skeletonFrame.SkeletonArrayLength];        }        skeletonFrame.CopySkeletonDataTo(skeletons);        Skeleton closestSkeleton = (from s in skeletons                                    where s.TrackingState == SkeletonTrackingState.Tracked &&                                          s.Joints[JointType.Head].TrackingState == JointTrackingState.Tracked                                    select s).OrderBy(s => s.Joints[JointType.Head].Position.Z)                                            .FirstOrDefault();        if (closestSkeleton == null)            return;        var head = closestSkeleton.Joints[JointType.Head];        var rightHand = closestSkeleton.Joints[JointType.HandRight];        var leftHand = closestSkeleton.Joints[JointType.HandLeft];        if (head.TrackingState != JointTrackingState.Tracked ||            rightHand.TrackingState != JointTrackingState.Tracked ||            leftHand.TrackingState != JointTrackingState.Tracked)        {            //Don't have a good read on the joints so we cannot process gestures            return;        }       ProcessForwardBackGesture(head, rightHand, leftHand);    }}

Slide control through pose recognition is simple and efficient, but there are also two main problems:

First, if a video, Flash, or other multimedia elements are embedded in a slide, you may not be able to properly control the playback and suspension of these elements. One solution is to use animation, this allows you to start playing when you click the keyboard to display multimedia. Another method is to use vsto to write plug-ins for PowerPoint to listen to the mouse to control multimedia playback.

The second problem is misoperations. This problem is the biggest problem in the use of posture-based recognition. Sometimes you may need to open your arms or bend down to pick up something out of your physical language, in this way, the relative relationship between the head and hand nodes may meet the distance we set before, resulting in misoperations.

The first problem of using pose recognition is a common problem. Even if you use a PPT controller, the PPT controller seems to be able to control slides by sending keyboard click events. The second problem can be avoided to some extent by means of gesture recognition.

 

Second, implement PPT control through Gesture Recognition

 

Gesture Recognition identifies a series of consecutive actions in a certain period of time to identify the actions. In the tenth article, we will detail how to identify the swip action. A slideshow gesture-WPF example is added to the Kinect for Windows developer toolkit V 1.5.

 

We can open the source code directly. Unfortunately, the gesture recognition code is encapsulated in a DLL named Microsoft. samples. Kinect. swipegesturerecognizer,

You can use reflector to view the implementation method. The general principle is similar to that in my tenth article. However, it provides a gesture library-based recognition method. I will introduce it in detail later, here we can directly use the method provided by the DLL.

To use the gesture recognition provided by the DLL, you must first create a recognizer object. Then initialize and change the recognizer. During initialization, register the operations performed after the left and right swing recognition. This operation can be provided through the method. Because the method body is small, lambda expressions are used here.

private readonly Recognizer activeRecognizer;this.activeRecognizer = this.CreateRecognizer();private Recognizer CreateRecognizer(){    // Instantiate a recognizer.    var recognizer = new Recognizer();    // swipe right to  press right key .    recognizer.SwipeRightDetected += (s, e) =>    {            System.Windows.Forms.SendKeys.SendWait("{Right}");    };    // swipe left to  press left key ..    recognizer.SwipeLeftDetected += (s, e) =>    {            System.Windows.Forms.SendKeys.SendWait("{Left}");    };    return recognizer;}

Call the sensor_skeletonframeready event.

private Skeleton[] skeletons = new Skeleton[0];private void sensor_SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){    // Get the frame.    using (var frame = e.OpenSkeletonFrame())    {        // Ensure we have a frame.        if (frame != null)        {            // Resize the skeletons array if a new size (normally only on first call).            if (this.skeletons.Length != frame.SkeletonArrayLength)            {                this.skeletons = new Skeleton[frame.SkeletonArrayLength];            }            // Get the skeletons.            frame.CopySkeletonDataTo(this.skeletons);            // Pass skeletons to recognizer.            this.activeRecognizer.Recognize(sender, frame, this.skeletons);        }    }}

Put the above Code in the program, run it, you can see that you use the left hand to wave to the left, use the right hand to wave to the right to control the slide to the left to switch. As described in the previous article, a gesture recognition action has a time threshold and a certain rule. If the action is not completed within a certain time period, the recognition fails. Therefore, in this example, if the waving action is too slow, the recognition may fail. Unlike the previous example of using pose recognition, you only need to maintain an action to complete the recognition operation, which reduces the probability of mistaken recognition in the first case to a certain extent.

The running effect is as follows:

 

Summary

This article introduces two methods for controlling PPT playback by Using Kinect. One is based on Posture Recognition and the other is based on Gesture Recognition, the advantages and disadvantages of the two methods and some code for implementation are introduced. In fact, the use of Kinect can fully implement functions on commonly used PPT controllers. For example, the use of the LED indicator can also achieve this function, that is, to map the position of the user gesture to the screen, it is displayed in red. Of course, for entertainment purposes only, you can also use voice to control the playback of the PPT. I believe that you have read some of the articles I have written and added the code above, so it is very easy to implement the control of Slide playback by Using Kinect. The original code is not provided here. I hope this article will help you understand the posture and Gesture Recognition in Kinect.