Kinect's Learning Notes development (i) Kinect Introduction and application

Kinect's Learning Notes development (i) Kinect Introduction and application

[Email protected]

Http://blog.csdn.net/zouxy09

First, the Kinect brief introduction

Kinectfor Xbox 360, referred to as Kinect, is developed by Microsoft. Peripherals that apply to the Xbox 360 console. It allows the player to use voice commands or gestures to manipulate the Xbox360 system interface without having to hand or trample the controller.

It also captures the player's movements from top to bottom. Play with the body and bring the player "no controller game and entertainment experience".

It was listed in the U.S. on November 4, 2010 and is priced at $149. Kinect sold 8 million in the first 60 days of sales and has now filed for the world record to become the world's fastest-selling consumer electronics.

February 1, 2012, Microsoft officially released the Kinect version number for Windows system "Kinect for Windows", the proposed price of $249.

and later in 2012. Microsoft will also announce a special version of Kinect for "educational users". (The above from the Wiki encyclopedia)

1.1. Hardware

The Kinect has three lenses, and the middle lens is an RGB color camera used to set up color images. The left and right lens are the 3D structure light depth sensor which is composed of infrared emitter and infrared CMOS camera respectively. Used to set depth data (the distance from the object to the camera in the scene). The color camera supports up to 1280*960 resolution imaging, and the IR camera supports 640*480 imaging maximum.

The Kinect is also paired with the Chase technology, and the base motor rotates with the focusing object moving. Kinect also built-in array microphones, four microphones at the same time radio, compared to eliminate noise, and through its collection of voice recognition and acoustic source positioning.

1.2. Software development environment

1.2.1, unofficial combination

When Microsoft launched Kinect on the Xbox 360, there was no development package in Windows. And because of the Kinect's powerful features and relatively low prices, geeks have expressed the hope that they will be able to use it on their computers.

So there are a number of Daniel developed the drive, at the moment I learned that there are three:

1) CL NUI Platform

Developed by Nui's Daniel Alexp (other well-known products he developed include PS3 's Windows drivers). Can be downloaded here , the target platform is Windows7. Can obtain the color camera, the depth sensor and the accelerometer sensor data, the use is simple and convenient.

2) Openkinect/libfreenect

Launched by Hector Martin, the first to hack Kinect, to download here , the target platform is Linux and Mac. It is said that someone successfully ported to Windows. Because a lot of geek are Mac fans, so the development of a lot of participants, not only a data-Driver,geek have written other advanced things, such as I mentioned the skeletal, color camera as a texture on the depth of data, etc., very attractive.

3) Openni

Openni (Opennatural Interface Open Natural interaction) is a multi-lingual, cross-platform framework that defines the authoring application. and take advantage of its naturally interacting APIs. To be able to here download . To infer from the name that its ultimate goal is about the effect of a minority report, some of the demos that have been released at the moment are probably beyond the ultimate goal.

It is not specifically developed for Kinect. But there is support from Kinect's producer, PrimeSense. This feeling is also a relatively more informal combination ofSensorkinect + NITE + openni; Sensorkinect is the Kinect driver. Nite is a middleware provided by PrimeSense, capable of analyzing the data the Kinect reads, outputting human motion, and so on.

1.2.2, Microsoft Official SDK

The Kinect somatosensory game was very well rated on the Xbox 360, but the development on the Windows platform has been using only unofficial solutions (said above). For example, Nkinect with the CL NUI SDK, but Microsoft eventually launched the Kinect forwindows SDK Beta in June 2011. In particular, it can be developed using C # with the. NETFramework 4.0来. The Kinect for Windows SDK is designed primarily for Windows7. Contains drivers, a rich raw data flow development interface, natural user interface, installation files and reference routines. The Kinect for Windows SDK makes it easy for programmers using C + +, C #, or VisualBasic languages with MicrosoftVisualStudio2010 tools. Now the newest is V1.6.

Kinectfor the Windows SDK:

Http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/download.aspx

The Kinect SDK now only supports Windows 7. Divided into x86 and x64 two version numbers.

Development tools also require the support of the. NET Framework 4.0 and Visual Studio 2010 (minimum Express version number).

This is explained later in the configuration of the development environment.

Advantages and disadvantages of 1.2.3, unofficial and official development kits

1) Official SDK:

Strengths:

Provides audio support, adjustable tilt motor, and body tracking for bone tracking: non-standard posture detection (surrender posture relative to Openni ...) ), the head, hands, feet, clavicle detection and joint occlusion details of the treatment more carefully (but the accuracy is not determined).

In addition, multi-sensor support (multiple Kinect);

Disadvantages:

Microsoft's restrictions on non-commercial use. In addition, gesture recognition and tracking are not provided, and the RGB image/depth image is not aligned with each other. Only the alignment of the individual coordinate system is provided. In full-body skeleton tracking, the SDK only calculates the position of the joint and does not derive its rotational angle. From a portable point of view, the SDK beta can only be used for the kinect/win7 platform, while Openni also supports at least ASUS Wavi Xtion somatosensory devices, and there may be many other hardware platforms to support in the future. In contrast, SDK Beta does not support the Unity3d game engine, does not support recording/playback data to disk, does not support the original infrared video data stream, and does not support event response mechanisms like Openni's role entry and exit.

2) Unofficial Openni/nite:

Strengths:

It can be used for commercial development, including gesture recognition and tracking, and can proactively align depth images and RGB images. Full body tracking, joint rotation angle calculation, look good performance, there are many game product applications, support record/playback data to disk, support the original infrared video data stream, support role entry and Exit event response mechanism. Support PrimeSense and ASUS Wavi Xtion hardware platforms and software platforms such as Windows, Linux, and Macs. The code comes with full support for the Unity3d game engine.

Disadvantages:

No audio function, no support for adjusting tilt of the rotating motor, in the whole body tracking bone tracking: Unable to track the head, hands, feet and clavicle rotation, the need for standard posture detection (known as the surrender posture ...) ), there appears to be an algorithmic bug in the processing of details such as joint occlusion.

You cannot proactively install and identify the Kinect multi-machine environment.

Installation process is more cumbersome, especially nite also need to apply for development certificate code. Openni also does not provide an event-triggering mechanism for available video and depth-map inputs (but Openni functions that provide similar functionality can be used.) Although not a callback function. But also very useful).

Summary:

Openni's biggest advantage is agreeing to cross-platform multi-device, as well as commercial applications.

But from the collection of raw data and preprocessing technology. Microsoft's SDK seems to be more stable, and it also provides good bone and voice support.

For part of the body part recognition function, Sdkbeta does not provide local identification and tracking, which requires its own development (at least for a certain period of time Microsoft may not provide such features). Openni/nite Although gesture recognition and tracking are provided. However, there are many other Microsoft products that are used to identify and track the skeletal posture of the whole body.

So, according to the current performance in the community. The SDK beta and openni/nite is not a good or bad to be sure. And as more and more developers add to Microsoft's side, the popularity of the SDK beta may be faster. But in a higher level of application, the choice of the two often requires a certain wisdom.

(This part: http://www.hanyi.name/blog/?p=330)

Second, Kinect application development Summary

This article (Baidu Library above the content, source unknown) summarizes the current use of Kinect SDK for Windows developed a variety of applications, click on the appropriate link to see the corresponding Kinect app demo video.

Virtual applications

The Kinect fitting mirror, a fantastic fitting mirror based on the Kinect somatosensory technology, allows customers to try clothes at high speed, improving sales efficiency and corporate image.

Http://v.youku.com/v_show/id_XMjU4MjExNjgw.html

Kinect homemade Application 3D fitting room

Http://v.youku.com/v_show/id_XMjg0MDA3MDYw.html

3D Modeling

3D camera with two Kinect to achieve the basic effect of 3D camera.

Http://v.youku.com/v_show/id_XMjI3MDQ2MjEy.html

Sculpture tools. Lidengkequ Kinect into the street high-speed portrait sculpture tool, using Kinect to 3D modeling the human body, and then according to the human body's 3D information, connect the corresponding molding equipment, mold the body statue.

Http://v.youku.com/v_show/id_XMjU2Njg2MzE2.html

Mechanical control

Using the Kinect to manipulate the RC helicopter

Http://v.youku.com/v_show/id_XMjgzOTYxODk2.html

Kinect Robo. Use Kinect as the head of the robot to detect the surrounding environment through the Kinect. and conduct 3D modeling to guide the robot's actions.

Http://v.youku.com/v_show/id_XMjIzNDMyMjg0.html

Kinect control up to robots

Http://v.youku.com/v_show/id_XMjM3NDUxMjg4.html

Virtual Instruments

Air guitar, which operates virtual guitar with Kinect gestures to play music.

Http://v.youku.com/v_show/id_XMjI5NjM3NzMy.html

The Kinect plays ancient Chinese instruments and can play the sounds of different ancient Chinese instruments by changing gestures.

Http://v.youku.com/v_show/id_XMjI1MDA4MTM2.html

Virtual Entertainment

Kinect cracked the "Hatsune" and applied the somatosensory control to the comic character-the first tone.

Http://v.youku.com/v_show/id_XMjMxNjQ5MzAw.html

Turn into Ultraman. Capture player skeleton data, virtual for Ultraman's form and its follow-up, and add some special effects.

Http://v.youku.com/v_show/id_XMjM1MTY1NDY4.html

Kinect cracked play lightsaber, Kinect detects the player's movements. The image of a virtual lightsaber, with which it moves.

Http://v.youku.com/v_show/id_XMjI0NTIzOTIw.html

Computer-related applications

Kinect gesture Operation Browser, through the Kinect gesture to the browser to page, drop, zoom and other operations.

Http://v.youku.com/v_show/id_XMjI1MjczMzg4.html

Air Presenter. Make your speech stand out from the crowd and use the Kinect software to speak.

Http://v.youku.com/v_show/id_XMjU1MTMyNzky.html

Kinect multi-touch, using the Kinect to achieve space-separated multi-touch. Browse pictures, maps, and more.

Http://v.youku.com/v_show/id_XMjM1MzE2MTY0.html

The Kinect somatosensory control watch, the four military Medical University of Western Beijing Hospital orthopedic surgeons will crack the Kinect application in the operating room, in surgery. The patient is able to view the patients ' images through somatosensory control.

Great generosity was the doctor's surgery. Reduced movement of people in the operating room.

Http://v.youku.com/v_show/id_XMjM2Nzg2NDky.html

Virtual experiment

Kinect crayon Physics, using Kinect gestures to paint. The picture is drawn by the somatosensory control and has physical properties, such as gravity. Attraction and so on.

Http://v.youku.com/v_show/id_XMjI4MDMwMzM2.html

Control Lightning with Kinect (Tesla coils)

Http://v.youku.com/v_show/id_XMjY5NjE2MzQ0.html

Game

Kinect hack play Max Payne

Http://v.youku.com/v_show/id_XMjM2NjYxNDg0.html

Kinect Hack Play Survival Tour 2 body feel Zombie

Http://v.youku.com/v_show/id_XMjM3Mjk5MjQw.html

Kinect hack play World of Warcraft

Http://v.youku.com/v_show/id_XMjMzMDM1MzI0.html

Kinect Hack play Street Fighter

Http://v.youku.com/v_show/id_XMjM4MjkzNjA0.html

Kinect hack play Super Mario Bros

Http://v.youku.com/v_show/id_XMjI1NzY1OTM2.html

Kinect cracked the Ranger paradox

Http://v.youku.com/v_show/id_XMjM0NDk2MDg4.html

Kinect Hack play Modern War

Http://v.youku.com/v_show/id_XMjM1NjU4NjE2.html

A group of students from Carnegie Mellon used the Kinect interactive feature to achieve 18 interesting ideas of various kinds.

They took only two weeks, ranging from sophomores to graduate students. Let's take a look at the fantastic ideas they've come to realize.

Source and video Information link http://golancourses.net/2011spring/projects/project-3-interaction/

1. Comic Kinect

This demo sample mainly applies Kinect's skeleton tracking technology and player segmented data to show the interaction of boxing and kick through visual comic effects. It also emits some quasi-sound effects synchronously.

2. Mario Boo

When the Kinect sensor detects that someone is out of sight, a ghost remains behind, moving with the movement of the person. It also changes its size depending on the depth of the information.

3. Magrathea

Magrathea uses Kinect to dynamically generate topographic maps based on whatever objects are on the table.

The camera reads the changing depth of the object on the table, revealing a process similar to the gradual evolution of the Earth's topography.

4. We are monsters

Get inspired by the Noon lion dance. Using Kinect's skeleton tracking technology, two of people were sitting on their own limbs and manipulating the limbs and tail of the virtual monster.

5. Mix&match Interaction Toy

Kinect/openni skeleton technology was used. The body of 3 cards makes it possible to move with the player. and change the picture by sliding the hand.

6. Kinect Flock

The author creates a particle system when the user moves. Things like cotton-wool are going to surge, and when the user is smart, they are clustered into the depth area of the participant.

7. Roboscan

The Roboscan is a 3D model + scanner that secures a Kinect device to an ABB 4400 manipulator. The set action and the operator control the 3D position of the robot and the camera at the same time. The Kinect's depth data is used to produce a model of an accurate external environment.

8. Neurospasta

Neurospasta is a free-form game platform that requires full-body input. The participants were able to control their respective Kinect-based puppets. It is also possible to control the avatar of others through function setting.

9. Will-o-the-wisp

This design is full of magical colors and allows the player to control a glowing sphere. The sphere moves according to the movement of the player's hand. To become larger or smaller depending on the depth information.

Ten. Balloon Grab

By checking the palms open or clenched gestures. The author developed a simple mini-game based on simulated balloon flight.

Hand-tracking Visualization

The software uses gestures to control the visualization of the audio, combining the depth of the data detected in the scene with the hand-distance Kinect.

The position of the participant's hand. The speed and other parameters are used to create a visual effect of an interactive sound.

Feel the Kinect the app also occasional word: Kinect the app relies on your imagination!

Kinect's Learning Notes development (i) Kinect Introduction and application