Kinect development tutorial 4: Use a mouse to control the PC version of Fruit Ninja

Recently, the video of connecting to the Xbox to play Fruit Ninja is very popular. Unfortunately, only the local version and the Kinect can be used to play somatosensory games on the Xbox. Fortunately, after searching, Xiao Jin found that the Fruit Ninja has a PC version. Since we have been able to let the Kinect recognize our gestures in the previous tutorial, we use our hands to control the mouse, you can play on the PC!

Video address: http://v.youku.com/v_show/id_xmjk2otu3mjy20.html.

In the previous tutorial, we used raisehand to capture the position of the raised hand. Therefore, Xiao Jin decided to use raisehand to trigger the mouse movement event and click to trigger the mouse clicking, however, the test results are unsatisfactory. The reason why the mouse moves one card and one card is that raisehand recognition takes time and does not meet the real-time standard. What should I do? Xiao Jin read the openni document and found the tracking API. In this way, after we identify the shot, we can use the tracking method to get the real-time location of the hand and solve the problem of moving the mouse! This is like following a person in the vast sea of people is easier than finding someone!

Because the amount of code in this tutorial is a little too large, so I don't have to worry about it. I will first go to the main function and then explain the callback function.

The content of Main. cpp is as follows:

#include <stdlib.h>#include <iostream>#include "opencv/cv.h"#include "opencv/highgui.h"#include <XnCppWrapper.h>#include "KinectGesture.h"#include "Appmessage.h"using namespacestd;using namespacecv;//Generatorxn::GestureGenerator gestureGenerator;xn::HandsGenerator handsGenerator;xn::ImageGenerator imageGenerator;int isRealMouseControl=0;//【1】// main functionvoid main(){    IplImage* drawPadImg=cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,3);    IplImage* cameraImg=cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,3);     cvNamedWindow("Gesture",1);    cvNamedWindow("Camera",1);    clearImg(drawPadImg);    CvFont font;    cvInitFont( &font,CV_FONT_VECTOR0,1, 1, 0, 3, 5);    XnStatus res;    char key=0;    // Context    xn::Context context;    res = context.Init();    xn::ImageMetaData imageMD;    // Generator    res = imageGenerator.Create( context);    res = gestureGenerator.Create( context);    //【2】    res=handsGenerator.Create(context);    // Add gesture    gestureGenerator.AddGesture("Wave", NULL);    gestureGenerator.AddGesture("Click", NULL);    // Register callback functions    XnCallbackHandle gestureCBHandle;    XnCallbackHandle handsCBHandle;    gestureGenerator.RegisterGestureCallbacks(GRecognized, GProgress,(void*)drawPadImg,gestureCBHandle );    //【3】    handsGenerator.RegisterHandCallbacks(Hand_Create, Hand_Update,Hand_Destroy, (void*)drawPadImg, handsCBHandle);    // Start generate    context.StartGeneratingAll();    res = context.WaitAndUpdateAll();     while( (key!=27) && !(res = context.WaitAndUpdateAll())  )    {        res = context.WaitAndUpdateAll();       imageGenerator.GetMetaData(imageMD);       memcpy(cameraImg->imageData,imageMD.Data(),640*480*3);       cvCvtColor(cameraImg,cameraImg,CV_RGB2BGR);       cvPutText(drawPadImg,"Wave Your Hand to Start Tracking",cvPoint(20, 20), &font, CV_RGB(255,0,0));       cvShowImage("Gesture",drawPadImg);       cvShowImage("Camera",cameraImg);       key=cvWaitKey(20);       switch(key){              case 'c':                  clearImg(drawPadImg);                  break;              //【4】              case 'm'://simulate real mouse                  isRealMouseControl=1-isRealMouseControl;                  break;              default:                  if(key != -1) printf("You Press%d\n",key);       }    }    cvDestroyWindow("Gesture");    cvDestroyWindow("Camera");    cvReleaseImage(&drawPadImg);    cvReleaseImage(&cameraImg);    context.StopGeneratingAll();    context.Shutdown();}

[1] after the program is executed, the form and display are the same as the previous Gesture Recognition routine. Because you want to use mouse_event to control the mouse, Xiao Jin chose the MFC framework, it is mainly a dialog and a button. Because we use the highgui of opencv for Image Display during program execution, click the button on the form here to create and start the thread, the thread function kinectgesturemain (), that is, the main function here.

[2] The content in kinectgesturemain () is mostly the same as that in the previous routine. in [2], Xiao Jin created a handsgenerator, this generator is mainly responsible for tracking. Its creation method is the same as that of other generators. Pass a context to the CREATE () method.

[3] similar to gesturegenerator, We need to register a callback function for handsgenerator,

XnStatusxn::HandsGenerator::RegisterHandCallbacks ( HandCreate  CreateCB,  HandUpdate UpdateCB,  HandDestroy  DestroyCB, void *  pCookie,  XnCallbackHandle &  hCallback ) 

Handcreate is called when a new hand (TRACE) is created. handdestroy is called when the hand disappears, and updatecb is called when the hand changes position. In addition, pcookie is a pointer sent to the callback function, which can put some user data. Xiao Jin transfers the artboard image pointer of the program, so that it can be directly drawn in the callback function. Phcallback is the handle of a callback function, which can be used to cancel the callback function.

[4] Click m to enter the mouse control mode.

Other code is similar to the previous routine. Let's look at the callback function.

// callback function for gesture recognizedvoid XN_CALLBACK_TYPEGRecognized( xn::GestureGenerator &generator,                              const XnChar *strGesture,                              const XnPoint3D *pIDPosition,                              const XnPoint3D *pEndPosition,                              void *pCookie ){    int imgStartX=0;    int imgStartY=0;    int imgEndX=0;    int imgEndY=0;    //【5】    imgStartX=(int)(640/2-(pIDPosition->X));    imgStartY=(int)(480/2-(pIDPosition->Y));    imgEndX=(int)(640/2-(pEndPosition->X));    imgEndY=(int)(480/2-(pEndPosition->Y));    IplImage* refimage=(IplImage*)pCookie;    if(strcmp(strGesture,"Wave")==0)    {       cvLine(refimage,cvPoint(imgStartX,imgStartY),cvPoint(imgEndX,imgEndY),CV_RGB(255,255,0),6);       //【6】       handsGenerator.StartTracking(*pEndPosition);    }    else if(strcmp(strGesture,"Click")==0)    {       cvCircle(refimage,cvPoint(imgStartX,imgStartY),6,CV_RGB(0,0,255),12);       //【7】       if(isRealMouseControl)       {           messageHandler(cvPoint(imgStartX,imgStartY),0,REAL_MOUSE_CLICK);       }    }}// callback function forgesture progressvoid XN_CALLBACK_TYPEGProgress( xn::GestureGenerator &generator,                            const XnChar *strGesture,                            const XnPoint3D *pPosition,                            XnFloat fProgress,                            void *pCookie ){}

[5] because of the coordinates of pidposition and pendposition, It is the coordinate system with the center of the screen (0, 0). The display in opencv is the (0, 0) Point in the upper left corner of the screen, so here is a conversion.

[6] in this Code, our gesturegenerator recognized the "waving" gesture and called the starttracking () method of handsgenerator to start the tracker at the pendposition, pendposition is the ending position of the gesture.

[7], if the "Forward push" gesture is identified and the mouse control mode is enabled, the messagehandler () method defined by Xiao Jin is called to simulate Mouse clicking. This messagehandler () will be implemented in appmessage. cpp.

Then, Xiao Jin implemented the hands-related callback function:

//【8】void XN_CALLBACK_TYPEHand_Create(xn::HandsGenerator& generator,XnUserID nId,const XnPoint3D*pPosition, XnFloatfTime, void*pCookie){    addTrackingId(nId);} void XN_CALLBACK_TYPEHand_Update(xn::HandsGenerator& generator,XnUserID nId,const XnPoint3D*pPosition, XnFloatfTime, void*pCookie){    int imgPosX=0;    int imgPosY=0;    char locationinfo[100];    imgPosX=(int)(640/2-(pPosition->X));    imgPosY=(int)(480/2-(pPosition->Y));    IplImage* refimage=(IplImage*)pCookie;    cvSetImageROI(refimage,cvRect(40,450,640,30));    CvFont font;    cvInitFont( &font,CV_FONT_VECTOR0,1, 1, 0, 3, 5);    cvSet(refimage,cvScalar(255,255,255));    if(isRealMouseControl)    {       sprintf(locationinfo,"MouseCtrl: %dth HandLoc: %d,%d",nId,(int)pPosition->X,(int)pPosition->Y);    }    else    {       sprintf(locationinfo,"Normal: %dth HandLoc: %d,%d",nId,(int)pPosition->X,(int)pPosition->Y);    }    cvPutText(refimage,locationinfo ,cvPoint(30,30), &font, CV_RGB(0,0,0));    cvResetImageROI(refimage);    CvPoint thisLocation=cvPoint(imgPosX,imgPosY);    //【9】    if(isRealMouseControl)    {       //cvCircle(refimage,cvPoint(imgPosX,imgPosY),1,CV_RGB(255,0,0),2);       messageHandler(thisLocation,nId,REAL_MOUSE_MOVE);    }    else    {       cvCircle(refimage,cvPoint(imgPosX,imgPosY),1,CV_RGB(255,0,0),2);    }}void XN_CALLBACK_TYPEHand_Destroy(xn::HandsGenerator& generator,XnUserID nId,XnFloat fTime,void* pCookie){    //printf("Lost Hand: %d\n", nId);    removeTrackingId(nId);}

Hand_create () in [8] Is createcb. The callback function is defined as follows:

void XN_CALLBACK_TYPEHand_Create(xn::HandsGenerator& generator,XnUserID nId,const XnPoint3D*pPosition, XnFloatfTime, void*pCookie)

Generator specifies the generator that triggers hands.

NID is the ID of the created hand (TRACE). This ID increases with the creation and disappearance of the hand to identify each hand.

Pposition is the current position of the hand, ftime is a timestamp, and pcookie is the Data Pointer passed in by the user.

In this method, Xiao Jin uses his own addtracking () method, which is also implemented in appmessage. cpp. Xiao Jin maintains an array in the file to identify whether the hand of each ID is in the tracking status.

[9] the parameters of hand_update () and hand_destroy () are the same as those of hand_create (). In addition to a pile of drawing processes, Xiao Jin uses the messagehandler () method to simulate mouse movement.

The content of appmessage. cpp is as follows:

#include "AppMessage.h"//Location and move anglelast time for each userId(Hand Id)CvPoint lastLocation[MAX_HAND_NUM];int isHandTracking[MAX_HAND_NUM]={0};int isClickDown=0;void addTrackingId(int userId){    isHandTracking[userId]=1;}void removeTrackingId(int userId){    isHandTracking[userId]=0;}CvPoint getLastLocation(int userId){    return lastLocation[userId];}void messageHandler(CvPoint &location,int userId,int flag){    //initialize the lastLocation from the location obtained bythe first time    if(lastLocation[userId].x==0&&lastLocation[userId].y==0)    {       lastLocation[userId].x=location.x;       lastLocation[userId].y=location.y;    }       if(flag==REAL_MOUSE_CLICK)       {           if(!isClickDown)           {              mouse_event(MOUSEEVENTF_LEFTDOWN,0,0,0,0);           }           else {              mouse_event(MOUSEEVENTF_LEFTUP,0,0,0,0);                }           isClickDown=1-isClickDown;       }       else if(flag==REAL_MOUSE_MOVE)       {       //【10】           int firstHandId=-1;           for(int i=0;i<MAX_HAND_NUM;i++)           {              if(isHandTracking[i]!=0)              {                  if(firstHandId==-1)                  {                     firstHandId=i;                     break;                  }              }           }           if(abs(location.x-lastLocation[userId].x)<5)           {              location.x=lastLocation[userId].x;           }           if(abs(location.y-lastLocation[userId].y)<5)           {              location.y=lastLocation[userId].y;           }           //【11】           if(userId==firstHandId)           {                                 mouse_event(MOUSEEVENTF_ABSOLUTE|MOUSEEVENTF_MOVE,                  (location.x-160)*65536/640*2,(location.y-120)*65536/480*2,0,0);              }            }       lastLocation[userId].x=location.x;       lastLocation[userId].y=location.y;}

[10] locate the first hand to be tracked based on the ishandtracking array maintained by the program.

[11] considering the possibility that multiple hands may be captured and tracked, let's get started first. Here, let the mouse listen to the command of the first hand and move it.

Mouseeventf_absolute is used here. The so-called absolute coordinate is the coordinate system that defines the computer screen as 65536*65536 points. If the resolution is 640*480 and you want to move the mouse to the center of the screen, you can call mouse_event (mouseeventf_absolute | mouseeventf_move, 320/640*65536,240/480*65536,0, 0 );

In this Code, because the resolution output by openni is 640*480, Xiao Jin is used to raising his right hand to cut fruit, so he only uses the right half side to control the mouse, at the same time, we have made some minor adjustments so that you can set them based on your preferences.

In Fruit Ninja PC, the switch action is triggered by holding the left mouse button (mouse_leftdown) and moving (mouse_move. Therefore, Xiao Jin uses the push gesture in the program to switch between mouse_leftdown and mouse_leftup.

Use the following method: Open the program, press the M key to enter the mouse control mode, enter the game, push the mouse, make the mouse in the mouse_leftdown state, the screen will show a knife, a sword, a shadow, do not want to play, push it again.

Okay, I want to cut a few fruit slices for a break. I hope this article will help and inspire you.

If you are too lazy to build a project with code, you can click here to download the source program in this tutorial.

There is also an MFC version. Click here to download it.

----------------------------------

Author: Chen Jin)

This article is an original article. If you need to repost and quote it, please specify the original author and link. Thank you.