Kinect V2 Development Topic (3) Equipment information acquisition and audio function exploration

 Kinect V2 Development Project (3)

1. Kinect Equipment Information Acquisition

2, Audio function exploration

1. Kinect Equipment Information Acquisition

Since the Kinect supports multiple devices, in theory, we should be able to get all the device information, and here we explore.

Through the official audiobasics-d2d, this demo found that it takes only one step to get the Kinect device:

ikinectsensor* M_pkinectsensor; Getdefaultkinectsensor (&m_pkinectsensor);

We looked in the interface list (https://msdn.microsoft.com/en-us/library/dn791996.aspx) and found a class called Ikinectsensorcollection, It is obvious that the sensor collection of the class, our footsteps start here.

Create a new project, follow the second section, set the properties, and add the header file. Okay, here we go:

Uh, sorry. Why, then? Because the Microsoft of Pit X gave this feature to cut last August, so ... You're not going to be able to use it, you might add it in the future. We know the next, I found a copy of the old code you can see.

    ikinectsensorcollection* pkinectcollection = nullptr;
    ienumkinectsensor* penumkinect = nullptr;
    ikinectsensor* pkinect = nullptr;
    Get Kinect set
    HRESULT hr =:: Getkinectsensorcollection (&pkinectcollection);
    Gets the Kinect enumerator
    if (SUCCEEDED (HR)) {
        hr = Pkinectcollection->get_enumerator (&penumkinect);
    }
    Enumerates the Kinect
    if (SUCCEEDED (HR)) {
        BOOLEAN available = false;
        while (true) {
            //Get Next
            if (succeeded Penumkinect->getnext (&pkinect)) {
                //Judge validity
                Pkinect->get_isavailable (&available);
                if (Available && Yourjudgmentfunc (Pkinect)) {break
                    ;
                }
                Saferelease (Pkinect);
            }
            else break
                ;
        }
    }
    Saferelease (penumkinect);
    Saferelease (pkinectcollection);

Do not care about these details, this is a small matter. We can pass Getdefaultkinectsensor (_com_outptr_ ikinectsensor** defaultkinectsensor); This function gets the default Kinect device, hahaha.

The following is a Test applet:

#include "stdafx.h"
#include "kinect.h"

int _tmain (int argc, _tchar* argv[])
{
	printf ("Hello, Wellcome to Kinect world!\n ");
	Ikinectsensorcollection AA;
	ikinectsensor* BB;
	HRESULT hr = Getdefaultkinectsensor (&BB);
	if (FAILED (HR))
	{
		printf ("No Kinect connect to your pc!\n");
		Goto endstop;
	}
	BOOLEAN bavaliable = 0;
	Bb->get_isavailable (&bavaliable);
	printf ("Bavaliable:%d\n", bavaliable);
	BOOLEAN bisopen = 0;
	Bb->get_isopen (&bisopen);
	printf ("Bisopen:%d\n", bisopen);
	DWORD dwcapability = 0; 
	Bb->get_kinectcapabilities (&dwcapability);
	printf ("Dwcapability:%d\n", dwcapability);
	TCHAR bbuid[256] = {0};
	Bb->get_uniquekinectid (256, bbuid);
	printf ("UID:%s\n", bbuid);


Endstop:
	System ("pause");
	return 0;
}

The result of not having the Kinect plugged in:

Here it is easy to find a strange problem, get the default Kinect device success, but this device is not available, what information can not be obtained, it seems getdefaultkinectsensor very unreliable ah, need to pass the isavaliable judge to use.

Next, we plug in the Kinect device to continue testing:

Dizzy Ah, dumbfounded, why would such a son? I debug AH debugging, and then take the SDK in the demo debugging, and even add the Bb->open () debugging, or that look. After a variety of toss, finally found the Kinect had to say the secret AH.

The Kinect defaults to shutdown, and you have to open it to determine whether it is valid, and so on. But, the Kinect's open takes time, and it takes about 3S on my computer, or even if isopen is 1, avaliable may still be 0.

and continue to study the Kinect header file found that it uses RPC technology, which causes the function's state to be restored as fast as the local machine code, so you need a certain amount of time to wait for the Kinect to update its state.

The new code is as follows:

#include "stdafx.h" #include "kinect.h" int _tmain (int argc, _tchar* argv[]) {printf ("Hello, Wellcome to Kinect world!\
	n "); ikinectsensor* BB; Apply for a sensor pointer HRESULT hr = getdefaultkinectsensor (&BB);
		Gets a default sensor if (FAILED (HR)) {printf ("No Kinect connect to your pc!\n");
	Goto Endstop;
	BOOLEAN bisopen = 0; Bb->get_isopen (&bisopen);

	Check to see if printf has been turned on ("Bisopen:%d\n", Bisopen);
		if (!bisopen)//Not open, attempt to open {hr = Bb->open ();
			if (FAILED (HR)) {printf ("Kinect Open failed!\n");
		Goto Endstop; printf ("Kinect opened!
		But it need sometime to work!\n "); There must be more waiting here, otherwise the following judgment is wrong printf ("Wait for 3000 ms ...").
		\ n ");
	Sleep (3000);
	} bisopen = 0; Bb->get_isopen (&bisopen);
	Whether printf has been turned on ("Bisopen:%d\n", Bisopen);
	BOOLEAN bavaliable = 0; Bb->get_isavailable (&bavaliable);

	Whether printf is available ("Bavaliable:%d\n", bavaliable); 
	DWORD dwcapability = 0; Bb->get_kinectcapabilities (&dwcapability); Get capacity printf ("DwCApability:%d\n ", dwcapability);
	TCHAR bbuid[256] = {0}; Bb->get_uniquekinectid (256, BBUID);

	Get Unique ID printf ("UID:%s\n", bbuid);
Bb->close ();
	Endstop:system ("pause");
return 0; }

Then the Kinect results are as follows:

Haha, complete the task successfully. However, the actual situation should be polling to check the Kinect state, should not use sleep () hastily replaced.

2, the Kinect audio function exploration

Since the Kinect audio is a microphone matrix, it is possible to detect the direction of the sound, which is a more gratifying feature. Of course, you can also use it with Microsoft's speech recognition engine for speech recognition and control, this is a relatively large topic, there is no discussion.

Let's start by making a sound source judgment. Next, get the default Kinect device via Getdefaultkinectsensor, then open, then get the audio source through its member function Get_audiosource, and then according to the member function of the audio source get_ Audiobeams gets the beam list, and then Openaudiobeam gets the first beam, why is the first one. Because now only the first one is supported. Finally, the member function of beam beam get_beamangle get the angle, get_beamangleconfidence obtain the corresponding credibility.

#include "stdafx.h" #include "kinect.h" int _tmain (int argc, _tchar* argv[]) {printf ("Hello, Wellcome to Kinect world!\
	n "); ikinectsensor* BB; Apply for a sensor pointer HRESULT hr = getdefaultkinectsensor (&BB);
		Gets a default sensor if (FAILED (HR)) {printf ("No Kinect connect to your pc!\n");
	Goto Endstop;
	BOOLEAN bisopen = 0; Bb->get_isopen (&bisopen);

	Check to see if printf has been turned on ("Bisopen:%d\n", Bisopen);
		if (!bisopen)//Not open, attempt to open {hr = Bb->open ();
			if (FAILED (HR)) {printf ("Kinect Open failed!\n");
		Goto Endstop; printf ("Kinect opened!
		But it need sometime to work!\n "); There must be more waiting here, otherwise the following judgment is wrong printf ("Wait for 3000 ms ...").
		\ n ");
	Sleep (3000);
	} bisopen = 0; Bb->get_isopen (&bisopen);
	Whether printf has been turned on ("Bisopen:%d\n", Bisopen);
	BOOLEAN bavaliable = 0; Bb->get_isavailable (&bavaliable);

	Whether printf is available ("Bavaliable:%d\n", bavaliable); 
	DWORD dwcapability = 0; Bb->get_kinectcapabilities (&dwcapability); Get capacity printf ("DwCApability:%d\n ", dwcapability);
	TCHAR bbuid[256] = {0}; Bb->get_uniquekinectid (256, BBUID);

	Get Unique ID printf ("UID:%s\n", bbuid);
	Audio data acquisition iaudiosource* audios = nullptr;
	UINT naudiocount = 0;
	hr = Bb->get_audiosource (&audios);
		if (FAILED (HR)) {printf ("Audio Source get failed!\n");
	Goto Endclose;
	} iaudiobeam* AUDIOBM = nullptr;
	iaudiobeamlist* audiobml = nullptr;
	Audios->get_audiobeams (&AUDIOBML); Audiobml->openaudiobeam (0, &AUDIOBM);
	Currently only supports the first float fangle = 0.0f;
	float fangleconfidence = 0.0f;
		while (true) {fangle = 0.0f;
		Fangleconfidence = 0.0f; Audiobm->get_beamangle (&fangle); Gets the audio angle, [ -0.872665f, 0.8726665f] audiobm->get_beamangleconfidence (&fangleconfidence);
		Gets the credibility of the audio (0-1) printf ("Angle:%3.2f (%1.2f) \ n", (fangle/3.1415926f) *180.0f, fangleconfidence);
	Sleep (200);
} endclose:bb->close ();
	Endstop:system ("pause");
return 0;
 }

The following is the acquisition of audio data.

We can get the audio stream as usual, please note that the audio stream obtained from here is the raw data: the multichannel audio from the microphone array, and the microphone array is not used for noise reduction. The code can look at an example of an SDK that gets the raw data, because it's cumbersome to deal with a general-purpose device, not to be explained here.

This is the use of a self-contained method to obtain processed audio data. The processed data information is as follows:

L Code: 32-bit standard floating-point (IEEE float)

L Channel: 1

L Sampling Rate: 16000Hz

There are two ways to get processed audio streams in the SDK, one for audio frames, and similar for previous frames:

Get audio Source (Audiosource)  
    if (SUCCEEDED (HR)) {  
        hr = M_pkinect->get_audiosource (&paudiosource);  
    }  
    Then get the audio frame Reader  
    if (SUCCEEDED (HR)) {  
        hr = Paudiosource->openreader (&m_paudiobeamframereader);  
    }
    Registers the Pro Frame event  
    if (SUCCEEDED (HR)) {  
        m_paudiobeamframereader->subscribeframearrived (&m_ haudiobeamframearrived);  
    }  

This initializes. After use, like before, according to the event to get Audiobeamframearrivedeventargs, and then get audiobeamframereference audio frame reference, and then get audiobeamframelist audio frame list, At present, the list has only one element to get Audiobeamframe audio frames directly. The audio frame may contain complex audiobeamsubframe audio frames (for example, I have 2 here) to get real information about the audio stream.

There is IStream, the front of this is not referring to the C + + standard library input stream, but the COM component "flow interface", readable and writable. The initialization code is as follows:

if (SUCCEEDED (HR))
    {
        hr = M_pkinectsensor->get_audiosource (&paudiosource);
    }
    if (SUCCEEDED (HR))
    {
        hr = Paudiosource->get_audiobeams (&paudiobeamlist);
    }
    if (SUCCEEDED (HR))
    {
        hr = Paudiobeamlist->openaudiobeam (0, &m_paudiobeam);
    }
    if (SUCCEEDED (HR))
    {        
        hr = M_paudiobeam->openinputstream (&m_paudiostream);
    }

M_paudiosteam is the Steam object, when used Isteam::read (void*, ULONG, ulong*) to actively obtain audio data. In contrast, the use of audio frames can be both active and the use of event mechanisms, and stream can only be actively acquired.

Here we use the second stream of the way to write a demo, the code is as follows:

#include "stdafx.h" #include "kinect.h" #define _use_math_defines #include <math.h> int _tmain (int argc, _tchar* ar
	Gv[]) {printf ("Hello, Wellcome to Kinect world!\n"); ikinectsensor* BB; Apply for a sensor pointer HRESULT hr = getdefaultkinectsensor (&AMP;BB);
		Gets a default sensor if (FAILED (HR)) {printf ("No Kinect connect to your pc!\n");
	Goto Endstop;
	BOOLEAN bisopen = 0; Bb->get_isopen (&bisopen);

	Check to see if printf has been turned on ("Bisopen:%d\n", Bisopen);
		if (!bisopen)//Not open, attempt to open {hr = Bb->open ();
			if (FAILED (HR)) {printf ("Kinect Open failed!\n");
		Goto Endstop; printf ("Kinect opened!
		But it need sometime to work!\n "); There must be more waiting here, otherwise the following judgment is wrong printf ("Wait for 3000 ms ...").
		\ n ");
	Sleep (3000);
	} bisopen = 0; Bb->get_isopen (&bisopen);
	Whether printf has been turned on ("Bisopen:%d\n", Bisopen);
	BOOLEAN bavaliable = 0; Bb->get_isavailable (&bavaliable);

	Whether printf is available ("Bavaliable:%d\n", bavaliable); 
	DWORD dwcapability = 0; Bb->get_kinectcapAbilities (&dwcapability);
	Get volume printf ("Dwcapability:%d\n", dwcapability);
	TCHAR bbuid[256] = {0}; Bb->get_uniquekinectid (256, BBUID);

	Get Unique ID printf ("UID:%s\n", bbuid);
	Audio data acquisition iaudiosource* audios = nullptr;
	UINT naudiocount = 0;
	hr = Bb->get_audiosource (&audios);
		if (FAILED (HR)) {printf ("Audio Source get failed!\n");
	Goto Endclose;
	} iaudiobeam* AUDIOBM = nullptr;
	iaudiobeamlist* audiobml = nullptr;
	Audios->get_audiobeams (&AMP;AUDIOBML); Audiobml->openaudiobeam (0, &AMP;AUDIOBM);
	Only the first istream* STM = nullptr is currently supported;
	Audiobm->openinputstream (&stm);
	Audios->release ();

	Audiobm->release ();
	float fangle = 0.0f;
	float fangleconfidence = 0.0f;
	ULONG lread = 0;
	Const ULONG lbuffersize = 3200;
	float* Fdataarr = new Float[lbuffersize];
		while (true) {fangle = 0.0f;
		Fangleconfidence = 0.0f; Audiobm->get_beamangle (&fangle); Gets the audio angle, [ -0.872665f, 0.8726665f] Audiobm->get_beamangleconfidence (&fanglEconfidence); Gets the credibility of the audio (0-1) if (Fangleconfidence > 0.5f) printf ("Angle:%3.2f (%1.2f) \ n", (Fangle) *180.0f/static_cast&lt
		;float> (M_PI), fangleconfidence);
		Audio data Lread = 0;
		memset (Fdataarr, 0, lbuffersize);
		Stm->read (Fdataarr, Lbuffersize, &lread);
		if (Lread > 0) {printf ("Audio Buffer:%d\n", lread);
	Sleep (200);
} endclose:bb->close ();
	Endstop:system ("pause");
return 0;
 }

Complete