Research on aligning Scene Depth Map and color map in Kinect

The color map and depth-of-Field Graph of Kinect are not aligned. depthimageframe provides the maptocolorimagepoint method to calculate the color map points corresponding to the coordinates of the specified depth-of-Field Graph. It was originally thought that the coordinate from the depth of field map to the color map coordinate is an affine transformation relationship. Therefore, when alignment, three points (0,400), (), () are specified in the depth of field map ), calculate the color map coordinates corresponding to the three points, then obtain the transformation matrix and perform coordinate transformation. Yesterday, the customer responded that when the character came to the right of the camera, there would be an image Dislocation:

 

Check whether the depth of field chart is aligned with the color chart. The following is a detailed test. The depth of field chart samples every 10 pixels, calculates the corresponding color chart coordinates, and draws them on the same graph. The depth of field chart coordinates are red points and the color chart coordinates are green points.
MatchCode:

Private List <point> points = New List <point> ();
For ( Int X = 0 ; X < 640 ; X + = 10 )
{
For ( Int Y = 0 ; Y < 480 ; Y + = 10 )
{
Colorimagepoint P = depthimageframe. maptocolorimagepoint (X, Y, colorimageformat. rgbresolution640x480fps30 );
Points. Add ( New Point (P. X, p. y ));
Points. Add ( New Point (x, y ));
}
}

Draw code:

Point [] list = points. toarray ();
Bitmap match = New Bitmap ( 640 , 480 );
Graphics G = graphics. fromimage (MATCH );
Pen pen1 = New Pen (color. Red, 2 );
Pen pen2 = New Pen (color. Green, 2 );
Int I = 0 ;
Foreach (Point P In List)
{
I ++;
If (I % 2 = 0 )
{
G. drawline (pen1, P, New Point (p. x + 1 , P. Y + 1 ));
}
Else
{
G. drawline (pen2, P, New Point (p. x + 1 , P. Y + 1 ));
}
}
Match. Save ( " Match _ " + Timestring + " . PNG " , System. Drawing. imaging. imageformat. PNG );

Here, timestring records the current time.

The result is as follows:

It can be seen that it is a very complex field. It can be seen with the naked eye that there are 16 special locations:

 

Let's not go into what these sixteen points are. As can be seen from the above, this transformation is not an affine transformation and cannot use an affine transformation matrix. Point-by-point transformation should be performed. The kinectsensor class provides a point-by-point transformation method: mapdepthframetocolorframe.

Now, we need to know whether the coordinate transformation between the depth of field chart and the color chart is stable or unstable (will the transformation relationship change over time ?), If it is stable, we only need to require one transformation. If it is not stable, we need to solve the transformation every time.

Drag the matching images generated by two different time points to Photoshop and align the red points. You can find that the green points are also aligned, except for one image. The upper right corner is not aligned, and the difference is not big:

It can be seen that the alignment relationship is relatively stable and can be calculated only once. The results are stored. During coordinate transformation, the table can be queried.