Computer vision and Pattern Recognition (3)--facemorphing

This is a very interesting application that can gradually transition a person's face to another person's face. Spent about 1 days to complete the most basic functions, about 3 days to improve it, there may be a lot of bugs waiting for me to modify, but first to record the current progress.

Graphics Library: CImg

Environment: Win10, c++11, VS2013

One:

Two:

Ideas:

1) capture control points (coordinates) for image A and image B, respectively

2) Use the Delaunay algorithm to divide the triangles to ensure that the triangle one by one of image A and image B correspond and do not overlap.

3) Calculate the transition triangle of each pair of triangles, the transition amount is 0~1.

4) compute the affine transformation matrix of triangles to triangles.

5) Transform the image A and image B to the transitional triangular mesh by using the transformation matrix respectively.

6) Use the transition amount to calculate the pixel color of the synthetic chart.

I think the difficulty mainly lies in the Delaunay algorithm and the affine transformation matrix computation, followed by the application robustness enhancement. Because there is no technology to learn face recognition at present, it is necessary to make some "Dirty work" by artificially calibrating control points. Therefore, I also divided the project into two: first, the main project, specifically responsible for the triangular segmentation of the image processing. Second, ancillary engineering, specifically responsible for the acquisition of control points and triangular segmentation. It can be said that the core of the main project is morphing, the core of the auxiliary engineering is Delaunay.

First look at the auxiliary engineering Bar, pre-defined the commonly used structure, you can write Delaunay algorithm. The Delaunay segmentation criterion is: Any triangle ABC, does not exist a little d, within its circumscribed circle. Considering that the number of control points in the face is generally low by two digits, and the computer calculates millions data on average per second, which is to support the calculation of O (n^4) per second, a brute force algorithm can be used:

void Delaunay::builddelaunayex (const std::vector<dot*>& Vecdot, std::vector<triangle*>& Vectriangle) {int size = Vecdot.size (), if (Size < 3) return;for (int i = 0; i < size-2; ++i) {for (int j = i + 1; J & Lt size-1; ++J) {for (int k = j + 1; k < size; ++k) {dot* A = vecdot[i];D ot* B = vecdot[j];D ot* C = vecdot[k];  triangle* tri = new Triangle (*a, *b, *c); gettrianglecircle (*tri); bool find = true;for (int m = 0; m < size; ++m) {dot* P = Vecdot[m];if (*p = = *a | | *p = *B | | *p = *c) continue;if (pointincircle (*p)) {find = False;break;}} if (find) {vectriangle.push_back (tri);}}}}

I tried it. When the control point is not more than 40, it is 1 seconds to run out. Of course, this is one of the most stupid and simple ways.
In order to make the triangular division of image A and image B correspond to one by one, I only Delaunay the image a, and the image B is segmented according to the triangular composition of image a. Therefore, I will record the ordinal number of each point of image A, and the sequence number of three points per triangle. The serial numbers are eventually output in text form.

#include "CImg.h" #include "Delaunay.h" using namespace cimg_library; #include <iostream> #include <fstream> #include <string>using namespace Std;int main () {//Input image path string patha, Pathb;cout << "image A path:"; CIN >> patha;cout << "Image B path:"; Cin >> PATHB; Cimg<double> Sourcea (Patha.c_str ()), Sourceb (Pathb.c_str ()); Cimgdisplay Adisp (Sourcea, "image A"), Bdisp (Sourceb, "image B");//Control point array vector<dot*> Vecdota, vecdotb;// The image corner is pre-placed into the array vecdota.push_back (new dot (0, 0, 0)), Vecdota.push_back (new dot (0, Sourcea.height () 1, 1)), Vecdota.push_back (New Dot (Sourcea.width ()-1, Sourcea.height ()-1, 2)), Vecdota.push_back (New Dot (Sourcea.width () 1, 0, 3)); Vecdotb.push _back (new dot (0, 0, 0)), Vecdotb.push_back (new dot (0, Sourceb.height () 1, 1)), Vecdotb.push_back (New Dot (Sourceb.width ( )-1, Sourceb.height ()-1, 2)); Vecdotb.push_back (New Dot (Sourceb.width () 1, 0, 3));//dot Line color int color[3] = {0, 255, 0} ;//Click the mouse to get the coordinates while (!) Adisp.is_closed ()) {ADisp.wait (); if (Adisp.button () & 1 && adisp.mouse_y () >= 0) {dot* click = new Dot (adisp.mouse_x (), Adisp.mou Se_y (), Vecdota.size ()), Sourcea.draw_circle (Click->x, Click->y, Sourcea.width ()/+, color), Sourcea.display ( ADISP); Vecdota.push_back (click);}} while (! Bdisp.is_closed ()) {bdisp.wait (); if (Bdisp.button () & 1 && bdisp.mouse_y () >= 0) {dot* click = new Dot (bdisp . mouse_x (), bdisp.mouse_y (), Vecdotb.size ()), Sourceb.draw_circle (Click->x, Click->y, Sourceb.width ()/40, color); Sourceb.display (Bdisp); Vecdotb.push_back (click);}} Triangle Array vector<triangle*> Vectria, vectrib;//to image a Delaunay Triangle Division Delaunay (). Builddelaunayex (Vecdota, Vectria) ;//synchronous image B triangle split for (int i = 0; i < vectria.size (); ++i) {dot* A = vecdotb[vectria[i]->a.value]; dot* B = vecdotb[vectria[i]->b.value]; dot* C = vecdotb[vectria[i]->c.value]; Vectrib.push_back (New Triangle (*a, *b, *c)); } cimg<double> Targeta (Patha.c_str ()), Targetb (Pathb.c_str ()); CimgdisplayTadisp (Targeta), Tbdisp (TARGETB);//Click the mouse to gradually display the triangle int i = 0;while (! Tadisp.is_closed ()) {tadisp.wait (), if (Tadisp.button () && i < Vectria.size ()) {Targeta.draw_line (vectria[i ]->a.x, Vectria[i]->a.y, vectria[i]->b.x, vectria[i]->b.y, color); Targeta.draw_line (vecTriA[i]->a.x , Vectria[i]->a.y, vectria[i]->c.x, vectria[i]->c.y, color); Targeta.draw_line (vectria[i]->b.x, vecTriA[ I]->b.y, vectria[i]->c.x, vectria[i]->c.y, color); Targeta.display (Tadisp); Targetb.draw_line (vecTriB[i]- >a.x, Vectrib[i]->a.y, vectrib[i]->b.x, vectrib[i]->b.y, color); Targetb.draw_line (vecTriB[i]->a.x, VECTRIB[I]-&GT;A.Y, vectrib[i]->c.x, vectrib[i]->c.y, color); Targetb.draw_line (vectrib[i]->b.x, VecTriB[i ]->b.y, vectrib[i]->c.x, vectrib[i]->c.y, color); Targetb.display (tbdisp); ++i;}} Get image naming, change named txt format string Filenamea = Patha.substr (0, Patha.find_last_of (".")) + ". txt"; string filenameb = Pathb.substr (0, Pathb.find_last_of (".")) + ". txt";//ControlPoint and Triangle Write text Ofstream outputa (Filenamea, Ios::out), Outputb (Filenameb, ios::out), Outputa << "[Points]" << Endl;outputb << "[Points]" << endl;for (int i = 0; i < vecdota.size (); ++i) {Outputa << Vecdota[i]-&gt X << "," << vecdota[i]->y << endl;outputb << vecdotb[i]->x << "," << Vecdotb[i] ->y << Endl;} Outputa << "[Triangles]" << endl;outputb << "[Triangles]" << endl;for (int i = 0; i < VECTRIA.S Ize (); ++i) {Outputa << vectria[i]->a.value << "," << vectria[i]->b.value << "," << vectria[ I]->c.value << endl;outputb << vectrib[i]->a.value << "," << vectrib[i]->b.value <& Lt "," << vectrib[i]->c.value << Endl;}}

Next is the main project, with the text data provided by the auxiliary engineering, the main project needs to calculate the transition triangles for each triangle:

Calculate transition Triangles triangle* Middletriangle (triangle* A, triangle* B, float rate) {Float ax = rate* (a->a.x) + (1-rate) * (b->a . x); float ay = rate* (a->a.y) + (1-rate) * (B->A.Y); float bx = rate* (a->b.x) + (1-rate) * (b->b.x); float by = R ate* (A->B.Y) + (1-rate) * (B->B.Y); float cx = rate* (a->c.x) + (1-rate) * (b->c.x); float cy = rate* (A->C.Y) + (1-rate) * (B->C.Y); return new Triangle (dot (ax, ay), dot (bx, by), Dot (CX, CY));}

Then calculate the transformation matrix coefficients:

matrix3x3 Warp::triangletotriangle (float u0, float v0, float U1, float v1, float U2, float v2,float x0, float y0, float x1 , float y1, float x2, float y2) {//| A|int Deta;deta = u0*v1 + u1*v2 + u2*v0-u2*v1-u0*v2-u1*v0;//a*int A11, A12, A13, A21, A22, A23, A31, A32, A33; A11 = V1-v2; A21 =-(V0-V2); A31 = V0-V1; A12 =-(U1-U2); A22 = U0-U2; A32 =-(U0-U1); A13 = U1*V2-U2*V1; A23 =-(u0*v2-u2*v0); A33 = U0*v1-u1*v0; matrix3x3 result;result.a11 = (float) (x0*a11 + x1*a21 + x2*a31)/Deta;result.a21 = (float) (y0*a11 + y1*a21 + y2*a31)/de Ta;result.a31 = (float) (A11 + A21 + A31)/Deta;result.a12 = (float) (x0*a12 + x1*a22 + x2*a32)/deta;result.a22 = (float) (Y0*a12 + y1*a22 + y2*a32)/deta;result.a32 = (float) (A12 + A22 + A32)/Deta;result.a13 = (float) (x0*a13 + x1*a23 + x2*a )/Deta;result.a23 = (float) (y0*a13 + y1*a23 + y2*a33)/deta;result.a33 = (float) (A13 + A23 + A33)/Deta;return Resul t;}

The purpose of computing the Transition Triangle transformation matrix is to convert the original image into a transition image, so that the two original images are superimposed on a consistent triangular grid (MORPH). In order to achieve this goal, we need to map the triangular meshes of the original image to the transitional triangular meshes using the transformation matrix. Each pair of triangles has a transform matrix, which is stored in a dynamic array.

Compute trigonometric Transformation matrices vector<matrix3x3*> vecab, vecba;for (int i = 0; i < vectria.size (); ++i) {matrix3x3 HAM = Warp::triangl Etotriangle (*vectria[i], *vectrim[i]); Vecab.push_back (new matrix3x3 (HAM)); matrix3x3 HBM = Warp::triangletotriangle (*vectrib[i], *vectrim[i]); Vecba.push_back (new matrix3x3 (HBM));}

Triangular mapping is relatively simple, to determine whether each pixel is within the original triangle, and then calculate the coordinates within the map triangle, and finally the coordinates of the pixel on the original image to do a bilinear interpolation calculation, it can be used as the target pixel color value. Take image A as an example:

A triangular transformation of the original image a Cimg_forxy (Targeta, x, y) {bool isfind = false;for (int i = 0; i < vectrib.size (); ++i) {if (vectrib[i]-> Pointintriangle (Dot (x, y)) {float tx = x*vecba[i]->a11 + y*vecba[i]->a12 + vecba[i]->a13;float ty = x*vecba[i]-&  Gt;a21 + y*vecba[i]->a22 + vecba[i]->a23;if (TX >= 0 && TX < width && ty >= 0 && ty < height) {Cimg_forc (Sourcea, C) Targeta (x, y, 0, c) = Sourcea.linear_atxy (tx, Ty, 0, c);} Isfind = True;break;}} if (!isfind) {Cimg_forc (Sourcea, C) Targeta (x, y, 0, c) = Sourcea.linear_atxy (x, y, 0, C);}}

The pixel color of the final composite target graph:

Compute color interpolation cimg_forxyzc (target, x, Y, z, c) target (x, Y, z, c) = Rate*targetb (x, Y, Z, c) + (1-rate) *targeta (x, Y, z, c);

To this, the application has been basically molded. More testing is still going on, and I want to eventually make it into a web-based face-changing app through canvas.

Computer vision and Pattern Recognition (3)--facemorphing