Optical flow Optical Flow introduction and OPENCV realization

What is optical flow (optic flow)? The name is very professional, feel very strange, but in essence, we are the most familiar. Because of this visual phenomenon we are experiencing every day. In essence, the light flow is the obvious visual movement you feel in this moving world (hehe, relativity, no absolute stillness, no absolute movement). For example, when you sit on a train and look out the window. You can see the trees, the ground, the buildings and so on, and they all go backwards. This movement is the light flow. And we all find that they are moving at a different speed. This gives us a very interesting message: Judging their distance from each other by the speed of movement of different targets. Some distant targets, such as clouds and mountains, move slowly and feel like stillness. But some objects that are closer to each other, such as buildings and trees, go back faster, and the closer they are to us, the faster they will move back. Some very close objects, such as the markings on the pavement, the grass, and so on, as if to make a swish sound in our ears.
Optical flow provides angular information in addition to proximity. The object moving in the direction of our eyes is 90 degrees faster than the other angle, when the small to 0 degrees, that is, the object in our direction directly to the collision, we just do not feel its movement (light flow), it seems to be stationary. When it gets closer to us, it gets bigger (of course, we usually see the feeling still has speed, because the object is bigger, its edge is still and our human eye has more than 0 angle).
Hehe, said so much, as if have not entered the more official, the definition of research. Then put one on it.
The concept of optical flow was first proposed by Gibson in 1950. It is the instantaneous velocity of the pixel motion of the space moving object in observing the imaging plane, it is to find out the correspondence between the previous frame and the current frame by using the change of the pixel in the image sequence and the correlation between the neighboring frames, so as to calculate the motion information of the object between the adjacent frames. In general, the light flow is caused by the movement of the foreground target itself, the motion of the camera, or the joint motion of the two in the scene.
When the human eye observes a moving object, the image of the object forms a series of continuously changing images on the retina of the human eye, a series of continuously changing information that constantly "flows" through the retina (the image plane), as if it were a "stream" of light, so called the optical flow (optical flow). The optical flow expresses the change of the image, because it contains the information of the target motion, so it can be used by the observer to determine the motion of the target.
The purpose of studying the optical flow field is to approximate the field which can not be obtained directly from the image sequence. The sports ground is actually the movement of the object in the three-dimensional real world, and the light flow field is the projection of the playground on the two-dimensional image plane (human eye or camera).
The popular saying is that through a sequence of pictures, each pixel in each image of the movement speed and direction of movement to find the light flow field. How do you find it? We intuitively understand that the position of point A at T-frame is (x1, y1), then we find point A at frame t+1, if it is (x2,y2), then we can determine the movement of point A: (UX, VY) = (x2, y2)-(x1,y1).
How do you know the position of point A at frame t+1? There are many ways to calculate the optical flow.
In 1981, Horn and Schunck creatively connected the two-dimensional velocity field with the gray scale, and introduced the optical flow constraint equation to obtain the basic algorithm of optical flow calculation. Based on different theoretical basis, people propose various methods of optical flow calculation, and the performance of the algorithm is different. Barron and others have summed up a variety of optical flow techniques, dividing them into four types according to the difference between theoretical basis and mathematical method: Gradient-based method, matching-based method, energy-based method, phase-based method. In recent years, the method of neuro-dynamics has been emphasized by scholars.
The other we do not say, return to the application (oh, too advanced, I can not go on). Many optical flow algorithms are implemented in OPENCV.

1) calcopticalflowpyrlk

The optical flow (sparse optical flow) of some point sets is computed by the pyramid Lucas-kanade Optical flow method. If you understand, you can refer to this article: "Pyramidal implementation of the Lucas Kanade Feature trackerdescription of the algorithm"

2) Calcopticalflowfarneback

The dense optical flow (i.e., the optical flow of all pixels on the image) is computed using the Gunnar Farneback algorithm. Its related papers are: "Two-frame Motion estimation Based on Polynomialexpansion"

3) CALCOPTICALFLOWBM

A block-matching method is used to calculate the optical flow.

4) CALCOPTICALFLOWHS

The dense optical flow is calculated using Horn-schunck algorithm. Relevant papers seem to be this article: "Determining Optical Flow"

5) Calcopticalflowsf
This one is the implementation of an article in the 2012 European visual Conference: "Simpleflow:a non-iterative, Sublinear Optical flowalgorithm", the Project site is:/http GRAPHICS.BERKELEY.EDU/PAPERS/TAO-SAN-2012-05/is introduced in the new version of OpenCV.

Dense optical flow requires an interpolation method to interpolate between pixels that are more easily tracked to resolve pixels that are not well-defined, so the computational overhead is quite large. In the case of sparse optical flow, he needs to specify a set of points (easy-to-track points, such as corner points) before being traced, so we need to use Cvgoodfeaturetotrack () to look for corner points before using the LK method, and then use the Pyramid LK algorithm to track motion. But personally, for less-textured objects, such as hands, LK sparse light streams are more likely to be lost.

For instructions on how to use their API, we directly refer to OpenCV's official manual:

http://www.opencv.org.cn/opencvdoc/2.3.2/html/modules/video/doc/motion_analysis_and_object_tracking.html# Calcopticalflowfarneback

IJCV2011 There is an article, "a Database and Evaluation methodology for Optical flow" in the face of the mainstream optical flow algorithm to do a brief introduction and evaluation of different algorithms. The URL is:

http://vision.middlebury.edu/flow/

I feel this article is very good in the interpretation of the optical flow algorithm, the regulations are very clear. If you want to know about optical flow, it is recommended to read this article. Another problem that needs to be mentioned is that the optical flow field is a shift in the X and y directions for each pixel in the image, so the flow of light flowing at the end of the above light flow calculation is a dual-channel image equal to the original image size. How do you visualize it? This article is displayed using the Munsell color system.

About the Munsell color system (Munsellcolor systems), you can see wikibaike. It is a color description system created by American artist Albert Munsell (Albert H. munsell,1858-1918) in 1898.
The space of the Munsell color system is roughly a cylindrical shape:
North-South axis = lightness (value), from full black (1) to full white (10).

Longitude = Hue (hue). Each week is divided into five main tones and five intermediate colors: Red (R), red-yellow (YR), yellow (Y), yellow-green (GY), Green (G), green-Blue (BG), Blue (B), Blue (PB), Violet (P), magenta (RP). The adjacent two positions were evenly divided by 10 copies, a total of 100 copies.

Distance from axis = Chroma (chroma), indicating the purity of the hue. Its value from the middle (0) outward with the tone of the purity of the increase, there is no theoretical upper limit (the average color is actually limited to about 10, reflective, fluorescent and other materials can be as high as 30). Because the human eye has different sensitivity to various colors, the chroma does not necessarily match the combination of each hue and lightness.

The specific color is identified as: Hue + lightness + chroma.

On the top of that evaluation site there is this MATLAB and C + + code shown from flow to color. But I feel the C + + code is divided into several files, a little messy, and then I myself organized into two functions, and with the OPENCV mat format.

The following code uses Calcopticalflowfarneback to calculate dense optical flow and display it with this color system. This method of calculating dense optical flow is relatively fast compared to other several, 640x480 video my is about 200ms a frame, but the other generally need more than two seconds. In the result diagram, different colors represent different moving directions, and the depth indicates the speed of movement.

void Calcopticalflowfarneback (Inputarray previmg, Inputarray nextimg,inputoutputarray flow, double pyrScale, int levels , int winsize, intiterations, int Polyn, double polysigma, int flags)

Most of the parameters in the paper have a good set of values, the direct use of their good.

//Farneback dense optical flow calculate and show in Munsell system of colors//Author:zouxy//Date:2013-3-15//Homepage:http://blog.csdn.net/zouxy09//email: [Email protected]//API Calcopticalflowfarneback () comes from OpenCV, and this//2D dense optical flow algorithm from the following paper://Gunnar farneback.  "Two-frame Motion estimation Based on polynomial Expansion". //And the OpenCV source code locate in. \opencv2.4.3\modules\video\src\optflowgf.cpp#include <iostream>#include "opencv2/opencv.hpp"using namespaceCvusing namespace STD;#define UNKNOWN_FLOW_THRESH 1e9//Color encoding of flow vectors from://Http://members.shaw.ca/quadibloc/other/colint.htm//This code was modified from://http://vision.middlebury.edu/flow/data/voidMakecolorwheel ( vector<Scalar>&colorwheel) {intRY = the;intYG =6;intGC =4;intCB = One;intBM = -;intMR =6;intI for(i =0; i < RY; i++) Colorwheel.push_back (Scalar (255,255*i/ry,0)); for(i =0; i < YG; i++) Colorwheel.push_back (Scalar (255-255*i/yg,255,0)); for(i =0; i < GC; i++) Colorwheel.push_back (Scalar (0,255,255*I/GC)); for(i =0; i < CB; i++) Colorwheel.push_back (Scalar (0,255-255*I/CB,255)); for(i =0; i < BM; i++) Colorwheel.push_back (Scalar (255*I/BM,0,255)); for(i =0; i < MR; i++) Colorwheel.push_back (Scalar (255,0,255-255*I/MR)); }voidMotiontocolor (Mat Flow, Mat &color) {if(Color.Empty ()) color.create (Flow.rows, Flow.cols, CV_8UC3);Static  vector<Scalar>Colorwheel;//scalar r,g,b    if(Colorwheel.empty ()) Makecolorwheel (Colorwheel);//Determine motion range:    floatMaxrad =-1;//Find max flow to normalize FX and FY     for(intI=0; i < flow.rows; ++i) { for(intj =0; J < Flow.cols; ++J) {vec2f flow_at_point = flow.at<vec2f> (i, j);floatFX = flow_at_point[0];floatFY = flow_at_point[1];if((fabs(FX) > Unknown_flow_thresh) | | (fabs(FY) > Unknown_flow_thresh))Continue;floatrad =sqrt(FX * FX + FY * fy); Maxrad = Maxrad > rad?          Maxrad:rad; }      } for(intI=0; i < flow.rows; ++i) { for(intj =0; J < Flow.cols; ++J) {Uchar *data = color.data + color.step[0] * i + color.step[1] * J; vec2f Flow_at_point = flow.at<vec2f> (i, j);floatFX = flow_at_point[0]/Maxrad;floatFY = flow_at_point[1]/Maxrad;if((fabs(FX) > Unknown_flow_thresh) | | (fabs(FY) > Unknown_flow_thresh)) {data[0] = data[1] = data[2] =0;Continue; }floatrad =sqrt(FX * FX + FY * fy);floatAngle =atan2(-fy,-FX)/cv_pi;floatFK = (angle +1.0) /2.0* (Colorwheel.size ()-1);intK0 = (int) FK;intK1 = (K0 +1)% colorwheel.size ();floatf = fk-k0;//f = 0;//uncomment to see original color wheel             for(intb =0; b <3; b++) {floatCol0 = Colorwheel[k0][b]/255.0;floatcol1 = Colorwheel[k1][b]/255.0;floatCol = (1-F) * col0 + f * col1;if(Rad <=1) col =1-Rad * (1-Col);//increase saturation with radius                ElseCol *=. the;// out of rangedata[2-B] = (int)(255.0* col); }          }      }  }intMainint,Char* *) {videocapture cap; Cap.open (0);//cap.open ("test_02.wmv");     if(!cap.isopened ())return-1;      Mat Prevgray, Gray, flow, cflow, frame; Namedwindow ("Flow",1); Mat Motion2color; for(;;) {DoubleT = (Double) Cvgettickcount ();          Cap >> frame;          Cvtcolor (frame, Gray, Cv_bgr2gray); Imshow ("Original", frame);if(prevgray.data) {calcopticalflowfarneback (Prevgray, gray, Flow,0.5,3, the,3,5,1.2,0);              Motiontocolor (flow, motion2color); Imshow ("Flow", Motion2color); }if(Waitkey (Ten) >=0) Break;STD:: Swap (Prevgray, gray); T = (Double) Cvgettickcount ()-t;cout<<"Cost Time:"<< t/((Double) cvgettickfrequency () *.) << Endl; }return 0; }

This is the effect:

A waving hand:

Although the background is moving, but because their direction of movement is not the same, so it can be identified in front of the hand, in the foreground and background movement is not uniform, it can be identified.

Optical flow Optical Flow introduction and OPENCV realization