Optical Flow)

Optical Flow Method

Reprinted from yaoyi big white blog http://www.linyaoyi.com is also lamented on the network on the Optical Flow Method of Chinese information too little, so I plan to write snacks, because I have just come into contact with the optical flow method soon, if you are not talking about anything, I hope you can give me more information. This article mainly covers: Briefly describe the concept of optical flow, deduce the optical flow constraint equation, and explain a simple code example. 1. Concept First, what is optical flow? In space, motion is described in a sports field. But how can we know the motion of an object through a two-dimensional image plane? We know that an image is projected from a three-dimensional space, and each of the above points has a corresponding intensity value (intensity value ), the motion of an object is described by the changes in the intensity of the image sequence. The movement of a sports field in the space is transferred to the image, and the light flow field is displayed. is it hard to understand? Okay. Let's look at some equations first. 2. Optical Flow Constraint Equation We first assume that the brightness of the space is a constant (brightness constancy), and the following equation can be obtained: (1) Use the chain rule to expand =>; (2) =>; (3) -- optical flow constraint equation The changes in the light intensity of the image plane relative to the X and Y axes (adjacent pixels on the same image) indicate the changes in the intensity of the same pixel at the adjacent time (the same pixel on the continuous image ),, here is the optical flow. The academic point is that optical flow is a vector field that can represent the direction and amplitude of a continuous image intensity change. It can also be understood as a instantaneous velocity field generated by pixels with gray scale moving on the image plane. If you like, you can use Taylor series expansion to obtain the optical flow constraint equation: The optical flow constraint equation can also be obtained by keeping the brightness unchanged. Note that the time interval here is, And the offset is. -------------------------------- Now let's take a look at the (1) formula, or understand what is brightness unchanged. We can assume that there is a point (x, y) on the object moving in space ), after a period of time, the position of the point on the image plane is moved, and the brightness is constant, that is, no matter which position the point is moved to, its intensity is always unchanged, so we have or (X (t) and Y (t) At the beginning, which represent the original object (x, y), but after a certain period of time, the position on the image has changed), or three of them express one meaning. Okay. Do you have any feelings from the first expression of brightness constancy? In my understanding, the optical flow method aims to track a point on an object determined by (x, y, we need to track the position of the image after time. How can we trace it? Set t moment image to image_a, and the image after time to image_ B. Okay, as mentioned earlier, the brightness of the same point remains unchanged, then, the brightness value on image_ B must be equal to the brightness value on image_a (x, y) at t time, and this is optical flow. As for how to find it, we will try again later. To sum up, we can estimate the locations of interest points x, y, and brightness constancy in the original image, that is, to determine where the original feature point went, at this time, if the frame rate is used together, the moving speed of the feature points can be obtained. 3. Code Implementation The following is the code taken from the Stanford optical flow course. I have explained it briefly, and I have attached the code to download it. It contains the original text comment. If you are interested, you can download it. The purpose of this example is to allow friends who are new to the optical flow method to have a perceptual knowledge. later articles will explain the algorithm in detail. (Omitted) The Stanford author's source code and the video files used can be found at the following URL: Http://ai.stanford.edu /~ Dstavens/cs223b/ Source code after modification (camera operation): optical_flow_demo_edited.cpp In addition, the external/location where your opencv is installed/samples/C/lkdemo. C. You can also take a look at this demo if it is okay. The compilation method is as follows: $gcc -o optical_flow_demo_edited optical_flow_demo_edited.cpp `pkg-config opencv --libs --cflags opencv`

Introduction to optical flow

The previous article mainly introduced the basic concepts of optical flow. I hope you can have some experience on what optical flow is and what it is used for. This article will introduce you to a classic optical flow solution: the context of this article is as follows: first, let's briefly review the content in this article, then deduce the formula of Lucas-kanade, and talk about my understanding, at last, we provide pseudocode to help you better understand this method.
1. Overview
A point on a moving object appears at different positions on the image plane at different times. to track an object, we need to track these points, but how can we track them? We assume that as long as the brightness of the same vertex remains unchanged, that is, the same vertex is expressed. For the time T derivation, we obtain an optical flow constraint equation:

                                 (1)

Here is the required optical flow, but the trouble is that there are two unknown here (u_x, u_y) but there is only one equation, so we cannot solve it, you need to try to add another constraint (that is, the equation). Well, now you have some experience. In fact, these methods are just adding the equation in the idea and then solving them simultaneously, in this article, how does the Lucas-kanade Optical Flow Method Add constraints to obtain the optical flow?

2. Lucas-kanade Optical Flow Method
The Lucas-kanade optical flow method was proposed in 1980s. Now it has been 30 years and I don't know if it is out of date. Now I should have a more advanced method, I hope you will give me more advice. Okay. Let's talk about the theory of Lucas-kanade. On the premise that the time between frames is short and the motion of objects is very insignificant, we can assume that the image is in a small range (such as a small window with Axa = m) the optical flow (velocity) of similar pixels is equal. If there are M pixels in a small area, we can obtain M equations based on the optical flow constraint equation:

Convert to Matrix Representation:

Haola looks very good. m equations are two unknown. I feel like I have wasted all these resources for how to find them. But is such a derivation correct? Please consider it for a moment.
.......
Can you see the problem?
Note that when we make a speed hypothesis for neighboring pixels, the U in the optical flow constraint equation (1) is no longer the original U, and the U has become the estimated value, which is expressed as follows, repeat it again: U is not a real value, but an estimation of the real value, expressed.
Now let's take a closer look. If you still remember the Taylor extension mentioned in the previous chapter

  (2)

In the above formula, the three arrows are unknown, because u becomes the estimated value, so the formula should also be a correct estimate, expressed as, and also known from the constant photometric (this condition remains unchanged)

Where

While

(3) The error is an estimation error.

From equation (2) (3), we can see that the optical flow constraint equation becomes

 (4)

After obtaining the new optical flow constraint equation (4), we will re-analyze it. Assuming that the selected window has M pixels, We can get m equations:

 
Then we will apply the least squares (least-square) pull. I will write a special article on this method. I will not go into details here. The least square method requires a cost function. Here is
Where
To minimize the cost function, our common method is to evaluate it and make it zero.

=>

Ling
If M is not odd (full rank), the optical flow can be obtained.
If M is singular, it means that there is only one equation, and the optical flow along the gradient is obtained.
In addition, the window size is usually 5 × 5. If it is too small, the aperture problem (M singular) is easily displayed. In this way, the obtained optical flow may not be able to describe the motion of an object, the reason for saying "possible" is that if the object happens to be in the direction of the photometric gradient, it will be hit. Well, I will write an article about aperture problem if I have time, so I will not talk about it more.
3. pseudocode of Lucas-kanade Optical Flow Method
First, define the target: Tracking! Obtain the corresponding vertex Q on imageb from a point P on imagea.
The intensity on imagea is counted as I (x, y), and the intensity on B is counted as J (x, y)
Suppose that number_features feature points are found in imagea, And the location information is features []
Array
Set window size W = 5
For I = 0 up to number_features with step of + 1

/* In implementation, the difference approximation is used. Here Centrl is used.
Difference operator, and many other operators such as Sharr Operator
Sobel operator. opencv uses the Sobel operator */

Photometric gradient in the X direction:
Photometric gradient in the Y direction:

Light variation in the t direction:

Calculation coefficient matrix m:

Calculation matrix B:


/* Determine whether M is singular, and solve the optical flow */
If (| M |)
{
/* Obtain optical flows through normal matrix operations */
} Else
{
/* Only normal flow can be calculated */
}
End of for-loop on I
This is probably the case. The next article will introduce pyramid Lucas-kanade optical flow method.