[Opencv-python] OpenCV part IX Object detection part X in computational photography

Part IX
Computational photography

49 image denoising
target
? Learn to remove noise from images using the nonlocal mean denoising algorithm
? Learning functions Cv2.fastnlmeansdenoising (), cv2.fastnlmeansdenoisingcolored (), etc.
principle
In the previous chapters we have learned a lot about image smoothing techniques , such as Gaussian smoothing, median smoothing, and so on, the effects of these techniques are good when the noise is relatively small. In these techniques, we select a small neighborhood around the pixel and then replace the center pixel with a Gaussian mean or median mean value. In simple terms, noise removal at the pixel level is restricted to local neighborhoods. The
noise has a property. We believe that noise is a random variable with an average of one. Consider a pixel with noise, p = P 0 + N, where P 0 is the true value of the pixel, and N is the noise of the pixel. We can select a large number of identical pixels (N) from different images and then calculate the average. Ideally we would get p = p 0. Because the average noise is 0.
We can remove these noises with a simple setup. Hold a static camera in one position for a few seconds. So we get enough image frames, or lots of images of the same scene. Write a code to solve the average of these frames (this should be a piece of cake for you). Compare the final result with the first frame image. You will find that the noise is reduced. Unfortunately, this simple approach does not always apply to cameras and motion scenes. In most cases we only have one guide with noisy images.
The idea is simple, we need a similar set of images that can be used to remove noise by averaging. Considering a small window (5x5) in the image, there is a great possibility that other areas in the image also have a similar window. Sometimes this similar window is around the neighborhood. What if we find these similar windows and take their averages? This is good for a particular window. As shown in.

　　　　

The Blue window appears to be similar. The green window also looks similar. So we can select a small window containing the target pixel, then search the image for a similar window, and then the average of all the windows, and replace the value of the target pixel with this value. This method is the non-local mean noise reduction. This algorithm consumes more time than the smoothing techniques we learned before, but the results are good. You can find more details and online demos in more resources.
For color images, first convert to the CIELAB color space, and then the L and AB components are respectively de-noising.

Image de-noising in 49.1 OpenCV
OpenCV provides four copies of this technology.
1. Cv2.fastnlmeansdenoising () uses the object as a grayscale image.
2. Cv2.fastnlmeansdenoisingcolored () uses the object as a color map.
3. Cv2.fastnlmeansdenoisingmulti () for short-time image sequences (grayscale images)
4. Cv2.fastnlmeansdenoisingcoloredmulti () for short time image sequences (color images)
Common parameters are:
? H: Determines the filter strength. High h is good for noise removal, but it also erases the details of the image. (take 10 of the effect is good)
? Hforcolorcomponents: Same as H, but used with color images. (Same as H)
? Templatewindowsize: Odd. (Recommended value is 7)
? Searchwindowsize: Odd. (Recommended value is 21)
See the details of getting these parameters with multiple resources.
Here we will show 2 and 3, and the rest will be left to you.

49.1.1 cv2.fastnlmeansdenoisingcolored ()
As mentioned above, it can be used to remove noise from color images. (assumed to be Gaussian noise). The following is an example.

Import NumPy as NP Import Cv2  from Import  = cv2.imread ('die.png'= cv2.fastnlmeansdenoisingcolored (Img,none, 10,10,7,21) Plt.subplot (121), Plt.imshow (IMG) plt.subplot (122), Plt.imshow (DST) plt.show ()

Here is a larger version of the result, our input image contains a variance of 25 noise, the following is the result.

　　　　

49.1.2 Cv2.fastnlmeansdenoisingmulti ()
Now we're going to use this method for a video. The first parameter is a list of noise frames. The second parameter Imgtodenoiseindex sets those frames that need to be de-noising, and we can pass in the index of a frame. The third parameter, Temporawindowsize, can set the number of adjacent frames used for de-noising, which should be an odd. In this case, the image of the temporawindowsize frame is used for denoising, and the middle frame is the frame to be de-noising. For example, we passed in a 5-frame image, Imgtodenoiseindex = 2 and temporalwindowsize = 3. Then the first frame, the second frame, and the third frame image will be used for the denoising of the second frame image. Let's take a look at an example.

ImportNumPy as NPImportCv2 fromMatplotlibImportPyplot as Pltcap= Cv2. Videocapture ('Vtest.avi')#Create a list of first 5 framesIMG = [Cap.read () [1] forIinchXrange (5)]#convert all to grayscaleGray = [Cv2.cvtcolor (i, Cv2. Color_bgr2gray) forIinchimg]#convert all to Float64Gray = [Np.float64 (i) forIinchGray]#Create a noise of varianceNoise = Np.random.randn (*gray[1].shape) *10#Add This noise to imagesNoisy = [i+noise forIinchGray]#Convert to Uint8Noisy = [Np.uint8 (Np.clip (i,0,255)) forIinchNoisy]#Denoise 3rd Frame Considering all the 5 framesDST = Cv2.fastnlmeansdenoisingmulti (Noisy, 2, 5, None, 4, 7, 35) Plt.subplot (131), Plt.imshow (Gray[2],'Gray') Plt.subplot (), Plt.imshow (Noisy[2],'Gray') Plt.subplot (133), Plt.imshow (DST,'Gray') plt.show ()

I got a larger version of the result.

The calculation consumes a considerable amount of time. The first picture is the original image, the second one is a noise figure
Like, the third one is the image that goes after the noise.

　　　　

50 Image Patching
Goal
? Use patching techniques to remove small noises and scratches from old photos
? Using OpenCV-related functions in the patch technology

50.1 Basics
In each of our homes there may be a few pictures of degraded old, sometimes the top accidentally on top of a few stains or a few strokes. Have you ever thought about fixing these photos? We can use the Brush tool to easily apply two, but it doesn't work, you just replace the black strokes with white strokes. At this point we are going to turn to image patching technology. The basic idea of this technique is simple: use the pixels around the bad point to replace the bad, so it looks more like the surrounding pixels. As shown (photo from Wikipedia):

　　　　
To achieve this, scientists have proposed several algorithms, and OpenCV offers two of them. Both of these algorithms can be implemented by using the function Cv2.inpaint ().
The first algorithm was implemented according to the article published by Alexandru_telea in 2004. It is based on the fast marching algorithm. Take an example of an area to be patched in the image. The algorithm moves slowly from the boundary of the region to the inside of the region, filling the region boundary pixels first. It selects a small neighborhood around the pixel to be patched, uses the normalized weighting in the neighborhood, and updates the pixel values to be repaired. The choice of weights is very important. For pixels close to the point of repair, the pixel points near the normal boundary and the pixels on the contour are given a higher weight. When a pixel is repaired, use the Fast Travel algorithm (FMM) to move to the next nearest pixel. FMM ensures that the bad points near the known (non-degraded) pixels are repaired first, which is similar to manual heuristic operations. You can set the label parameter to CV2. Inpaint_telea to use this algorithm.
The second algorithm was implemented according to Bertalmio,marcelo,andrea_l.bertozzi, and Guillermo_sapiro published in 2001. The algorithm is based on fluid dynamics and uses partial differential equations. The basic principle is heuristic. It first moves along the boundary of the normal region toward the degraded region (since the boundary is continuous, so that the boundary of the degenerate region's non-boundary and normal area should be continuous). It extends the equal intensity line (isophotes, a line practiced by a point of equal gray value) by matching the gradient vectors in the area to be repaired. To achieve this goal, the authors are used in some methods of fluid dynamics.

After you complete this step, the gray values in this area are minimized by filling the color. You can set the label parameter to CV2. Inpaint_ns to use this algorithm.

50.2 code
We will create a mask image equal to the size of the input image and set the pixel in the area to be repaired to 255 (elsewhere 0). All the operations are simple. There are several black strokes in the image I want to fix. I added it using the brush tool.

Import NumPy as NP Import  = cv2.imread ('messi_2.jpg'= cv2.imread ('mask2.png '  = Cv2.inpaint (img,mask,3, Cv2. Inpaint_telea) cv2.imshow ('DST', DST) cv2.waitkey (0) Cv2.destroyallwindows ()

The results are as follows. The first image is a degraded input image, and the second is a mask image. The third one is the result of using the first algorithm, and the last pair is the result of using the second algorithm.

　　　　

Part X
Object detection

51 Face Detection using the Haar classifier
Goal
? Facial detection technology based on Haar feature classifier
? Extend the face detection to eye detection and more.

51.1 Basics
Object detection based on Haar feature classifier is a very effective object detection technology (2001 Paul_viola and Michael_jones). It is based on machine learning, using a large number of positive and negative sample image training to get a cascade_function, and finally use it to do object detection.
Now we're going to study facial testing. Initially, the algorithm requires a large number of positive sample images (face images) and negative sample images (without facial images) to train the classifier. We need to extract the features from them. The Haar feature is used in the. They are like our convolution cores. Each feature is a value that is equal to the pixel value in the black rectangle minus the sum of the pixel values in the white rectangle. Use all possible cores to calculate a sufficient number of features. (Imagine how much calculation this would take?) Just a 24x24 window has 160,000 features). For the calculation of each feature we need to calculate the pixels inside the white and black rectangles as well. To solve this problem, the author introduces integral images, which can greatly simplify the summation operation, for pixels in any region and only four pixels on the integral image. Very beautiful, it can make the operation speed fast!
But most of the features we have calculated are irrelevant. As shown in.

　　　　

The

Top row shows two good features, and the first feature looks like a description of the area around the eye, because the eyes are always darker than the nose. The second characteristic is that the eyes are darker than the bridge of the nose. But if you put these two windows on the cheek, it's irrelevant. So how do we choose the best features from more than 160000+? Use Adaboost.
For this purpose, we apply each feature to all training images. For each feature, we need to find the best threshold for distinguishing between positive and negative samples. However, it is clear that this can result in errors or incorrect categorization. We want to choose the features with the lowest error rates, which means they are the best feature for detecting facial and non-facial images. (This process is actually not as simple as we say.) At the beginning, each image has the same weight, and after each classification, the weight of the image being divided is increased. The same process will be done again. Then we get new error rates and new weights. Repeat this process to know the accuracy of the arrival requirement or the error rate or the required number of features found). The
Final classifier is the weighted sum of these weak classifiers. The reason for being weak classifiers is that they are not enough to classify images, but together with other classifiers is a very strong classifier. The article says 200 features can provide 95% accuracy. They finally use 6,000 characteristics. (reduced from 160000 to 6000, the effect is significant!) ）。

　　　　
Now you have an image of each 24x24 window using these 6,000 features to do a check to see if it is a face. Is it very inefficient and time-consuming? This is true, but the author has a better solution.
Most areas in an image are non-facial areas. So it's a good idea to have a simple way to prove that this window is not a facial area, and if not, discard it without having to deal with it. Instead of concentrating on studying the area is not the face. In this way we can spend more time in areas that may be facial.
To achieve this goal, the author proposes the concept of cascade classifier. Instead of doing these 6,000 feature tests on the window in the first place, these features are grouped into groups. Use each in a different classification phase. (usually few of the previous phases use less feature detection). If a window in the first phase of the detection is not enough to directly discard the subsequent test, if it passes through the second stage of detection. If a window passes all the tests, the window is considered to be the face area.
This plan is not very handsome!!!
The author divides more than 6,000 features into 38 stages, and the first five phases feature numbers of 1,10,25,25 and 50 respectively. (The two features in fact are the best features obtained from Adaboost).
According to authors, on a average, features out of 6000+
is evaluated per Sub-window.
The above is a visual explanation of how our viola-jones facial detection works. Reading the original literature or more resources from non-references will help you a lot.

Haar cascade detection in 51.2 OpenCV
The OpenCV comes with a trainer and a detector. If you want to train a classifier yourself to detect cars, airplanes, etc., you can use OpenCV to build. Details of which are here: Cascade Classifier Training
Now let's learn how to use detectors. OpenCV already contains a number of well-trained classifiers, including: face, eyes, smile, etc. These XML files are saved in the/opencv/data/haarcascades/folder. Below we will use OpenCV to create a face and eye detector.
First we need to load the required XML classifier. The input image or video is then loaded in grayscale format.

Import NumPy as NP Import  = Cv2. Cascadeclassifier ('haarcascade_frontalface_default.xml'= Cv2. Cascadeclassifier ('haarcascade_eye.xml'= cv2.imread ('  Sachin.jpg'= Cv2.cvtcolor (img, Cv2. Color_bgr2gray)

Now we detect the face in the image. If a face is detected, it returns the rectangle where the face is located
Region Rect (X,Y,W,H). Once we have this position, we can create an ROI and
Which carries out eye detection. Who let the eyes grow on the face!!! ）

 faces = Face_cascade.detectmultiscale (gray, 1.3, 5 for  (X,Y,W,H) in   faces:img  = Cv2.rectangle (IMG, (x, y), (x+w,y+h), (255,0,0), 2 = gray[y:y+h, X:x+w] roi_color  = img[y:y+h, X:x+w] eyes  = Eye_cascade.detectmultiscale (Roi_gray)  for  (Ex,ey,ew,eh) in   Eyes:cv2.rectangle (Roi_color, (Ex,ey), (ex  +ew,ey+eh), (0,255,0), 2" cv2.imshow (  " img   " ,img) cv2.waitkey (0) cv2.destroyallwindows ()  

The results are as follows:

　　　　

[Opencv-python] OpenCV part IX Object detection part X in computational photography