Image cutout Algorithm Learning-Shared sampling for real-time Alpha matting

Http://www.tuicool.com/articles/63aANv

I. Preamble

On a rolling succession if accumulated, I estimate that at least 20 dozen friends add me qq, to me to consult about the algorithm of the cutout, unfortunately, I have not studied this aspect before. In addition to using the same technology as the magic wand in Photoshop, or a similar color-range algorithm in the selection menu in Photoshop (the two I have at least 90% consistent code) is to implement a simple cutout, now some state of art algorithms I do not understand. As a result, many opportunities to transform knowledge into assets are also wasted. Year 30 that day, a chance, a friend recommended I read an article on the cutout, and a matching implementation code, so I decided to start from this article on my cutout algorithm research trip.

This article is a shared sampling for real-time Alpha matting, and some of the information about this article can be found on this site: http://www.inf.ufrgs.br/~eslgastal/ sharedmatting/, a matching code in the CSDN can be downloaded, see: http://download.csdn.net/detail/jlwyc/4676516

The title of this article is very attractive, the publication date is 2010, also is relatively new. Before we go any further, I would like to remind you that there are a lot of limitations in real-time: (1) It has to rely on a strong GPU, and (2) the background of the applied cutout should be relatively simple.

No matter how, because there are matching implementation code, as a starting research, the article is still good.

From the current popular cutout technology, the idea of this article is a relatively outdated one.

Second, technical details

Well, no matter how much, I'll put in some of the papers in the formula and some instructions to describe the main body of the article.

To put it simply, the problem of keying is to solve a super-morbid equation as follows:

In the formula: C p is the color of the image we observe, F p, B p, and αp are unknown quantities, which can be called foreground, background and transparency respectively.

To solve such a morbid equation, it is necessary to add some additional constraints, in general, such a constraint can be scribbles with the same size as the image of the Trimap or user call off the scribbles form, as shown in the following two figure (if not specifically stated, The general white part represents the foreground, black indicates the background, and Gray indicates the part to be recognized):

Trimap scribbles

This constraint allows us to know that the part is clearly in the foreground (Αp =1), and that part belongs to the background (αp =0), then the main task below is to fix the αp values of those unknown areas.

According to the thesis, the main method of solving matting problem around 2010 is based on sampling, pixel af?nities or the combination of both, especially the latter two are the mainstream way. But both of these need to solve a large linear system, the size of the system and the number of unknown points is proportional (I simply see the next closed form the code of the drawing document, the use of a large sparse matrix), so for the size of 1MB figure, the solution time in a few seconds to a few minutes. The algorithm proposed in this paper should be based on sampling technology, he took full advantage of the similarity between neighboring pixels, and took advantage of the inherent parallelism of the algorithm, combined with GPU programming, to realize the real-time display of the cutout.

In general, the algorithm presented in this paper can be divided into 4 steps:

The first step: Expansion, for the user's input, the known area (foreground or background) for small scale expansion;

The second step: sample and Gather, each point in the remaining unknown region according to a certain rule sampling, and select the best pair of foreground and background sampling point;

Step three: Re?nement the optimal pairing of each point within an unknown region, within a certain field range.

Fourth step: Local smoothing, smoothing the resulting foreground and background pairs and transparency values to reduce noise.

2.1 Expansion

This step, according to my experience, can not do, his only role is to reduce the number of unknown points, may be to a certain extent, reduce the amount of later calculation, the principle is very simple, is to an unknown point, within its radius of a certain neighborhood (the recommended value of 10,

and is a circular radius), if there is a known background point or foreground point, calculate its color and the distance of these known point colors, and then return the unknown point to the object (foreground or background) whose color distance is less than a certain value and closest to that point.

In the reference code provided by CSDN, this part of the coding in fact is still very special, his loop is different from our normal neighborhood coding, he is from the pixel point to the external loop, a bit like the left side of this cycle of the way (actually a little difference, the actual is the top and bottom two lines to deal with, In the left and right two columns of processing, and then spread to the outer layer, the obvious advantage of this process is that, as long as a certain point color distance is less than the set value, you can stop the loop, because this point must be the first to meet the color distance conditions and meet the minimum physical distance requirements.

This step does not do, the final result of a certain impact, but he does not have a qualitative impact.

2.2 Sample and Gather

in general, this step is the core of the algorithm, but also the most impact on the results, his steps are actually very simple, we look first.

In this diagram, the P and Q points are unknown areas, we need a certain principle in the known area for a certain sample pair, the paper proposed the extraction method is:

Set a parameter kg, meaning a point can be sampled at most of the foreground and the number of background points, also means that the most sampling pairs for the kg*kg group, usually this value can be taken as 4 or more, the paper suggested that 4 can be, the larger the program is more time-consuming.

In this way, for each unknown point, starting from that point, the kg path is drawn, the angle of 360/kg between each path is recorded, and the first encountered foreground or back spot in the route passing through each path is noted until the edge of the image is exceeded.

For the sake of the stability of the algorithm, every 3*3 in the rectangular region (4*4 or 5*5), the starting angle of the path changes periodically, so that the adjacent pixels of the kg path through the region has a larger difference can be more efficient result set.

As can be seen, in many cases, the foreground and background of unknown points can not reach the number of kg, or even in extreme cases, no one can find a sampling point, so that the point can not be used for the calculation of transparency, which depends on the process behind.

Foreground sampling point Quantity distribution background sampling point Quantity distribution foreground + background sampling point Quantity distribution

The number of sampling points of each part of the unknown area in the previously enumerated trimap is plotted, the closer the color is to the white, indicating that the larger the number of sampling points, it can be seen that some unknown points in the corner of the image are not particularly ideal, but basically did not take a sample of the situation, So we're looking at the results of the scribbles picture.

Foreground sampling point Quantity distribution background sampling point Quantity distribution foreground + background sampling point Quantity distribution

In particular, the results of the foreground sampling distribution seem unsatisfactory, and some parts of the sample count are 0, as discussed below.

After the sample calculation is completed, we need to find out which of these sampling points is the best combination, this time involves the general optimization often talked about the objective function, in this paper, the objective function is calculated by the product of four small functions, respectively, as follows:

1:

which

For the sake of comprehensiveness, we will list the formula Αp in the equation:

Formula (2) The reason is very obvious, with a pair of f/b to calculate the alpha value if very reasonable, then the alpha-binding f/b to recalculate the color should be the same as the original color gap is very small. The formula (3) shows that in a certain field, the average value of the difference should be very small because the pixels will not have a mutation in general.

For the convenience of understanding, I posted part of the code to calculate α:

/// <summary>///The corresponding alpha value is calculated by the color values of the current point, the front spot, and the background point, corresponding to the formula of the paper (12)./// </summary>/// <param name= The BGR Color component value of the current point, BC, GC, RC >.</param>/// <param name= BGR Color component values for "BF, GF, RF" > Pre-Attractions.</param>/// <param name= The BGR Color component value of the "BF, GF, RF" > Background points.</param>/// <remarks>alpha will appear not in the [0,1] interval and therefore need to be suppressed.</remarks>Double Calcalpha (int BC,int GC, int RC, int BF, int GF, int RF, int BB, int GB,  int RB) {double Alpha = (double) ((BC-BB) * (BF-BB) + (GC-GB) * (GF-GB) + (RC-RB) * (RF-RB))/((BF-BB) * (BF-BB) + (GF-GB) * (GF-GB) + (RF-RB) * (RF-RB) + // here 0.0000001 for EPS in the Localsmooth phase seems to be wrong, there is abnormal noise generated  if (Alpha > 1) Alpha = 1; else if (Alpha < 0) Alpha = return Alpha;}             

2: The author considers the unknown point to the foreground of the sampling and the back of the line between the location of the path, should try to minimize the mutation of the pixel, for example, if the path needs to pass through the edge of the image, you should design a function to make the return value of the function is larger, so the author uses the following formula:

On the way to the pixel color along the path of integration, after discretization is some accumulation, CSDN provided code in the process of the function is wrong, because his last judgment condition makes the loop only once, interested friends can change their own.

According to the meaning of equation (4), the possibility that an unknown point belongs to the foreground can be represented by the following formula:

、

And a good combination should also minimize the following formula:

3, the unknown point and the physical distance between the former attractions, a good combination of the foreground point should be as close as possible to the unknown point;

4, unknown points and the physical distance between the scenic spots, a good combination of background points should be as close as possible to the unknown point;

By combining these four conditions, you end up with the following objective function:

The index data for each sub-item can be found in the paper itself.

According to this request, the previous sampling to get the data processed, and recorded the smallest of the pair of combinations, the initial determination of the best sampling point.

In fact, at this time we can initially obtain the processed α value, for example, we said earlier Trimap, its original image and after the sample and gather processing results such as:

From the processing results, it has been possible to get a rough treatment of the effect.

2.3, Re?nement

After preliminary gather processing, as stated earlier, the results are not sufficiently delicate, and some unknown points cannot be processed due to the fact that the sampling process does not collect valid foreground and background data, so this problem needs to be addressed further in the re?nement phase.

The paper puts forward, first of all, in a certain neighborhood, such as the radius of 5 in the field, the first statistical formula (2) corresponding to the lowest MP value of 3 point-related color data, and the weighted average of these data, get the data pair:

The new foreground, background, transparency, and confidence calculations are then calculated according to the following formulas.

The calculation of credibility is prepared for the next part of the smoothing, and he reflects a measure of the reasonableness of the sampling points we have determined in this step, which, by this step, we can get the transparency and composition diagram as follows:

It can be seen that the results obtained in this step are already quite perfect.

2.4 Local Smoothing

This step is really I did not spend too much energy to see, his implementation process is probably a bit similar to Gaussian blur, but there are many other aspects of the processing, a good thing is in the code provided by CSDN for this part of the implementation of each formula is correct, is also complete, therefore, Interested friends need to read more papers and corresponding code.

Third, the effect of the algorithm

According to the relevant information provided by the paper, I collected some of the graphs and matching trimap test some of the results of the algorithm, are now posted as follows:

After the original trimap is synthesized

It can be seen that for these trimap graphs, in many cases it is possible to obtain a more satisfactory effect.

I also found some simple diagrams, which are handled using Scribble, as shown in the following:

Original operation interface result diagram

I chose these are the relatively simple background diagram, therefore also can obtain the more ideal effect, if is the more complex diagram, uses the Scribble is basically obtains the very ideal effect, unless the manual carefully divides the boundary.

Since CSDN provided the code, I do not provide my own writing here. After all, that code is just a prototype.

Provides three sample programs for interested friends to test the effect: Http://files.cnblogs.com/Imageshop/SharedMatting.rar

Basically I do not provide the source code, but I will try to use the text to describe the corresponding algorithm clearly or provide reference documents *******************************

Because by their own efforts and practice to write out the effect is really their own things, people must rely on their own ***************************

Image cutout Algorithm Learning-Shared sampling for real-time Alpha matting