Visualization: An algorithm for image theme color extraction

Visualization is one of the hottest areas of cloud application. Gathered countless Daniel, small and innovative enterprise representatives. This paper, from the Stanford visualization group led by Pat Hanrahan and Jeffrey Heer two-bit visualization, focuses on sharing the image's theme-Color extraction algorithm.

The visualization and Visual analysis Group of CAD&CG State Key Laboratory of Zhejiang University in particular, the paper was collated, the following is a summary of the article.

Stanford Visualization Group is very necessary to introduce, the leading two Daniel is Professor Pat Hanrahan, across the scientific visualization and information visualization two areas, even if not know the name that this year red-hot data visualization listed companies tableau should know, he is the co-founder, Tableau was born in his Polaris system; another Jeffrey Heer is the field of information visualization and human-computer interaction in recent years, the popular fried chicken, the paper takes into account the innovative and practical, stunning.

Back to the point, this article solves the problem of color extraction of the theme of the image, which belongs to the topic of color modeling. The paper in the same year in Eurovis, Chi and Siggraph have published a color modeling article (Siggraph thesis is now in conditionally accepted state), Eurovis and Chi are one of the best papers, it is really thick fat.

This is really back to the point, back to this paper. The general theme color extraction methods include K and fuzzy C-means clustering method of pixel color value and the method of peak color histogram. In fact, the idea of the paper is not complicated, the image defines a series of characteristics, apply multivariate linear regression model lasso, set up the task collection training set on the platform Amazon Turkish robot, lasso the weight of the key feature to reduce the influence of the redundancy characteristic through the learning of training set. So as to generate a better theme color extraction model. The following descriptions describe the feature definition, regression model, and user study three parts respectively.

In fact, the concept of the theme of color really is the public right, the woman said that the woman is reasonable, judge an image of the theme color, 1000 reader can not get the same answer, but their answers are mostly approximate. Therefore, it is reasonable to use the user-defined theme color as the standard answer. For each image, the article uses the K method to calculate the 40 colors of the image as the base color of the k=40. The user can only select 5 colors from these 40 colors as the theme color of the image.

This article defines the following 6 aspects of the characteristics of the extraction of the calculation of 79 feature variables, here for a simple description:

Visual significance saliency: This paper defines the visual significance of each pixel in the image by the user's eye movement tracking data. To define the visual significance of a set of theme colors in the image is the superposition of the visual significance of the pixels of all theme colors, while defining the ratio of the pixel to the number of pixels for a color visual significant density.

Coverage error coverage error: Coverage error is defined as the theme color to cover the image and the original image of the color error, hard error and soft error, the difference is whether a pixel is covered by a single theme color or by a number of theme color overlay. Similarly, the coverage error of pixels in color channels such as brightness, saturation, red-green, blue-yellow and so on is defined, and the coverage error is computed by region after the image is segmented.

Colour Diversity Color Diversity: Color diversity takes into account the average, maximum, and minimum distance between colors.

Color impurity: The color concentration considers the distance between the first 5% pixels that are similar to the theme color.

Color nameability and color statistics colour statistics: These two sounds intuitive, actually very vague, the text is not described in detail.

After defining these 79 features, it will be lasso to play. LASSO (least differs shrinkage selection operator) is a multivariate linear regression method, which achieves the feature selection through a constraint condition in the traditional multivariate linear regression equation (the following figure is excerpted from the LASSO text). where x is the feature, β is the weight of the feature, if the constraint T is an infinite value, then with the general multiple linear regression is not different, but t gradually reduce the time the feature weight received extrusion (shrinkage), so as to remove the redundancy feature selection (selection) role. The 79 features defined by the lasso approach to the training set are reduced to very limited.

There is a more detailed introduction to the idea and development of this method.

User study is the author set up 40 images on this crowdsourcing platform, each user receives 10 images of the task, in the base color to find the image of the 5 theme colors. In contrast, the author also found 11 students of the art department to perform the same task.

The following figure is an image of the user study results statistics, you can see that the choice of theme colors and art students are still similar, but with the automatic selection of the color difference is larger.

For modeling the theme color, the author and the user study to identify whether and the real theme of the image, by the user to score the way to judge, give the topic to give 1 to 5 points, 5 points is very close and 1 points is very close. From the figure below, you can see that modeling gets the color and the scoring of the user group is well received (the average in the upper-left corner), while the other two methods are slightly inferior.

Finally look at the comparison of the modeling of the new image: You can see that the article method can extract some of the pixel coverage is not very high, but visually significant areas, such as the white Butterfly and the Red sun at sea and so on.

Finally, the weight of these 79 features is given, which seems to be used in a kind of image retrieval based on theme color. But in fact, because the visual significance is obtained by the user's eye movement tracking data, it is impossible to model the image without visual significance, which greatly reduces the usability. If this feature is improved, the method can be used more widely.

A few additions:

1. This post is an academic review of the article, which has been reported before.

2.Eurovis of the article for the data entity itself color semantics and design of the corresponding color consistency problem, such as the fruit of the data, the blueberry blue, banana with yellow and so on, interested in the reader can see the original paper.