Image segmentation with color space

Original address: https://realpython.com/python-opencv-color-spaces/

• What is color space?
• Use color space for simple segmentation
• Color space and reading images using OPENCV
• Visualize clown fish in RGB color space
• Visual clown fish in HSV color space
• Select Range
• Can this division be generalized to the relatives of clownfish?
• Summarize

This may be an era of deep learning and big data, where complex algorithms analyze images by displaying millions of of images, but the color space is still very useful for image analysis. The simple method is still powerful.

In this article, you will learn how to use OpenCV to simply segment objects from an image based on colors in Python. OpenCV is a popular computer vision library, written in C/s + +, with Python bindings, providing a simple way to manipulate color space.

While you don't need to be familiar with OPENCV or the other helper packages used in this article, let's say you have at least a basic understanding of the coding in Python.

What is color space?

In the most common color spaces RGB (red, green, blue), the color is represented by its three components: red, green, and blue. In more specialized terms, RGB describes color as a tuple of three components. Each component can take a value between 0 and 255, where the tuple (0,0,0) represents black and (255,255,255) represents white.

RGB is considered to be an additional color space, the color can be imagined as a large number of red, blue and green light on the black background to produce, the following are some examples of RGB color:

RGB is one of five main color space models, each with a number of branches. There are so many color spaces, because different color spaces are useful for different purposes.

In the field of printing,CMYK is useful because it describes the color combinations that are required to produce colors from a white background. The 0-tuple in RGB is black, while the 0-tuple in CMYK is white. Our printers include cyan, magenta, yellow, and black cartridges.

In some types of medical care, slides with stained tissue samples are scanned and saved as images. They can be analyzed in the HED space, where HED space is the type of staining applied to the original tissue-the representation of the Mu Jing, Shu-red, and dab--saturation.

HSV and HSL are descriptions of hue, saturation, and brightness, and are particularly useful for identifying the contrast in an image. These color spaces are often used for color selection tools in software and web design.

In fact, color is a continuous phenomenon, which means there is an infinite number of colors. However, it is acceptable that the color space is represented by a discrete structure (a fixed number of integer values), because the human eye and perception are also limited. The color space perfectly represents all the colors we can distinguish.

Now that we understand the concept of color space, we can continue to use them in OpenCV.

Use color space for simple segmentation

To demonstrate the color space segmentation technique, we have provided a Nemo image dataset in the Real-python material library for you to download and play. Clown fish are easily identified by their bright orange, so they are good candidates for segmentation. Let's see how accurate it is to find Nemo in a picture.

The key Python package you need to follow is the most important scientific calculation package in Numpy-python, the matplolib-drawing library, and of course OPENCV.

Color space and reading images using OPENCV

First, you need to set up your environment. This article assumes that Python 3.x is installed on your system. Note that although the current version of OPENCV is 3.x, the name of the package to be imported is still CV2 and is pip3 install opencv-python installed by command (no PIP3, with PIP).

>>> import cv2

After successfully importing OPENCV, you can view all the color space transformations provided by OPENCV and save them all in a variable:

>>> flags = [i for i in dir(cv2) if i.startswith(‘COLOR_‘)]

Depending on your version of OPENCV, the list and number of flags may be slightly different, but there will be a lot of flags anyway! See how many flags you have available:

>>> len(flags)258>>> flags[40]‘COLOR_BGR2RGB‘

The first character after the COLOR_ represents the original color space, and the character after 2 represents the target color space. This flag represents the conversion from BGR (blue, green, red) to RGB. As you can see, these two color spaces are very similar, with only the first and last channels exchanged.

You need to Matplotlib.pyplot to view the image and need to numpy to process some images. If Matplotlib or numpy are not already installed, you will need to PIP3 install Matplotlib and PIP3 install NumPy before attempting to import:

>>> import matplotlib.pyplot as plt>>> import numpy as np

Now you can load and examine the image. Note that if you are working from the command line or terminal, your image will appear in the pop-up window. If you work on a jupyter notebook or something like that, they will simply show up below. Regardless of your settings, you should see the image generated by the show () command:

>>> nemo = cv2.imread(‘./images/nemo0.jpg‘)>>> plt.imshow(nemo)>>> plt.show()

Hey, Nemo ... Or Dolly? You'll notice that the blue and red channels seem to have been mixed up. In fact, by default, OpenCV reads images in BGR format. You can use Cvtcolor (images, logos) and the flags we see above to solve this problem:

>>> nemo = cv2.cvtColor(nemo, cv2.COLOR_BGR2RGB)>>> plt.imshow(nemo)>>> plt.show()

Now Nemo looks more like himself.

Visualize clown fish in RGB color space

HSV is a good choice for dividing color space by color, but to understand why, let's compare images in RGB and HSV color spaces by visualizing the color distribution of their pixels. The 3D figure shows this very well, with each axis representing a channel in the color space. If you want to know how to make 3D drawings, see the following sections:
To draw this diagram, you also need several matplotlib libraries:

>>> from mpl_toolkits.mplot3d import Axes3D>>> from matplotlib import cm>>> from matplotlib import colors

These libraries provide the functionality required for drawing. You want each pixel to be placed in its place, based on its component, and colored according to its color. Cv2.split () is very handy here; it splits the image into its component channels. These lines of code split the image and set up a 3D drawing:

>>> r, g, b = cv2.split(nemo)>>> fig = plt.figure()>>> axis = fig.add_subplot(1, 1, 1, projection="3d")

Now that you have set up the drawing, you need to set the pixel color. In order to color each pixel according to its true colors, some shaping and normalization is required. It looks messy, but in fact you need to flatten the color of each pixel in the image into a list and normalized it so that they can be passed to the facecolors parameter of Matplotlib scatter ().

Normalization only refers to narrowing the color range from 0-255 to 0-1 according to the requirements of the facecolors parameter. Finally, Facecolors wants a list instead of a numpy array:

>>> pixel_colors = nemo.reshape((np.shape(nemo)[0]*np.shape(nemo)[1], 3))>>> norm = colors.Normalize(vmin=-1.,vmax=1.)>>> norm.autoscale(pixel_colors)>>> pixel_colors = norm(pixel_colors).tolist()

Now we are ready to draw all the components: the pixel position of each axis and its corresponding color, in the format expected by facecolors. You can build a scatter plot and view it:

>>> axis.scatter(r.flatten(), g.flatten(), b.flatten(), facecolors=pixel_colors, marker=".")>>> axis.set_xlabel("Red")>>> axis.set_ylabel("Green")>>> axis.set_zlabel("Blue")>>> plt.show()

From this diagram, you can see that the orange portion of the image spans the red, green, and blue values of almost the entire range. Because part of Nemo extends to the entire plot, it is not easy to split Nemo in RGB space based on the range of RGB values.

Visual clown fish in HSV color space

We see Nemo in the RGB space, so now let's see him in the HSV space and compare it.
As mentioned briefly above, the HSV represents the hue, saturation, and value (or brightness), which is a cylindrical color space. The color or hue is modeled as the angular dimension that rotates around the center vertical axis, which represents the value channel. The value is from dark (0 at the bottom) to light (top 0). The third axis, saturation, defines the depth of the hue, from the least saturated on the vertical axis to the furthest farthest from the center:

To convert an image from RGB to HSV, you can use Cvtcolor ():

>>> hsv_nemo = cv2.cvtColor(nemo, cv2.COLOR_RGB2HSV)

Now, Hsv_nemo Stores Nemo's representation in the HSV. Using the same technique as above, we can look at the image graph in HSV, which shows the image in the same code as in RGB. Note that you use the same pixel_colors variable for pixel coloring because matplotlib want these values to be in RGB:

>>> h, s, v = cv2.split(hsv_nemo)>>> fig = plt.figure()>>> axis = fig.add_subplot(1, 1, 1, projection="3d")>>> axis.scatter(h.flatten(), s.flatten(), v.flatten(), facecolors=pixel_colors, marker=".")>>> axis.set_xlabel("Hue")>>> axis.set_ylabel("Saturation")>>> axis.set_zlabel("Value")>>> plt.show()

In the HSV space, Nemo's Orange is more localized and visually more detached. Oranges are really different in saturation and value, but they are mostly located in small areas on the tonal axis. This is a key point that can be used for fragmentation.

Select Range

Let's Judge Nemo's threshold based on a series of simple oranges. You can select a range by looking at the diagram above or by using the color selection application online, such as this RGB to HSV tool. The swatches selected here are light orange and dark orange, and are almost red:

>>> light_orange = (1, 190, 200)>>> dark_orange = (18, 255, 255)

One simple way to display colors in Python is to make small square images of the desired color and draw them in matplotlib. Matplotlib only interprets colors in RGB, but provides a convenient conversion function for the primary color space so that we can draw images in other color spaces:

>>> from matplotlib.colors import hsv_to_rgb

Then, build a small 10x10x3 square and fill the corresponding color. You can use NumPy to easily fill a square with color:

>>> lo_square = np.full((10, 10, 3), light_orange, dtype=np.uint8) / 255.0>>> do_square = np.full((10, 10, 3), dark_orange, dtype=np.uint8) / 255.0

Finally, by converting them to RGB for viewing, you can draw them together:

>>> plt.subplot(1, 2, 1)>>> plt.imshow(hsv_to_rgb(do_square))>>> plt.subplot(1, 2, 2)>>> plt.imshow(hsv_to_rgb(lo_square))>>> plt.show()

Produce these images and fill them with the selected color:

Once you have the right color range, you can use Cv2.inrange () to try the Threshold Nemo,inrange () with three parameters: image, lower range, and higher range . It returns the image size of the binary mask (Ndarray is 1 and 0), where the value 1 represents the value in the range, and the 0 value represents the value outside the range:

>>> mask = cv2.inRange(hsv_nemo, light_orange, dark_orange)

To add a matte to the top of the original image, you can use Cv2.bittage_and () to make the corresponding value in the Mask 1:

>>> result = cv2.bitwise_and(nemo, nemo, mask=mask)

To see exactly what you've done, let's look at the mask and the top of the original image with a matte:

>>> plt.subplot(1, 2, 1)>>> plt.imshow(mask, cmap="gray")>>> plt.subplot(1, 2, 2)>>> plt.imshow(result)>>> plt.show()

This has captured the orange part of the fish well. The only problem is Nemo has white stripes ... Fortunately, adding a second look at the white mask is very similar to what you've done with Orange:

>>> light_white = (0, 0, 200)>>> dark_white = (145, 60, 255)

Once you have specified a color range, you can see the colors you have selected:

>>> lw_square = np.full((10, 10, 3), light_white, dtype=np.uint8) / 255.0>>> dw_square = np.full((10, 10, 3), dark_white, dtype=np.uint8) / 255.0>>> plt.subplot(1, 2, 1)>>> plt.imshow(hsv_to_rgb(lw_square))>>> plt.subplot(1, 2, 2)>>> plt.imshow(hsv_to_rgb(dw_square))>>> plt.show()

The upper limit I choose here is very blue white, because white has blue color in the shadow. Let's make a second mask to see if it captures Nemo's stripes. You can build a second matte as you would build the first mask:

>>> mask_white = cv2.inRange(hsv_nemo, light_white, dark_white)>>> result_white = cv2.bitwise_and(nemo, nemo, mask=mask_white)>>> plt.subplot(1, 2, 1)>>> plt.imshow(mask_white, cmap="gray")>>> plt.subplot(1, 2, 2)>>> plt.imshow(result_white)>>> plt.show()

Not bad! Now you can combine these masks. Adding two masks together, wherever there is orange or white, produces 1 values, which is exactly what is needed. Let's add a mask together and draw the result:

>>> final_mask = mask + mask_white>>> final_result = cv2.bitwise_and(nemo, nemo, mask=final_mask)>>> plt.subplot(1, 2, 1)>>> plt.imshow(final_mask, cmap="gray")>>> plt.subplot(1, 2, 2)>>> plt.imshow(final_result)>>> plt.show()

Essentially, you have roughly split Nemo in the HSV color space. You'll notice that there are spurious pixels on the split boundary, and if you like, you can use Gaussian blur to clean up small error detection.

Gaussian Blur is an image filter that uses a function called Gauss to transform each pixel in an image. It has the effect of smoothing image noise and reducing detail. Here's how to apply a blur to our image:

>>> blur = cv2.GaussianBlur(final_result, (7, 7), 0)>>> plt.imshow(blur)>>> plt.show()

Whether this division can be generalized to the relatives of clownfish

For fun, let's see how this segmentation technique can be extended to other clown fish images. In this database, six images of Google Nemo are available for public use. The image is in a subdirectory with an index of nemoi.jpg, where I is the index of 0-5. First, load all of Nemo's relatives into a list:

path = "./images/nemo"nemos_friends = []for i in range(6):   friend = cv2.cvtColor(cv2.imread(path + str(i) + ".jpg"), cv2.COLOR_BGR2RGB)   nemos_friends.append(friend)

You can combine all the code above to split a fish into a function that takes an image as input and returns the segmented image. As shown below:

def segment_fish(image):    ‘‘‘ Attempts to segment the clownfish out of the provided image ‘‘‘    # Convert the image into HSV    hsv_image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)    # Set the orange range    light_orange = (1, 190, 200)    dark_orange = (18, 255, 255)    # Apply the orange mask     mask = cv2.inRange(hsv_image, light_orange, dark_orange)    # Set a white range    light_white = (0, 0, 200)    dark_white = (145, 60, 255)    # Apply the white mask    mask_white = cv2.inRange(hsv_image, light_white, dark_white)    # Combine the two masks    final_mask = mask + mask_white    result = cv2.bitwise_and(image, image, mask=final_mask)    # Clean up the segmentation using a blur    blur = cv2.GaussianBlur(result, (7, 7), 0)    return blur

With this useful feature, you can split all the fish:

results = [segment_fish(friend) for friend in nemos_friends]

Let's see all the results by drawing the results in a loop:

for i in range(1, 6):    plt.subplot(1, 2, 1)    plt.imshow(nemos_friends[i])    plt.subplot(1, 2, 2)    plt.imshow(results[i])    plt.show()

In general, this simple segmentation method has succeeded in finding Nemo's most relatives. However, it is clear that the use of specific light and background to separate a Nemo may not be well extended to the division of All Nemo fish.

Summarize

In this tutorial, you've seen several different color spaces, how an image is distributed in RGB and HSV color spaces, and how to use OPENCV to convert and split the range between color spaces.

In summary, you have learned how to use the color space in OpenCV to perform object segmentation in an image, and you want to see its potential in performing other tasks. This segmentation technique is simple, fast, and reliable in the case of lighting and background control, such as in an experimental environment or on a more uniform data set.

Image segmentation with color space