Yuv/rgb format and fast conversion algorithm

1 Preface

In order to give color a quantitative measure, it is necessary to establish a color space model to describe a variety of colors, because people's perception of color is a complex process of physiological and psychological joint action, so in different fields of application in order to better and more accurately meet their needs, There are a variety of color space models to quantify the descriptive colors. We are more often exposed to include rgb/cmyk/yiq/yuv/hsi and so on.

For the Digital electronic multimedia field, we often touch the concept of color space, mainly RGB, YUV two (in fact, these two systems contain a number of specific color representations and models, such as sRGB, Adobe RGB, YUV422, YUV420 ... ), RGB is based on the principle of three-color plus light system to describe the color, and YUV is based on the brightness, chromatic aberration principle to describe the color.

Even if only RGB YUV these two categories of color space, the knowledge involved is also very rich and complex, self-awareness does not have enough relevant expertise, so this article mainly for the application of engineering and algorithm to discuss.

2 YUV-related color space model 2.1 YUV and YIQ YCRCB

For YUV models, we actually confuse it with the YIQ/YCRCB model in many cases.

In fact, the YUV model is used in the PAL TV system, y means brightness, and UV is not the abbreviation of any word.

The YIQ model is similar to the YUV model for TV systems in NTSC format. The I and Q components in the Yiq color space are equivalent to a 33 degree rotation of the UV components in the YUV space.

YCbCr color space is a color space derived from YUV color space, which is mainly used in digital TV system. In the conversion from RGB to YCBCR, the input and output are in 8-bit binary format.

The conversion equations for the three and RGB are as follows:

RGB--YUV:

is actually:

Y=0.30R+0.59G+0.11B, u=0.493 (b-y), v=0.877 (r-y)

RGB-YIQ:

RGB-YCRCB:

One of the key points to understand from the formula is that the UV/CBCR signal is actually the blue difference signal and the red difference signal, and then, in fact, to a certain extent indirectly represents the intensity of blue and red, understanding this point for our understanding of various color transformation process will be a great help.

The YUV format we talked about in digital electronic multimedia is, in fact, a family of color models with multiple storage formats based on the YCRCB color space model (including yuv444/yuv422/yuv420/yuv420p, etc.). It is not a YUV model for PAL analogue TV in the traditional sense. The differences between these YUV models are mainly in the way of sampling and storage of UV data, which is not detailed here.

In camera sensor, the most commonly used YUV model is the YUV422 format, since it uses 4 bytes to describe two pixels, which is better compatible with the RGB565 model. Facilitates the hardware and software interface design of camera sensor and camera controller.

3 Yuv2rgb Fast Algorithm analysis

This refers to the YUV is actually YCRCB, Yuv2rgb conversion formula itself is very simple, but involves floating-point operations, so, if you want to implement a fast algorithm, the structure of the algorithm itself is not good to study, mainly by the use of integer operation or check table to speed up the calculation.
The conversion formula can be deduced first:

R = Y + 1.4075 * (V-128)
G = y–0.3455 * (u–128) –0.7169 * (v–128)
B = Y + 1.779 * (u–128)

3.1 Integral type algorithm

To use integer operations instead of floating-point arithmetic, of course, we have to use the shift method, we can easily get the following algorithm:

U = Yuvdata[upos]-128;
v = yuvdata[vpos]-128;

Rdif = v + ((v * 103) >> 8);
Invgdif = ((U *) >> 8) + ((v * 183) >> 8);
BDIF = U + ((u*198) >> 8);

r = Yuvdata[ypos] + rdif;
g = Yuvdata[ypos]-invgdif;
b = Yuvdata[ypos] + bdif;

In order to prevent overflow, it is also necessary to determine whether the results of the calculation in the range of 0-255, do similar to the following judgment.

if (r>255)
r=255;
if (r<0)
r=0;

To convert from RGB24 to RGB565 data, you also do the shift and/or operation:

RGBDATA[1] = ((R & 0xF8) | (G >> 5));
Rgbdata[0] = (((g & 0x1C) << 3) | (b >> 3));

3.2 Part Check Table method

The first thing the tabular method can think of is to replace the multiplication in the integer algorithm with the look-up table.

Rdif = Fac_1_4075[u];
Invgdif = Fac_m_0_3455[u] + fac_m_0_7169[v];
BDIF = Fac_1_779[u];

A total of 4 1-dimensional arrays are required here, with subscripts starting from 0 to 255, and the table occupying approximately 1K of memory space. UV can not need to do minus 128 of the operation. It is OK to calculate the value of the corresponding array element in advance.

For each pixel, partial look-up table replaces 2 subtraction operation with 4 multiplication and 4 shift operation. However, multiple addition operations and 6 comparison operations and possible assignment operations are still required, and the speed of operation of the first method is not obvious.

3.3 Full Table Check method

Then whether the corresponding RGB value can be obtained by the YUV Direct check table? At first glance seems unlikely, with the most complex G operation as an example, because G and YUV three are related, so similar to g=yuv2g[y][u][v] such an algorithm, a three-dimensional subscript size is 256 of the array will need to occupy 2 of 24 square 16 trillion space, absolutely unacceptable. Therefore, most of the current use of partial check table method.

However, if we analyze carefully, we can find that for g we do not really need to adopt a three-dimensional array, because Y is only related to the results of the UV operation, and the individual of the UV, so we can use two times to look up the table method to simplify the operation of the G to two two-dimensional array table operation, as follows:

G = yig2g_table[y [uv2ig_table[u] [v]];

And RB itself is only related to Yu or yv, so we need 4 8*8 two-dimensional table, need to occupy 4 times 2 16 square 256K memory. Basically acceptable. But for the embedded use of mobile phones, it is slightly larger.

Further analysis, we can see, because in the mobile phone and other embedded applications we ultimately want to convert the data into RGB565 format to send to the LCD screen display, so, for the RGB three components, we do not need 8bit so high precision, for the sake of simplicity and the unification of operations, For each component we actually only need 6bit of data is enough, so we can further change the table to 4 6*6 two-dimensional table, so that only need to occupy 16K memory! The final overflow judgment can also be done in advance when calculating the table element values. The final algorithm is as follows:

y = (Yuvdata[y1pos] >> 2);
U = (Yuvdata[upos] >> 2);
v = (Yuvdata[vpos] >> 2);

r = yv2r_table[y [v];
g = yig2g_table[y [uv2ig_table[u] [v]];
b = yu2b_table[Y] [u];

RGBDATA[1] = ((R & 0xF8) | (G >> 5));
Rgbdata[0] = (((g & 0x1C) << 3) | (b >> 3));

In this way, we added 3 shift operations to the relative partial table method, and further reduced 4 addition operations and 6 comparison assignments.

When calculating the values of the table elements, it is necessary to consider the rounding and offset factors so that the intermediate result of the calculation satisfies the non-negative requirements of the array subscript.

With the full-look-up table method, compared with the first algorithm, the final operation speed can be significantly improved, the specific performance can be increased, depends on the platform's CPU speed and memory access speed of the relative ratio. The faster the memory access speed, the more obvious performance improvement is caused by the table-checking method. The performance of the results tested on my PC is about 35% higher. On an arm platform, the test only increased by about 15%.

3.4 Further thinking

In fact, the above algorithm:

RGBDATA[1] = ((R & 0xF8) | (G >> 5));
Rgbdata[0] = (((g & 0x1C) << 3) | (b >> 3));

(R & 0xF8) and (b >> 3) Calculations can also be calculated in advance in the table. In addition, the value of the YU/YV can not actually cover the full range of 6*6, some of the points in the middle is never get input, RB's operation may also consider using the 5*5 table. These may further increase the speed of the operation and reduce the size of the table.

In addition, in the embedded application, if it is possible to put the table in high-speed memory such as SRAM should be more than in the SDRAM to play the advantages of the table-checking method.

4 RGB2YUV?

At present, it is impossible to calculate the table of 3-dimensional table to simplify the calculation of 2-dimensional table table. The multiplication can only be substituted by partial table-check method.

In addition, in most cases, we need to YUV2RGB conversion, because the data obtained from the sensor usually we use YUV data, in addition to JPG and MPEG is actually based on the YUV format encoding, so to display the decoded data need is also YUV2RGB operation. Here's a list of articles from DirectShow

Computer color display the principle of color display and color TV, is the use of R (Red), G (Green), B (Blue) additive blending principle: By emitting three different intensity of the electron beam, so that the inside of the screen covered with red, green, blue phosphorescent material to produce color. The representation of this color is called the RGB color space representation (it is also the most used color space representation method in Multimedia computer technology).
According to the three-color principle, any kind of shade F can be mixed with different components of the R, G, b three-color addition.

F = R [r] + G [g] + b [b]

Wherein, R, G, B are three primary colors to participate in the mixing coefficient. When the tri-primary component is 0 (weakest), the mixture is black, and when the Tri-primary component is K (strongest), it is blended into white light. Adjust the values of the R, G, and b three coefficients to mix a variety of shade between black and white light.
So where does YUV come from? In modern color TV system, usually use three-tube color camera or color CCD camera to camera, and then the color image of the captured signal by color separation, amplification and correction to get RGB, and then through the matrix transformation circuit to obtain the luminance signal Y and two chromatic aberration signal r-y (that is U), b-y (that is, V), The final transmitter encodes three signals of brightness and chromatic aberration, which are sent out using the same channel. This color is represented by the so-called YUV color space representation.
The importance of using YUV color space is that its luminance signal y and chroma signal u, v are separated. If only the Y signal component does not have a U, v component, then the image represented here is a black-and-white grayscale image. Color TV using YUV space is precisely to use brightness signal y to solve the compatibility problem between color TV and black and white TV, so that black and white machine can also receive color TV signal.
The formula for conversion between YUV and RGB is as follows (RGB values range from 0-255):

Y = 0.299R + 0.587G + 0.114B
U = -0.147r-0.289g + 0.436B
V = 0.615r-0.515g-0.100b

R = Y + 1.14V
G = y-0.39u-0.58v
B = Y + 2.03U

In DirectShow, the common RGB format is RGB1, RGB4, RGB8, RGB565, RGB555, RGB24, RGB32, ARGB32, etc. common YUV formats are YUY2, YUYV, Yvyu, UYVY, AYUV, y41p, Y411, Y211, IF09, IYUV, YV12, YVU9, YUV411, YUV420, etc. As an auxiliary description type (subtype) of the video media type, their corresponding GUID is shown in table 2.3.

Table 2.3 Common RGB and YUV formats

GUID Format Description
MEDIASUBTYPE_RGB1 2 colors, each pixel is represented by 1 bits, need palette
MEDIASUBTYPE_RGB4 16 colors, each pixel is represented by 4 bits, need palette
MEDIASUBTYPE_RGB8 256 colors, each pixel is represented by 8 bits, need palette
mediasubtype_rgb565 each pixel is represented by 16 bits, the RGB component uses 5-bit, 6-bit, 5-bit
mediasubtype_rgb555 each pixel is represented by 16 bits, and the RGB component uses 5 bits (the remaining 1 bits are not used)
Mediasubtype_rgb24 each pixel is represented by 24 bits, and the RGB component uses 8 bits each
Mediasubtype_rgb32 each pixel is represented by 32 bits, the RGB component uses 8 bits (the remaining 8 bits are not used)
Mediasubtype_argb32 each pixel is represented by 32 bits, the RGB components each use 8 bits (the remaining 8 bits are used to represent the alpha channel value)
MEDIASUBTYPE_YUY2 YUY2 format, packaged in 4:2:2 manner
MEDIASUBTYPE_YUYV yuyv Format (the actual format is the same as YUY2)
Mediasubtype_yvyu yvyu format, packaged in 4:2:2 manner
Mediasubtype_uyvy UYVY format, packaged in 4:2:2 manner
MEDIASUBTYPE_AYUV 4:4:4 YUV format with alpha channel
MEDIASUBTYPE_Y41P y41p format, packaged in 4:1:1 manner
mediasubtype_y411 Y411 Format (the actual format is the same as Y41P)
mediasubtype_y211 Y211 Format
MEDIASUBTYPE_IF09 IF09 Format
MEDIASUBTYPE_IYUV IYUV Format
MEDIASUBTYPE_YV12 YV12 Format
MEDIASUBTYPE_YVU9 YVU9 Format

The following is a description of each RGB format.

¨RGB1, RGB4, RGB8 are both color palette-type RGB formats, and when describing the format details of these media types, a palette (defining a series of colors) is typically followed by the Bitmapinfoheader data structure. Their image data is not a true color value, but rather the index of the current pixel color value in the palette. Take RGB1 (2-color bitmap) as an example, for example, the two color values defined in its palette are 0x000000 (black) and 0xFFFFFF (white), then the image data 001101010111 ... (1 bits per pixel) indicates the color of the corresponding pixels: black black and white black white

The ¨rgb565 uses a 16-bit representation of a pixel, 5 bits in 16 bits for R, 6 bits for g, and 5 bits for B. The program typically uses a single word (word, a word equal to two bytes) to manipulate a pixel. When a pixel is read, the meanings of each bit of the word are as follows:
High byte low byte
R-R r r R G G G G G G B b b B. b b
You can use the mask and shift operations together to get the values of the RGB components:

#define Rgb565_mask_red 0xf800
#define Rgb565_mask_green 0X07E0
#define Rgb565_mask_blue 0x001f
R = (Wpixel & rgb565_mask_red) >> 11; Value Range 0-31
G = (Wpixel & rgb565_mask_green) >> 5; Value Range 0-63
B = Wpixel & rgb565_mask_blue; Value Range 0-31

¨RGB555 is another 16-bit RGB format, and the RGB components are represented by 5 bits (the remaining 1 bits are not used). After reading a single pixel with one word, the meanings of each bit of the word are as follows:
High byte low byte
X r r r R G G G G G B b b b b (x indicates no, can be omitted)
You can use the mask and shift operations together to get the values of the RGB components:

#define Rgb555_mask_red 0X7C00
#define Rgb555_mask_green 0X03E0
#define Rgb555_mask_blue 0x001f
R = (Wpixel & rgb555_mask_red) >> 10; Value Range 0-31
G = (Wpixel & rgb555_mask_green) >> 5; Value Range 0-31
B = Wpixel & rgb555_mask_blue; Value Range 0-31

¨rgb24 uses 24 bits to represent a pixel, and the RGB component is represented by 8 bits, with a value range of 0-255. Note in memory that the RGB components are arranged in the order of: BGR BGR BGR .... You can usually use the Rgbtriple data structure to manipulate a pixel, which is defined as:

typedef struct TAGRGBTRIPLE {
BYTE Rgbtblue; Blue Component
BYTE Rgbtgreen; Green component
BYTE rgbtred; Red component
} rgbtriple;

The ¨RGB32 uses 32 bits to represent a pixel, the RGB components are 8 bits each, and the remaining 8 bits are used as alpha channels or not. (ARGB32 is a RGB32 with an alpha channel.) Note that in-memory RGB components are arranged in the order of: BGRA BGRA BGRA .... You can usually use the Rgbquad data structure to manipulate a pixel, which is defined as:

typedef struct TAGRGBQUAD {
BYTE Rgbblue; Blue Component
BYTE Rgbgreen; Green component
BYTE rgbred; Red component
BYTE rgbreserved; Reserved bytes (used as alpha channel or ignored)
} Rgbquad;

Various YUV formats are described below. The YUV format typically has two main classes: the packaged (packed) format and the planar format. The former stores YUV components in the same array, usually several adjacent pixels to form a macro pixel (Macro-pixel), and the latter uses three arrays to store the YUV three components separately, as if it were a three-dimensional plane. The YUY2 to Y211 in table 2.3 are packaged formats, and IF09 to YVU9 are flat formats. (Note: In the introduction of various specific formats, the YUV components will have subscript, such as Y0, U0, V0 represents the YUV component of the first pixel, Y1, U1, V1 represents the YUV component of the second pixel, and so on. ）

The ¨yuy2 (and YUYV) format retains the Y component for each pixel, while the UV component is sampled every two pixels in the horizontal direction. A macro pixel is 4 bytes, which actually represents 2 pixels. (4:2:2 means that there are 4 y components, 2 U-components, and 2 v components in a macro pixel.) The YUV components in the image data are arranged in the following order:
Y0 U0 Y1 V0 Y2 U2 Y3 V2 ...

The ¨yvyu format is similar to YUY2, except that the order of YUV components in the image data is different:
Y0 V0 Y1 U0 Y2 V2 Y3 U2 ...

The ¨uyvy format is similar to YUY2, except that the order of YUV components in the image data is different:
U0 Y0 V0 Y1 U2 Y2 V2 Y3 ...

The ¨AYUV format has an alpha channel, and the YUV component is extracted for each pixel, and the image Data format is as follows:
A0 Y0 U0 V0 A1 Y1 U1 V1 ...

The ¨y41p (and Y411) format retains the Y component for each pixel, while the UV component is sampled every 4 pixels in the horizontal direction. A macro pixel is 12 bytes, which actually represents 8 pixels. The YUV components in the image data are arranged in the following order:
U0 Y0 V0 Y1 U4 Y2 V4 Y3 Y4 Y5 Y6 Y8 ...

The ¨y211 format samples the Y component in a horizontal direction every 2 pixels, and the UV component is sampled every 4 pixels. A macro pixel is 4 bytes, which actually represents 4 pixels. The YUV components in the image data are arranged in the following order:
Y0 U0 Y2 V0 Y4 U4 Y6 V4 ...

The ¨YVU9 format extracts the y component for each pixel, and when the UV component is extracted, the image is first divided into several 4 x 4 macro blocks, and then each macro block extracts a U component and a V component. When the image data is stored, the first is the Y-component array of the entire image, followed by the U-component array, and the V-component array. The IF09 format is similar to YVU9.

The ¨IYUV format extracts the y component for each pixel, and when the UV component is extracted, the image is first divided into several 2 x 2 macro blocks, and then each macro block extracts a U component and a V component. The YV12 format is similar to IYUV.

¨yuv411, YUV420 format is found in DV data, the former is used in NTSC system, the latter is used in PAL system. The YUV411 extracts the Y component for each pixel, and the UV component is sampled every 4 pixels in a horizontal direction. The YUV420 is not a V-component sample of 0, but rather a YUV411 in the horizontal direction than the sampling frequency, in the vertical direction of the u/v interval to reduce the half chromatic aberration sampling, 2.12 shows.

Yuv/rgb format and fast conversion algorithm