YUV to RGB Conversion Implemented by shader

Ref: http://www.360doc.com/content/11/1008/17/1016783_154377024.shtml

There are many conversion programs from YUV to RGB on the Internet, but they are basically calculated based on the CPU. There are some methods based on CPU computing, the most primitive must be the direct floating-point operation based on the conversion formula. To increase the speed, you can use the left and right Shifts operation to convert the floating-point operation into an integer operation, which will multiply the conversion speed. In addition, you can also use the look-up table method, because YUV is between 0 and ~ Between 255, they always have a range. In this case, a large search table is formed, and each YUV component is directly searched for the RGB value. Of course, this search table will be very large, you can use the partial search method to narrow down the table size.

A good way to use CPU computing is to use the CPU's SSE command. A special term is used to describe SSE, which is "single command stream multi-data stream ", it means that he can calculate multiple data at a time. Of course, the speed is very fast.

Here, I mainly write about how to use shader for conversion. First, I will introduce YUV:

The YUV format introduction in Windows is very comprehensive on msdn. There are many YUV formats, with 444,422,420 points, and each specific format has many different standards. Here is a detailed description of msdn:
Http://www.microsoft.com/china/MSDN/library/enterprisedevelopment/softwaredev/VideoRende8BitYUV.mspx? MFR = true

However, for writing programs, I only care about the memory layout in each format.
YUV is divided into packed and planar. In the flat format, each YUV component is stored separately. Generally, all Y components are stored first, all U components are stored, and all V components are stored. In the packaging method, each component of YUV is not stored separately, but stored together. For example:

Packaging format (image from msdn), yuy2 format in 422:

Flat format, imc4 format in 420:

I used yuv420 imc4 format, which is a flat format. The feature of this format is to store all Y components first, and the total number of pixels is the number of Y components. Then, the U component is stored, and the V component is stored. Both the u and v occupy only 1/4 of the total pixels. Therefore, the memory space occupied by one frame in the yuv420 format can be calculated as follows:
M_frameheight; // video height
M_framewidth; // video width
M_imgsize = m_framewidth * m_frameheight; // width * Height
M_framesize = m_imgsize + (m_imgsize> 1); // the size of a YUV video frame. The right shift is equivalent to/2.

Therefore, the memory address of each component can be calculated as follows:
Unsigned char * ctemp [3];
Ctemp [0] = m_yuv + m_framesize * n; // y address
Ctemp [1] = ctemp [0] + m_imgsize; // U address
Ctemp [2] = ctemp [1] + (m_imgsize> 2); // v component address
M_yuv is the first address of the video, that is, the pointer to the video; n is the sequence number of the current frame.
Knowing the addresses of the YUV components of each frame, it is easier to convert them. There are many formulas for conversion. The conversion formula I use is as follows:
R = Y + 1.4022 * (V-128)
G = Y-0.3456 * (u-128)-0.7145 * (V-128)
B = Y + 1.771 * (u-128)

The shader is used for calculation according to this conversion formula. Since I was just getting started with shader programming, I am not very familiar with shader's working methods, I only program according to my own understanding.
First of all, when running the shader program, the video card is computed in parallel by many pixels, so the conversion speed is very fast. However, I found that the shader only knows his own pixel information for each pixel, including his own color (RGB), depth, texture coordinates, and so on. He does not know anything about other pixels. For example, a red pixel only knows what color he should display in red, not to mention the distance between pixels. Therefore, he can only use his own information for computation. In this case, a problem occurs. Because each component of YUV is stored in a long distance, a corresponding YUV component data is obtained in a shader program, adjust the data arrangement format of the memory. I used to adjust the video data format to yuvyuvyuv... in this way, the YUV data is first sent to the texture by pretending to be RGB data, and the video card is cheated. Then, the shader uses the conversion formula for calculation. Because the data range of the YUV component is also 0 ~ Between 255, so this method can be successful.
(This may not be true. I think that when running a shader, each pixel should have a way to get pixel information other than itself, however, I have only one week to learn shader, and I may not know the method. If there is such a method, the conversion efficiency will be higher, because I don't need to spend any time in the memory to arrange the data. But this may be very clear about how shader works. I will be familiar with it later and write a new program .)
The first step is the arrangement process, which occurs in the memory:
For (y = 0; y <m_frameheight; y ++)
{
For (x = 0; x <m_framewidth; X ++)
{
L = y * m_framewidth + X;
M = (y/2) * (m_framewidth/2) + X/2;
M_data [3 * l] = ctemp [0] [l]; // y
M_data [3 * L + 1] = ctemp [1] [m]; // U
M_data [3 * L + 2] = ctemp [2] [m]; // v
}
}
The arrangement method is very simple, as long as you understand the above memory layout structure, it is easy to know.
Then, the sorted YUV data is sent to the texture, and the shader program is run before each frame is rendered. The shader program code is as follows:
Vertex shader:
Void main ()
{
Gl_texcoord [0] = gl_multitexcoord0;
Gl_position = ftransform ();
}
Fragment shader:
Uniform sampler2d Tex;
Void main ()
{
Vec4 YUV = texture2d (Tex, gl_texcoord [0]. St );
Vec4 color;
Color. r = YUV. R + 1.4022 * YUV. B-0.7011;
Color. r = (color. r <0.0 )? 0.0: (color. r> 1.0 )? 1.0: color. R );
Color. G = YUV. R-0.3456 * YUV. G-0.7145 * YUV. B + 0.53005;
Color. G = (color. G <0.0 )? 0.0: (color. G> 1.0 )? 1.0: color. G );
Color. B = YUV. R + 1.771 * YUV. G-0.8855;
Color. B = (color. B <0.0 )? 0.0: (color. B> 1.0 )? 1.0: color. B );
Gl_fragcolor = color;
}
Why is there a constant like-0.7011, 0.53005, and 0.8855? It is because the color data has been converted into a floating point number in the form of a value ranging from 0.0 ~ So the coefficient after the conversion formula also needs to change accordingly.