Technology for OpenGL video playback

This document records the technology of OpenGL video playback. In the previous article, we introduced a simple way to use OpenGL to display video. But that's not the essence of OpenGL's display of video technology. Like Direct3D, OpenGL's better way to display video is through textures (Texture). This article describes the techniques that OpenGL uses to display video in a textured manner.

differences in the coordinates and Direct3D coordinates of OpenGL

The coordinates of the textures in OpenGL are not the same as the coordinates in the Direct3D.

In the Direct3D. The texture coordinates are shown in the following figure. The value is 0 to 1. The coordinate system origin is in the upper-left corner.

The object surface coordinates are shown in the figure below. The value is the actual pixel value. The coordinate system origin is in the upper-left corner.

The OpenGL texture coordinates range is 0-1, and the coordinate origin is in the lower-left corner. This is different from Direct3D, although the value of Direct3D texture coordinates is 0-1, but the origin of his coordinates is in the upper-left corner.

In OpenGL, the object surface coordinates range from 1 to 1. The coordinate system origin is in the center position.

the process of OpenGL video display

Knowledge of textures has been documented in the article "simplest AV playback example 4:direct3d playback RGB (via texture)". The concept of textures in OpenGL is essentially equivalent to the concept of textures in Direct3D, so it is no longer duplicated.

This document records the program that plays the pixel data in the yuv420p format. The program in the previous article can also play pixel data in yuv420p format. But they are different in principle. In the previous article, the input yuv420p pixel data was transferred to OpenGL playback after it was converted to RGB data by a normal function. That is, the conversion of pixels is done through the CPU. In this paper, the input yuv420p pixel data is converted to YUV data via shader and transmitted to OpenGL playback. The conversion of pixels is done through the GPU on the graphics card. With this program, you can learn the basics of GPU programming with OpenGL.

Using shader to play video through OpenGL textures (Texture) Generally, the following steps are required:
1. Initialize 1) initialization
2) Create window
3) Set the drawing function
4) Set the timer
5) Initialize Shader
The steps to initialize the shader are much more, mainly divided into 3 steps: Create shader, create program, initialize texture. (1) Create a Shader object 1) write vertex shader and fragment shader source code. 2) Create two instances of shader. 3) Specify the source code for the shader instance. 4) Online compiling Shaer source code. (2) Create a Program object 1) to create the program. 2) Bind shader to program. 3) Link program. 4) Use Porgram. (3) Initialization of texture. Can be divided into the following steps. 1) Define fixed-point array 2) set vertex array 3) Initialize texture 6) Enter message loop

2. Cycle display 1) Set texture
2) Drawing
3) display

The following is a detailed description of the steps to play YUV using the shader through OpenGL textures. Some places and the previous article is repeated, will be relatively simple to mention.
1. Initialization
1) Initialization
Glutinit () is used to initialize the glut library. It is prototyped as follows: void Glutinit (int *argcp, char **argv);

It consists of two parameters: ARGCP and argv. In general, simply pass the ARGC,ARGV in the main () function to it.
Glutinitdisplaymode () is used to set the initial display mode. Its prototype is as follows. void Glutinitdisplaymode (unsigned int mode);

It is important to note that if you use double buffering (glut_double), you need to draw with glutswapbuffers (). If you use single buffering (Glut_single), you need to draw with Glflush ().
When using OpenGL to play video, we can use the following code: Glutinitdisplaymode (glut_double | GLUT_RGB);

2) Create window
Glutinitwindowposition () is used to set the position of the window. You can specify X, y coordinates.
Glutinitwindowsize () Sets the size of the window. You can set the width and height of the window.
Glutcreatewindow () creates a window. You can specify the title of the window.
The above functions are very basic and are not described in detail.   Directly post a sample code: glutinitwindowposition (100, 100);   Glutinitwindowsize (500, 500); Glutcreatewindow ("simplest Video Play OpenGL");

3) Set the drawing function
Glutdisplayfunc () is used to set the drawing function. The operating system calls the function to redraw the form at the necessary moment. Similar to the process of WM_PAINT messages in Windows programming. For example, when a window is moved to the edge of the screen and then moved back, the function is called to redraw the window. Its prototype is as follows. void Glutdisplayfunc (void (*func) (void));

where (*func) is used to specify the redraw function.
For example, when the video is playing, specify the display () function for redrawing: Glutdisplayfunc (&display);

4) Set the timer
When playing a video, you need to play a certain screen (usually 25 frames) per second, so use the timer to call the drawing function to draw the graph every time interval. The prototype of the timer function Gluttimerfunc () is as follows. void Gluttimerfunc (unsigned int millis, void (*func) (int value), int value);

Its parameters have the following meanings: Millis: Timed time, in milliseconds. 1 seconds =1000 milliseconds.

(*func) (int value): The function used to specify the timer call.

Value: Parameters are passed to the callback function. Relatively high-end, no contact.

If you only write a gluttimerfunc () function in the main function, you will see that the function is called only once. It is therefore necessary to write a gluttimerfunc () function in the callback function and invoke the callback function itself. Only in this way can the callback function be called repeatedly.
For example, when the video is playing, specify that the Timefunc () function is called every 40 milliseconds:
Main function: Gluttimerfunc (+, timefunc, 0);

This is then set in the Timefunc () function.       void Timefunc (int value) {display ();   Present Frame Every-ms Gluttimerfunc (0, Timefunc); }

This enables the display () to be called once per 40ms.


5) Initialize shader
The steps to initialize the shader are much more, mainly divided into 3 steps: Create shader, create program, initialize texture. Their steps are as follows.
(1) Create a shader object
Shader is somewhat similar to a compiler for a program. Create a shader can be divided into the following 4 steps: 1) write vertex shader and fragment shader source code.
2) Create two shader instances: Glcreateshader ().
3) Specify the source code for shader instance: Glshadersource ().
4) Online compiling Shaer source code Glcompileshader ().

The following detailed analysis of these 4 steps.
1) write vertex shader and fragment shader source code.
A new language is used here: OpenGL Shader Language, abbreviated GLSL. It is a language similar to the C language specifically designed for GPUs, which can be run in parallel on the GPU.
OpenGL shaders have. FSH and. Vsh two files. These two files are compiled and linked to generate an executable program to interact with the GPU. Vsh is a vertex Shader (vertex shader) for vertex calculations that can understand the position of the control vertex, where we typically pass in the position of the current vertex, and the coordinates of the texture: FSH is fragment Shader (element shader), in which I can recalculate each pixel point.
The following diagram can better explain the role of vertex shader and fragment shader. This picture is the rendering pipeline for OpenGL. The information is too much to record first. As can be seen from the figure, Vertex shader in front, Fragment shader in the rear.

Here is the code for the FSH and vsh that post the sample program for this article.

Shader.vsh attribute vec4 vertexin;    attribute vec2 texturein;    varying vec2 textureout;   void main (void)    {        gl_Position = vertexIn;        textureOut =  texturein;  }  

Shader.fsh varying vec2 textureout;   uniform sampler2d tex_y;   uniform sampler2d tex_u;   uniform sampler2d tex_v;   Void main ( void)    {       vec3 yuv;       VEC3  rgb;           yuv.x = texture2d (tex_y,  Textureout) .r;       yuv.y = texture2d (tex_u, textureOut). r -  0.5;       yuv.z = texture2d (tex_v, textureout). r -  0.5;       RGB = MAT3 ( 1,        1,  &nbs