This article records the OpenGL Video Playback Technology. OpenGL is a technology at the same level as direct3d. Compared with direct3d, OpenGL has the cross-platform advantage. Although in the gaming field, DirectX's influence has gradually surpassed OpenGL and is adopted by most PC game developers, in the field of professional and high-end graphics, OpenGL is still the main character that cannot be replaced due to Color accuracy.
OpenGL Introduction
I collected some knowledge about OpenGL on the Internet and listed it here.
Open graphics library (OpenGL) is a specification that defines an application interface (API) across programming languages and platforms, it is used to generate two-dimensional and three-dimensional images.
OpenGL specifications were maintained by the OpenGL architecture Review Board (ARB) established in 1992. ARB is composed of companies that are particularly interested in creating a unified and universally available API. According to the OpenGL official website, the ARB voting in June 2002 includes 3 dlabs, Apple Computer, ATI Technologies, Dell Computer, Evans & Sutherland, Hewlett-Packard, IBM, Intel, matrox, NVIDIA, SGI, and Sun Microsystems, microsoft was one of the founding members, but withdrew in March 2003.
OpenGL is still the only API that can replace Microsoft's full control over 3D graphics technology. It still has some vitality, but Silicon Graphics no longer promotes OpenGL in any way that Microsoft is not happy with, so it has a high risk. OpenGL is dominant in high-end graphics devices and professional applications (not supported by direct3d currently ). The open source community (especially the mesa project) has been committed to providing OpenGL support.
OpenGL rendering pipeline
The following content is also viewed on the Internet. I understand some of it, but I have not fully understood some aspects of 3D because of its weak foundation.
OpenGL pipeline processes graphic information in a specific order. The graphic information can be divided into two parts: vertex information (coordinates, normal vectors, etc) and pixel information (images, textures, etc ). The graphic information is finally written into the frame cache, and the data (images) stored in the frame cache can be obtained by the application (used to save the results, or as the input of the application, etc, see the gray dotted line ).
Display list)
The display list is a set of OpenGL commands that are stored (Compiled) for subsequent execution. All data, geometric (vertex) data, and pixel data can be stored in the display list. Caching data and commands to the display list improves performance.
Vertex operation (vertex processing)
The vertex coordinates and the legal line coordinates pass through the pattern view matrix from the object coordinates to the eye coordinates ). If illumination is enabled, perform illumination calculation on the converted fixed point and normal line coordinates. Illumination calculation updates the color value of the vertex.
Primitive assembly)
After the vertex processing, the basic elements (points, lines, and polygon) are transformed by the projection matrix, and then cropped on the cropping plane of the visible body, which is converted from the observed coordinate system to the cropping coordinate system. Then, the perspective Division (divided by W) and viewport transform are performed to project the 3D scene to the window coordinate system.
Pixel transfer operation (pixel Operation)
After a pixel is unpacked from the customer's memory, it must be scaled, offset, mapped, and pulled (clamping ). These operations are pixel conversion operations. The converted data has a texture memory or is directly converted to fragment by rastering ).
Texture memory (texture memory)
The texture image is loaded into the texture memory and then applied to the geometric object.
Raterization (Rasterization)
Grating refers to the process of converting geometric (vertex coordinates, etc.) and pixel data into fragment. Each clip corresponds to a pixel in the frame buffer zone, this pixel corresponds to the color and opacity information at a point on the screen. A fragment is a rectangular array that contains color, depth, line width, point size, and other information (such as anti-sawtooth calculation ). If the rendering mode is set to gl_fill, the pixel information inside the polygon is filled in at this stage.
For example, in a triangle, the three vertex coordinates of the input triangle and their color, vertex operations will transform the vertex coordinates of the triangle and the normal vector, the color information does not need to be transformed, however, illumination calculation affects the color information of vertices. After rastering, triangles are discrete into points. They are not represented by three coordinates, but are composed of a series of points. Each point stores the corresponding color, depth, opacity, and other information.
Fragment operation)
This is the final processing to convert the fragment into pixels in the frame buffer. First, the texture unit (Texel) is generated. A Texture unit is generated by the data in the texture memory and then applied to each clip. Then perform fog calculation. After the fog computing is complete, several segments are tested in sequence, including scissor test, Alpha test, stencer test, and deep test. Finally, perform the hybrid, jitter, logical operations and masking operations, and finally store the pixels into framebuffer.
Comparison between OpenGL and direct3d
Video Display technology has been described in the direct3d article and will not be repeated here. I read some articles about their differences on the Internet, which are simple and clear. Here I will reference them:
Comparison between OpenGL and direct3d
OGL is better than d3d:
OGL is an industry standard and cannot be found in many non-Windows operating systems.
The color of OGL is better than that of d3d, and the surface is smoother.
OGL functions are quite regular. Unlike d3d functions, they all use pointer methods. The function name is too long !!
OGL is the right-hand coordinate system, which is used in mathematics. Although d3d can be changed to the right-hand coordinate system, it must be supported by d3dx9_36.dll.
Common matrices of OGL, such as worldmatrix, are encapsulated and must be written by d3d.
The drawing method of OGL is flexible, while that of d3d requires fvf to be defined in advance, and it will not be drawn until all information is written into stream. This generates vertexbuffer and indexbuffer. Does Microsoft suspect that there are not many d3d buffers? How hard is it to learn ?? Where does OGL need this?
There are many versions of d3d. If the video card does not support it, it will be useless. OGL has never changed since a few years ago, so most graphics cards support it.
Also, I found that the translucent function of d3d has a big problem !! It is the question of the order of two translucent objects-the front will be blocked by the back.
However, d3d is better than OGL:
D3d supports image files in many formats, and OGL has to write its own code to load JPG files.
Because d3d is a pointer call mode, it is difficult to create d3d hooks, which increases the difficulty of making plug-ins.
D3d is a member of DirectX. Programmers can use directmusic for sound playback, which is always good, while OGL can only draw pictures.
D3d is a connection library vigorously promoted by Microsoft. On the contrary, Microsoft has made great efforts to suppress OGL (because Microsoft is involved in the development of products, how is the treatment so big ?)
Because of this, d3d has become the mainstream in China's large-scale game industry (I think they follow suit blindly. In fact, many foreign games use OGL)
To use OpenGL to play a video, follow these steps:
1. Initialization
1) initialization
2) create a window
3) set the plotting function
4) set the timer
5) enter the message loop
2. loop display
1) Adjust the display position and image size
2) Drawing
3) display
There is something to note here. That is, OpenGL does not need to use the direct3d program initialization window that uses winmain () as the main function. You must do this in direct3d. That is, you can use the Win32 window program and call createwindow () to create a dialog box before drawing on the dialog box. OpenGL only needs to use a common console program (the entry function is main ()). Of course, OpenGL can also draw images in windows of Win32 programs like direct3d.
The following is a detailed analysis of the above process by combining the example code of OpenGL for YUV/RGB playback.
Before going into the playback process, let's talk about the obvious feeling of learning OpenGL: There are a lot of OpenGL functions. OpenGL functions feature a large number of functions, but each function has fewer parameters. Direct3d, in turn, has fewer functions, but each function has many parameters.
1. Initialization
1) initialization
Gluinit () is used to initialize the glut library. It is defined as follows:
void glutInit(int *argcp, char **argv);
It contains two parameters: argcp and argv. In general, pass the argc and argv in the main () function to it.
Here, we will briefly introduce three libraries in OpenGL: Glu, glut, and glew.
GLU is a practical library that contains 43 functions with the prefix of the function name: Glu. In order to reduce the heavy programming workload, Glu encapsulates OpenGL functions. by calling the functions of the core library, it provides relatively simple usage for developers and implements some complex operations.
Glut is a utility library used for Window Interfaces and cross-platform interfaces.
Glew is a cross-platform extension library. Not required. It automatically identifies all OpenGL advanced extensions supported by the current platform. No in-depth research has been conducted.
You can specify the initial display mode. It is defined as follows.
void glutInitDisplayMode(unsigned int mode)
The mode can be set to the following values or combinations:
Glu_rgb: Specifies the RGB color window.
Glu_rgba: Specifies the color window of rgba.
Glu_index: Specifies the color index mode window.
Glu_single: specifies a single cache window.
Glu_double: Specify the Double Cache window.
Glu_accum: Used to accumulate cache in the window
Glu_alpha: The color component of the window contains the Alpha value.
Glu_depth: The window uses the deep cache.
Keyword: The window uses the template cache.
Glu_multisample: Specifies a window that supports the multi-sample function.
Glu_stereo: Specifies the stereo window.
Glu_luminance: brightness color model for Windows
Note that if you use the double buffer (glu_double), you need to use the fig. If you use a single buffer (glu_single), you need to use glflush () for plotting.
When using OpenGL to play a video, we can use the following code:
glutInitDisplayMode(GLUT_DOUBLE | GLUT_RGB );
2) create a window
Gluinitwindowposition () is used to set the position of the window. The X and Y coordinates can be specified.
You can specify the window size. You can set the window width and height.
Create a window by using. You can specify the title of the window.
These functions are very basic and will not be described in detail. Paste a sample code directly:
glutInitWindowPosition(100, 100);glutInitWindowSize(500, 500);glutCreateWindow("Simplest Video Play OpenGL");
3) set the plotting function
Gludisplayfunc () is used to set the plotting function. The operating system will call this function to redraw the form whenever necessary. Similar to wm_paint messages in Windows programming. For example, when you move the window to the edge of the screen and then move it back, the function is called to re-paint the window. It is defined as follows.
void glutDisplayFunc(void (*func)(void));
(* Func) is used to specify the repainting function.
For example, when playing a video, specify the display () function for repainting:
glutDisplayFunc(&display);
4) set the timer
When you play a video, you need to play a certain number of images per second (usually 25 frames). Therefore, you can call the drawing function to draw the image at intervals. The definition of the timer function maid () is as follows.
void glutTimerFunc(unsigned int millis, void (*func)(int value), int value);
Its Parameter meanings are as follows:
Millis: the scheduled time in milliseconds. 1 second = 1000 milliseconds.
(* Func) (INT value): Specifies the function called by the timer.
Value: input parameters to the callback function. Relatively high-end, never touched.
If you write only one glutimerfunc () function in the main function, you will find that the function is called only once. Therefore, you need to write another glutimerfunc () function in the callback function and call the callback function itself. Only in this way can the callback function be called in a complex loop.
For example, during video playback, specify to call the timefunc () function every 40 milliseconds:
In the main function:
glutTimerFunc(40, timeFunc, 0);
In the timefunc () function, set as follows.
void timeFunc(int value){ display(); // Present frame every 40 ms glutTimerFunc(40, timeFunc, 0);}
In this way, display () is called every 40 ms ().
5) enter the message loop
The glumainloop () enters the glut event processing cycle. Once called, this program will never return. When you play a video, you can call this function to start playing the video.
2. loop display
1) Adjust the display position and image size
This step mainly adjusts the image size and position. If you do not use gldrawpixels () for plotting without this step, you will find that the image is located in the lower left corner of the window and is upside down (of course, if the window is as big as the image, there is no problem with the image in the corner ). Shows the effect.
To solve the above problem, you need to call the relevant functions to transform the image. The transformation uses two functions: glrasterpos3f () and glpixelzoom ().
Glrasterpos3f () can translate images. It is defined as follows.
void glRasterPos3f (GLfloat x, GLfloat y, GLfloat z);
X is used to specify the X coordinate, and Y is used to specify the Y coordinate. Z is not used yet.
Here we will introduce the coordinates of OpenGL. The origin is located in the center of the screen. The value on the edge of the screen is 1.0. It is basically the same as the coordinate system in mathematics. The lower-left corner of the screen is (-1,-1), and the upper-left corner is ).
For example, if we use glrasterpos3f (-1.0f, 0.0f, 0), the image will be moved to (-), as shown in.
Glpixelzoom () allows you to zoom in, zoom out, and flip an image. It is defined as follows.
void glPixelZoom (GLfloat xfactor, GLfloat yfactor);
Xfactor and yfactor are used to specify the magnification on the X axis and Y axis (if the value is smaller than 1, it is reduced ). If a negative value is specified, you can flip it. As mentioned above, if OpenGL is used to directly display pixel data, the image will be displayed upside down. Therefore, you need to flip the image in the Y axis.
For example, if the pixel data width and height are pixel_w and pixel_h, the window size is screen_w, and screen_h, you can use the following code to stretch the image to the window size and flip it over:
glPixelZoom((float)screen_w/(float)pixel_w, -(float)screen_h/pixel_h);
Combining the two functions above, that is, "translation + flip + stretch", you can get a full screen image, as shown in.
PS: This method is a stupid one. There should be better methods. However, no further research has been conducted.
2) Drawing
You can use gldrawpixels () to plot pixel data in the specified memory. The function is defined as follows.
void glDrawPixels (GLsizei width, GLsizei height,GLenum format,GLenum type,const GLvoid *pixels);
The parameter meanings of this function are as follows:
Width: the width of the pixel data.
Height: the height of the pixel data.
Format: the format of the pixel data, such as gl_rgb, gl_bgr, and gl_bgra.
Type: the format of pixel data in memory.
Pixels: pointer to the memory used to store pixel data.
For example, to plot data in rgb24 format, the width is pixel_w, the height is pixel_h, And the pixel data is stored in the buffer. You can use the following code.
glDrawPixels(pixel_w, pixel_h,GL_RGB, GL_UNSIGNED_BYTE, buffer);
3) display
When using double buffering, call the function gluswapbuffers () for display.
When using a single buffer, call the glflush () function for display.
Video Display Process summary
The function call structure displayed in the video can be summarized
Code
Paste the source code.
/*** Simplest example of OpenGL video playback (OpenGL RGB/YUV) * simplest video play OpenGL (OpenGL play RGB/YUV) ** leixiao Li Lei Xiaohua * [email protected] * China Media University/Digital TV technology * Communication University of China/Digital TV technology * http://blog.csdn.net/leixiaohua1020 ** this program uses OpenGL to play RGB/ YUV video pixel data. In fact, this program can only play RGB (rgb24, bgr24, bgra) data. If the input data is yuv420p *, convert the data to RGB before playing the video. * This program is the simplest example of using OpenGL to play pixel data. It is suitable for beginners of OpenGL. ** Function call procedure: ** [initialization] * gluinit (): Initialize the glut library. * Gluinitdisplaymode (): sets the display mode. * Ngcreatewindow (): Creates a window. * Gludisplayfunc (): sets the plotting function (called during repainting ). * Glutimerfunc (): sets the timer. * Glumainloop (): enters the message loop. ** [Cyclically rendered data] * glrasterpos3f (), glpixelzoom (): Adjust the display position and image size. * Gldrawpixels (): Draw. * Gluswapbuffers (): display. ** This software plays RGB/YUV raw video data using OpenGL. this * software support show RGB (rgb24, bgr24, bgra) data on the screen. * If the input data is yuv420p, it need to be convert to RGB first. * This program is the simplest example about play raw video data * Using OpenGL, suitable for the beginner of OpenGL. ** the process is shown as follows: ** [init] * gluinit (): init glut Library. * Gluinitdisplaymode (): Set display mode. * fig (): create a window. * gludisplayfunc (): Set the display callback. * glutimerfunc (): set timer. * glumainloop (): Start message loop. ** [loop to render data] * glrasterpos3f (), glpixelzoom (): Change picture's size and position. * gldrawpixels (): Draw. * gluswapbuffers (): Show. */# include <stdio. h> # include "glew. H "# include" glut. H "# Include <stdlib. h> # include <malloc. h> # include <string. h> // set '1' to choose a type of file to play # define load_rgb24 1 # define load_bgr24 0 # define load_bgra 0 # define limit 0int screen_w = 500, screen_h = 500; const int pixel_w = 320, pixel_h = 180; // bit per pixel # If load_bgraconst int bpp = 32; # Elif load_rgb24 | load_bgr24const int bpp = 24; # Elif load_yuv420pconst int bpp = 12; # endif // YUV filefile * fp = N Ull; unsigned char buffer [pixel_w * pixel_h * BPP/8]; unsigned char buffer_convert [pixel_w * pixel_h * 3]; inline unsigned char convert_adjust (double TMP) {return (unsigned char) (TMP> = 0 & tmp< = 255 )? TMP :( TMP <0? 0: 255);} // yuv420p to rgb24void convert_yuv420ptorgb24 (unsigned char * yuv_src, unsigned char * rgb_dst, int nwidth, int nheight) {unsigned char * tmpbuf = (unsigned char *) malloc (nwidth * nheight * 3); unsigned char y, U, V, R, G, B; unsigned char * y_planar, * u_planar, * v_planar; int rgb_width, u_width; rgb_width = nwidth * 3; u_width = (nwidth> 1); int ypsize = nwidth * nheight; int upsize = (ypsize> 2); int offset = 0; y_planar = yuv_src; u_planar = yuv_src + ypsize; v_planar = u_planar + upsize; For (INT I = 0; I <nheight; I ++) {for (Int J = 0; j <nwidth; j ++) {// get the Y value from the y planary = * (y_planar + nwidth * I + J ); // get the V value from the u planaroffset = (I> 1) * (u_width) + (j> 1); V = * (u_planar + offset ); // get the U value from the V planaru = * (v_planar + offset); // cacular the R, G, B values/ /Method 1R = convert_adjust (Y + (1.4075 * (V-128); G = convert_adjust (Y-(0.3455 * (u-128) -0.7169 * (V-128); B = convert_adjust (Y + (1.7790*(u-128 )))); /* // the following formulas are from Microsoft 'msdnint c, d, e; // method 2C = Y-16; D = u-128; E = V-128; R = convert_adjust (298 * C + 409 * E + 128)> 8 ); G = convert_adjust (298 * C-100 * D-208 * E + 128)> 8); B = Co Nvert_adjust (298 * C + 516 * D + 128)> 8); r = (R-128) *. 6 + 128)> 255? 255 :( R-128) *. 6 + 128; G = (G-128) *. 6 + 128)> 255? 255 :( G-128) *. 6 + 128; B = (B-128) *. 6 + 128)> 255? 255: (B-128 )*. 6 + 128; */offset = rgb_width * I + J * 3; rgb_dst [offset] = B; rgb_dst [Offset + 1] = g; rgb_dst [Offset + 2] = r ;}} free (tmpbuf);} void display (void) {If (fread (buffer, 1, pixel_w * pixel_h * BPP/8, FP )! = Pixel_w * pixel_h * BPP/8) {// loop fseek (FP, 0, seek_set); fread (buffer, 1, pixel_w * pixel_h * BPP/8, FP );} // make picture full of window // move to (-1.0, 1.0) glrasterpos3f (-1.0f, 1.0f, 0); // zoom, flipglpixelzoom (float) screen_w/(float) pixel_w,-(float) screen_h/(float) pixel_h); # If substring (pixel_w, pixel_h, gl_bgra, substring, buffer); # Elif substring (pixel_w, pixel_h, gl_rgb, Vertex, buffer); # Elif transform (pixel_w, pixel_h, kernel, kernel, buffer); # Elif transform (buffer, buffer_convert, pixel_w, pixel_h); gldrawpixels (pixel_w, pixel_h, gl_rgb, gl_unsigned_byte, buffer_convert); # endif // maid ();} void timefunc (INT value) {display (); // present frame every 40 MS glutimer Func (40, timefunc, 0);} int main (INT argc, char * argv []) {# If load_bgrafp = fopen (".. /test_bgra_320x180.rgb "," RB + "); # Elif load_rgb24fp = fopen (".. /test_rgb24_320x180.rgb "," RB + "); # Elif load_bgr24fp = fopen (".. /test_bgr24_320x180.rgb "," RB + "); # Elif load_yuv420pfp = fopen (".. /test_yuv420p_320x180.yuv "," RB + "); # endifif (FP = NULL) {printf (" cannot open this file. \ n "); Return-1;} // glut init gluinit (& argc, argv );// Double, use gluswapbuffers () to show gluinitdisplaymode (glu_double | glu_rgb); // single, use glflush () to show // gluinitdisplaymode (glu_single | glu_rgb); gluinitwindowposition (100,100 ); gluinitwindowsize (screen_w, screen_h); glucreatewindow ("simplest video play OpenGL"); printf ("simplest video play OpenGL \ n"); printf ("Lei Xiaohua \ n "); printf ("http://blog.csdn.net/leixiao112102020.n"); print F ("OpenGL version: % s \ n", glgetstring (gl_version); fig (& Display); fig (40, timefunc, 0); // start! Glumainloop (); Return 0 ;}
Code Note 1. You can set the macro defined at the beginning of the file to determine which format of pixel data to read (bgra, rgb24, bgr24, yuv420p ).
//set ‘1‘ to choose a type of file to play#define LOAD_RGB24 1#define LOAD_BGR24 0#define LOAD_BGRA 0#define LOAD_YUV420P 0
2. The window width and height are screen_w and screen_h. The width and height of pixel data are pixel_w and pixel_h. They are defined as follows.
//Width, Heightconst int screen_w=500,screen_h=500;const int pixel_w=320,pixel_h=180;
3. Note the different display modes.
Bgra, bgr24, and rgb24 formats can be directly displayed in gldrawpixels. Yuv420p cannot be directly displayed. In this example, yuv420p is converted to rgb24 and then displayed.
Running result
Regardless of the file to be loaded, the running results are the same, as shown in.
The download code is in "simplest media play ".
SourceForge project: https://sourceforge.net/projects/simplestmediaplay/
Http://download.csdn.net/detail/leixiaohua1020/8054395 (csdn)
The above project contains the use of various APIs (direct3d, OpenGL, GDI, directsound, sdl2) to play multimedia examples. The audio input is PCM sampling data. Output to the sound card of the system. The video input is YUV/RGB pixel data. Output to a display window for playing.
The code of this project allows beginners to quickly learn how to use these APIs to play videos and audios.
The following sub-projects are included:
Simplest_audio_play_directsound: Use directsound to play PCM audio sample data.
Simplest_audio_play_sdl2: Use sdl2 to play PCM audio sample data.
Simplest_video_play_direct3d: Use the direct3d surface to play RGB/YUV video pixel data.
Simplest_video_play_direct3d_texture: Use direct3d texture to play RGB video pixel data.
Simplest_video_play_gdi: Use GDI to play RGB/YUV video pixel data.
Simplest_video_play_opengl: uses OpenGL to play pixel data of RGB/YUV videos.
Simplest_video_play_opengl_texture: use OpenGL texture to play the YUV video pixel data.
Simplest_video_play_sdl2: Use sdl2 to play RGB/YUV video pixel data.
Example 4: OpenGL RGB/YUV
