Use d3d to render YUV video data

Source code download

 

 

On a PC, DirectDraw is generally used to display videos in YUV format, such as yv12 and yuy2, and overlay is used to display videos on the graphics card. Overlay is a hardware-based technology implemented on graphics cards to play VCD on PCs. The emergence of overlay solves the difficulties encountered in playing VCD on PC. Early PC processing capabilities were limited. During VCD playback, not only video decoding was required, but also color space conversion from YUV to RGB. The software implementation was very resource-consuming, the YUV overlay surface appears, and the color space conversion is transferred to the video card for implementation. It is a natural advantage for the video card to do this work.

 

With the development of graphics technology, the limitations of overlay are increasingly exposed. Generally, a video card supports only one overlay surface. It is difficult to use overlay to implement multiple images, and it is also difficult to overlay videos and texts. Of course, it is more difficult to implement some special effects. More importantly, the overlay technology belongs to the 2D module on the video card. Driven by high-quality 3D games, the current function and performance of the video card are mainly reflected in the 3D module, the most invested by the manufacturer is on the GPU 3D module. Overlay Technology cannot utilize and utilize the 3D performance of graphics GPU. Microsoft has long stopped its support for DirectDraw and encouraged developers to switch to direct3d. Therefore, overlay cannot use new APIs.

 

Early 3D rendering was mainly made using the CPU, with fewer video cards. Later, the processing capability of graphics card GPUs became stronger and stronger, and more 3D rendering functions were undertaken. Fixed rendering pipelines were used, that is, all rendering functions.AlgorithmThey are all built-in graphics cards. We can only use these algorithms in combination. Currently, all graphics cards use programmable rendering pipelines. That is to say, we can write our own rendering algorithms, download them to the graphics card for execution, and replace the Fixed rendering pipeline algorithm. This greatly improves the flexibility. With the improvement of GPU performance, graphics cards have been widely used in image processing, video processing, scientific computing, and other fields.

 

If you use d3d to render YUV videos, you can use d3d surface rendering or d3d texture rendering. Surface rendering is relatively simple, but there are many functional limitations. Next we will only discuss d3d texture video rendering. Texture video rendering is to fill the video data into a two-dimensional texture, combined with a section of our own pixel shaderCodeTo the GPU rendering pipeline of the video card for rendering.

 

The following assumes that the reader understands the details of the video format discussed.

 

For yv12 video data, you can create three textures to fill in the Y, U, and V video data respectively. For missing u and v sampling points, we can use the bilinear filter algorithm built in the graphics card to interpolation. This is a simple method. Of course, we can also adopt better image interpolation algorithms to achieve higher image quality. This requires us to use shader to implement our interpolation algorithm and then render U and V to the texture respectively, implement better interpolation. Compared with bilinear filtering, the disadvantage is high complexity and low efficiency. After u and v interpolation, the video data is converted from yuv420 to yuv444, and then the color space is converted from yuv444 to rgb32. The shader code for conversion is very simple, which is a matrix multiplication operation. For i420 data, the u and v Data of yv12 are reversed in the memory. Other processing is the same as that of yv12.

 

Nv12 format. The Y plane is the same as yv12, and the UV part is the packaging format. Therefore, the UV part needs to be processed independently. First, the UV part needs to be filled into a texture, and then the UV texture is simply rendered to the texture twice, then, the u and v Data are rendered to two textures respectively. After such processing, the data we get is the same as that of yv12. For the next processing, we can refer to the yv12 processing process and method.

 

For the data in the packaging format yuy2, the missing u and v sampling needs to be interpolated during rendering. Yuy2 only lacks u and v sampling on an odd number of pixels. In this way, the parity pixels need to be differentiated. For an even number of pixels, U and V interpolation is not required to directly convert the color space. For odd pixels, adjacent u and v samples must be used for interpolation. Catmull-Rom interpolation or linear interpolation can be used. After interpolation, you can convert the color space. Uyvy is similar to yuy2.

 

For the RGB format, the NVIDIA geforce 9800 GT video card we use supports rgb32, rgb555, rgb565, and rgb24. Therefore, for rgb24, You need to extract R, G, and B in the shader code, and then form the rgb32 output. Rgb32, rgb555, and rgb565 can be directly rendered without additional processing.

 

The cd3dfont class provided by Microsoft is used for the superposition of flat texts. However, this class does not support Chinese characters. I have made some modifications based on this class, implements a text overlay class that supports Chinese characters. These two classes work together to efficiently overlay the plain text.

 

The content discussed above is verified and tested using direct3d9.0c. I don't know much about OpenGL, but I think these methods should also apply to OpenGL. After all, these two 3D architectures implement similar functions on PCs.

 

You can use 3D to render a plane video to achieve a variety of special effects, such as multi-screen, in-picture, text superposition, and scaling, which are easy to implement and provide good performance. If we combine other 3D technologies, we can also make amazing and cool special effects, just look at our imagination.