DirectShow-based streaming media decoding and Playback

I. Preface

Streaming media is widely defined. Most of the time it refers to compressing continuous images and sound information and putting it on the website server so that users can download and watch and listen to it, video/audio transmission and compression technologies that can be viewed without waiting for the entire compressed file to be downloaded to your own machine. Streaming media also refers to a specific file format supported by this technology: Compressed streaming files, which are transmitted over the network and decoded through personal computer software.

MCI is a multimedia programming interface that Microsoft initially proposed for Windows. With the rapid development of multimedia technology and the application of various compression algorithms in this field, the MCI technology is becoming increasingly inadequate, the most obvious difference is that it does not support Variable Bit Rate compression algorithms. It is powerless to process various new media formats such as DVDs in recent years, using Multi-media libraries such as VFW provided by Microsoft is too troublesome. What should we do?
As the successor of MCI, Microsoft also launched the DirectShow technology based on DirectX (including DirectDraw, directsound, and direct3d), which is the media layer on DirectX, supports decoding and playback of media files of various video and audio compression formats from the local or network, capturing Multimedia Streams from devices, or processing streaming media processed by various compression algorithms. These formats include: MPEG audio and video standards, audio and video interaction standards (AVI), wave, Midi and advanced stream formats ASF. DirectShow uses stream media (multimedia stream) for media data processing. This method can be used in applications to greatly reduce programming complexity and automatically negotiate the conversion from data sources to applications, the stream interface provides a unified and predictable data access control method, so that the application does not need to consider its original source and format when playing media data.
　
　　2. Understand DirectX

DirectX is a programming environment for multimedia applications and hardware enhancement. It is developed and designed by Microsoft to build its Windows system into the best platform for various multimedia applications. DirectX has become part of Microsoft's own SDK, while DirectX is integrated in Windows 98/Windows 2000, indicating that it has become part of the operating system.

DirectX technology is an API (application interface). Each DirectX component is the sum of user-callable APIs. Its Applications can directly access computer hardware. In this way, applications can use the hardware accelerator (hardware accelerator ). If the hardware accelerator is unavailable, DirectX can also simulate the accelerator to provide a powerful multimedia environment.

To understand DirectX, we can divide the system into four layers:

● Hardware/network layer: multimedia devices are deployed, including Graphics accelerators, sound cards, input devices, and network communication devices;

● DirectX base layer: provides basic multimedia services for images, sounds, and devices;

● DirectX Media layer: provides API functions for animation production, audio and video;

● Component layer: Includes ActiveX Control and applications. It uses the advantages of DirectX APIs to provide multimedia services for users.

DirectShow is a technology built on the DirectX Media layer. Its predecessor is activemovie2.0. It is used as a set of API functions or ActiveX controls to enable developers to transmit high-quality audio and video signals over the network. It is worth mentioning that DirectShow provides an open development environment for us to customize components based on our own needs.

　　Iii. DirectShow technical structure

DirectShow defines how to use standard components to process streaming media data. These components are called filters. The filter has an input or output pin, or both. In DirectShow technology, the core component is the pluggable standard component used as a "filter". It is a COM object that executes a specific task. Filters can be subdivided into source filter, transform filter, and Renderer Filter. The filter operates streaming media by reading, writing, modifying, and displaying data to the output device. To complete the entire task, you must connect all the filters. These three filters form the filter chart structure, as shown in Figure 3.1:

Figure 3.1 filter graph)

As you can see from Figure 3.1, the filter chart is a collection of various filters, which are connected by the input and output pins of the filter "pin" in sequence, the pins of these filters are negotiated to determine the form of multimedia they will support. DirectShow supports reconfigurable filter chart structures, so multiple types of media can be played using the same software component. Developers can define their own filters to extend DirectShow's support for media.

In the filter chart structure, the source filter is used to obtain data from the data source and transmit the data to the filter chart. The data source here can be a camera, Internet, or disk file; the conversion filter is used to obtain, process, and transmit media data. It includes splitter transform filter, and video transform filter) decompress the audio data conversion filter (audio transform filter). The performance filter is used to display media data on hardware, such as the video card and sound card, or wherever media data can be accepted, such as disk files. It includes the video Renderer Filter Used to display images, and the audio Renderer Filter that sends audio data to the sound card ).

In the filter chart, all the necessary filters must be connected to complete specific tasks. Therefore, the output of the filter must be the input of the lower-level filter. A filter has at least one input pin and sends specific output to the output pin. Figure 3.2 shows a filter connection diagram:

3.2 filter connection Diagram

Your application does not need to process the filters in the filter chart separately, because at a higher level, DirectShow provides a component called the filter chart Manager (FGM) managing the connection between these filters and the flow of streaming media data between filters, FGM provides a set of COM interfaces through which applications can access filter charts, control streaming media, or receive filter events. If needed, It can automatically insert a suitable decoder and connect the output pins of the conversion filter to the performance filter. The application can control the activity of the filter chart by communicating with the filter chart manager. Developers only need to call API functions to control streaming media. For example, the run method starts streaming media flow in the filter graph. The pause method pauses streaming media playback; stop method to stop streaming media.

In addition, the filter graph manager can transmit event information to the application layer, enabling applications to respond to event processing, for example, you can play or search for streaming media data and stream end information for a specific period of time.

Figure 3.3 shows an example of MPEG decoding and playback. The Source Filter sends the obtained multimedia data to the mpeg decomposition and conversion filter through outpin. the mpeg decomposition and conversion filter has an input pin, the two output pins are decoded respectively by the video and audio interpreter. The last two channels are sent to the video card and sound card for playback respectively through the video representation filter and audio representation filter.

Figure 3.3 MPEG decoding instance Iv. DirectShow Program Development DirectShow is based on the COM component technology. Therefore, you must master the COM component technology to develop the DirectShow program. DirectShow is closely connected with COM. all its components and functions are constructed and implemented by the COM interface. Its development method is quite flexible and there is no fixed mode, generally, different com interfaces are used as needed. However, several important interfaces are frequently used: igraphbuilder interface, which is the most reusable COM interface used to create filter graph manager and imediacontrol interfaces, used to control the flow of streaming media in the filter graph, such as the start and stop of streaming media; imediaevent interface, this interface is used to create event flag information when filter graph has some events and send them to the application. The development of a typical DirectShow application follows the following steps: 1) create a filter graph manager instance using the cocreateinstance () function; 2) Call the QueryInterface () function to obtain the pointer of the filter graph and imediaevent components; 3) control filter graph and respond to events. The following is a simple example to illustrate how to use DirectShow technology to decode and play back Multimedia Streams. First, generate a single-document application named mediaplay and define a function named mediaplay. The specific implementation code of this function is: Void playmovie (lptstr lpszmovie) { Imediacontrol * PMC = NULL; Igraphbuilder * pgb = NULL; Imediaeventex * PME = NULL; Long evcode; // something to hold a returned Event code HR = cocreateinstance (clsid_filtergraph, null, clsctx_inproc, Iid_imediacontrol, (void **) & PMC ); HR = PMC-> QueryInterface (iid_igraphbuilder, (void **) & pgb ); HR = PMC-> QueryInterface (iid_imediaeventex, (void **) & PME ); HR = pgb-> renderfile (lpszmovie, null ); HR = PMC-> Run (); HR = PME-> waitforcompletion (infinite, & evcode ); If (PMC) PMC-> release (); If (pgb) pgb-> release (); If (PME) PME-> release (); } In the code above, the cocreateinstance () function creates a filter graph object and returns an imediacontrol interface, this interface uses a filter to display, pause, stop, and other media. However, at this time, the chart object does not contain a specific filter, because DirectX does not know what type of media needs to be played; next, create an interface for creating charts, this interface can be used to create filter charts, add and delete various filters to chart objects, list all filters in the Current Filter charts, and connect various filters in chart objects; in this example, the renderfile () function of the igraphbuilder interface is used to tell DirectX the media file name to be played. In this case, the igraphbuilder object interface depends on the multimedia file type, automatically adds various filters required for playing this type of media to the filter chart and connects them. Finally, the function calls the run () function of the imediacontrol interface object to start playing media files. To enable the playback of multimedia files from the beginning to the end, you must call the waitforcompletion () blocking function of the imediaeventex object interface to run the function until the media file ends.