Introduction to DirectShow development quick start)

Summary
: This document provides a general introduction to DirectShow.
As well as some basic concepts of DirectShow. Familiar with these basic knowledge
DirectShow Application Development
Or filter developers will be helpful.

DirectShow is Microsoft
The company provides a Streaming Media Processing Development Kit on the Windows platform, which is released together with the DirectX development kit. That
What can DirectShow do? Also, DirectShow is multimedia.
Stream capturing and playback provide powerful support. Using DirectShow, we can easily
Capture data on the acquisition card and perform corresponding post-processing and even store the data in the file. It supports a wide range of media formats, including ASF, MPEG, Avi, DV, MP3, and wave.
It is easy to play back multimedia data. In addition, DirectShow integrates other DirectX technologies (such as DirectDraw and directsound ).
Directly supports dv
D playback, non-linear Video
Sex
Edit and exchange data with the digital camera. More importantly, DirectShow provides an open development environment. You can customize your own components as needed.

　
The relationship between applications and the DirectShow component and the software and hardware supported by DirectShow is shown in 1.

Figure 1 DirectShow system diagram

1,
DirectShow Filter

DirectShow is based on modularization. Each function module adopts the COM component method, which is called filter.
DirectShow provides a series of standard modules for application development. developers can also develop their own function filters to expand DirectShow applications. Next we use
Example to illustrate how to use a filter to play an AVI video file.

1)
First, read Avi Data from a file to form a byte stream. (This task is done by the source filter)

2)
Check the Header Format of the Avi Data Stream and separate the video stream from the audio stream using the AVI split filter.

3)
Decode the video stream and select different decoder Filters Based on different compression formats.

4) redraw Video Images Using Renderer Filter
.

5) audio streams are sent to the sound card for playback. The default directsound device is generally used.
Filter.
For the process, see.

Figure 2 audio stream playback Graph

From the chart above
Filters are connected by one or two other filters. The connection points connected by two filters are also COM objects, which are called pin. Filter
Data is transmitted from one filter to another, which allows data to flow in a linked list composed of filters. The arrow in the figure indicates the direction of the data stream in the filter linked list.
Direction. In DirectShow, a filter linked list like above is called filter graph.

The filter has three statuses,
Run, stop, and pause. When a filter is running, it processes the media data stream. When it is stopped, the filter does not process the data.
Data. Data Flow in the filter graph chapter describes these concepts in detail.

Except for some special exceptions,
The status changes of all filters in the filter graph are unified. That is to say, all filters in the filte Graph
The status changes are consistent and coordinated. That is to say, we can also use filter graph to run, stop, and pause three states.

Filter
It is generally divided into the following types.

(1) Source Filter (Source
Filter ):
The Source Filter imports data into the filter chart. data sources can be files, networks, and photos.
Machines. Different Source filters process different types of data sources.

(2) transform filter (Transform)
Filter): The function of changing a filter is to obtain an input stream, process data, and generate an output stream. Data processing by transform filters includes encoding/decoding.
,
Format conversion, compression, and decompression.

(3) Renderer Filter: The submit filter is at the last level in the filter chart.
They receive data and submit the data to the peripherals.

(4) Splitter
Filter): Split the input stream into multiple outputs. For example, the AVI split filter splits an AVI byte stream into a video stream and an audio stream.

(5)
Hybrid filter (MUX
Filter): A hybrid filter combines multiple inputs into a separate data stream. For example, the AVI hybrid filter combines video streams and audio streams into an AVI byte stream.

　
These types of filters are not absolute. For example, an ASF reader filter is both a source filter and a segmentation filter.

　
2. filter graph Manager

Filter graph
Manager is also a COM object used to control all filters in the filter graph. It has the following functions:

1)
It is used to coordinate the state changes between filters, so that the State Changes of all filters in graph should be consistent.

2) create a reference clock.

　
3) return the filter Message notification to the application.

4) provides a method to create a filter graph.

Here
It is just a brief description. For details, refer to the documentation.

The state of the filter in graph should be the same. Therefore, the application changes the state
The command is directly sent to the filter, but the command for changing the status is sent to the filter graph
Manager, which distributes commands to each filter in graph. Seeking also works in the same way. First, the application sends the seek command
Filter graph manager, which is then distributed to each filter.

Refer to the clock. Filters in graph all use the same clock, which is called
For reference clock, the reference clock ensures that all data streams are synchronized. The time when a video or audio clip is submitted is called presentation.
Time. Presentation Time is determined relative to the reference clock. Filter graph
The manager should select a reference clock. You can select either the clock on the sound card or the system clock.

Graph event, graph
The manager uses the event mechanism to notify the application of events in graph. This mechanism is similar
Message loop machine
.

Graph construction method. The graph manager provides the application with the method to add the filter to the graph, connect to the filter method, and disconnect
Filter connection method.

However, graph
The manager does not provide a way to send data from one filter to another. This is done independently by the filter via pin internally.

3. Media type

Because DirectShow is based on the COM component, You need to describe the filter
The data format of each vertex in graph. For example, we also take the playback of an AVI file as an example. The data enters the graph in the form of a riff block and is then divided into video and audio streams. The video stream has a series
After decompression, the video stream is composed of a series of uncompressed bitmaps, and the audio stream follows the same steps.

Media Types:
How DirectShow Represents Formats

Media types are common and can be extended to describe digital media formats.
When two filters are connected, they use a media type to reach an agreement. The media type defines that the filter at the source will be sent to the downstream filter.
The data to be sent and the physical layout of the data. If two filters cannot support the same media type, they cannot be connected.

　
For most applications, you may not need to consider the media type. However, in some applications, you will apply the media type directly.

The media type is
The am_media_type structure is defined. Check the original definition.

typedef struct _MediaType {
　
GUID majortype;
　 GUID subtype;
　 BOOL bFixedSizeSamples;
　
BOOL bTemporalCompression;
　 ULONG lSampleSize;
　 GUID formattype;
　
IUnknown *pUnk;
　 ULONG cbFormat;
　 [size_is(cbFormat)] BYTE
*pbFormat;
} AM_MEDIA_TYPE;

Major
Type: it is a guid used to define the main type of data, including audio, video, unparsed byte stream, and Midi data. For details, refer to msdn.

　
Subtype: subtype, also a guid, used to further refine the data format, for example, in the video main type, also includes RGB-24, RGB-32,
Uyvy and so on, in the audio main type also includes PCM audio, MPEG-1
Payload and Other types. The subtype provides more detailed information than the primary type, but does not define all formats. For example, the subtype of a video does not define an image.
Size and renewal rate. These are defined by the following fields.

Bfixedsizesamples when this value is
True indicates that the sample size is fixed.

Btemporalcompression when this value is true, it indicates that the sample uses
The compression format indicates that not all partitions are key partitions. If the value is false, all partitions are key partitions.

Lsamplesize
Indicates the sample size. For compressed data, this value may be zero.
　　
Formattype is a guid value used to indicate the format of the memory block. Including
For example
Under: format_none, format_dvinfo, format_mpegvideo, format_mpeg2video, format_videoinfo, format_videoinfo2, format_waveformatex, guid_null.

　
Punk.

The size of the cbformat memory block.

Pbformat pointer to the memory block.

　
Next we will look at a piece of code to see how the filter detects the media type.

HRESULT
CheckMediaType(AM_MEDIA_TYPE *pmt)
{
　 if (pmt == NULL) return
E_POINTER;
　 // Check the major type. We’re looking for video.
　
if (pmt->majortype != MEDIATYPE_Video)
　 {
　　 return
VFW_E_INVALIDMEDIATYPE;
　 }
　 // Check the subtype. We’re looking
for 24-bit RGB.
　 if (pmt->subtype != MEDIASUBTYPE_RGB24)
　 {
　
　 return VFW_E_INVALIDMEDIATYPE;
　 }
　 // Check the format type
and the size of the format block.
　 if ((pmt->formattype ==
FORMAT_VideoInfo) && (pmt->cbFormat >=
sizeof(VIDEOINFOHEADER) &&
(pmt->pbFormat != NULL))
　 {
　
　 // Now it’s safe to coerce the format block pointer to the
　　 //
correct structure, as defined by the formattype GUID.
　　
VIDEOINFOHEADER *pVIH = (VIDEOINFOHEADER*)pmt->pbFormat;
　　 //
Examine pVIH (not shown). If it looks OK, return S_OK.
　　 return
S_OK;
　 }
　 return VFW_E_INVALIDMEDIATYPE;
}

The following is a brief introduction.
Several media type-related functions:

The am_media_type structure contains a pointer to a data block. Therefore, when you use this structure
Wait, be careful with memory allocation to prevent memory leakage.

Allocation Function

1) am_media_type * winapi
Createmediatype (am_media_type const * psrc );

This function assigns a new
Am_media_type structure, which contains data blocks of specific formats. Release the memory allocated by this function. You can call the deletemediatype function.

　　
2) stdapi createaudiomediatype (const waveformatex * pwfx, am_media_type
* PMT, bool bsetformat );

This function uses a given waveformatiex structure to initialize the media type. If
If the bsetformat parameter is true, the function allocates a new memory. If the original PMT already contains the memory, memory leakage may occur. To avoid Memory leakage, call this function
Call freemediatype () before the number. After this function returns, call freemediatype () again to release the format block.

　
3) hresult winapi copymediatype (am_media_type * pmttarget, const
Am_media_type * pmtsource );

This function copies a structure to another structure. This function also needs to re-allocate the memory to the destination node.
Structure. If pmttarget already contains a memory block, the memory will leak. Therefore, the freemediatype function must be called before and after the function is called.

　　
Release Function

4) void winapi deletemediatype (am_media_type * PMT );

　
Whether you use the cotaskmemalloc function or the memory allocated by the createmediatype function, you can use this function to release the memory.
Library, you can use the following code

Void mydeletemediatype (am_media_type * PMT)
{
　
If (PMT! = NULL)
{
Myfreemediatype (* PMT); // see the following
Freemediatype Function
Cotaskmemfree (PMT );
}
}

5)
Void winapi freemediatype (am_media_type & mt );

This function is used to release data blocks.
Memory. to delete the am_media_type structure, you can use the deletemediatype function.

void
MyFreeMediaType(AM_MEDIA_TYPE& mt)
{
　 if (mt.cbFormat != 0)
　
{
　　 CoTaskMemFree((PVOID)mt.pbFormat);
　　 mt.cbFormat = 0;
　　
mt.pbFormat = NULL;
　 }
　 if (mt.pUnk != NULL)
　 {
　　 //
Unecessary because pUnk should not be used, but safest.
　　
mt.pUnk->Release();
　　 mt.pUnk = NULL;
　 }
}

4. Media samples and allocators

Filters transmits data through the pin connection, and the data stream is output from a filter pin
Input Pin of the filter connected to the flow direction. The commonly used method for outputting pin data is to call the imeminputpin: receive method on the input pin.

　
For the filter, there are several ways to allocate memory blocks used by media data, which can be allocated on the stack, on the DirectDraw surface, or using the shared memory of GDI,
There are other methods. In DirectShow, the memory distributor (Allocator) is used for memory allocation tasks and is also a COM Object, exposing
Imemallocator interface.

When two pins are connected, one pin must provide an allocator, DirectShow definition.
A series of function calls are used to determine which pin provides Allocator and the number and size of buffer.

Starting from Data Flow
Allocator creates a memory pool (pool
Buffer). After the data stream is sent, the source filter will fill the data in an idle buffer in the memory pool and then pass it to the following filter. However, the source
The filter does not directly pass the memory buffer pointer to the downstream filter, but uses a media
Samples COM Object. This sample is created by allocator to manage the memory buffer. Media
Sample exposes the imediasample interface. A sample contains the following content:

A pointer to the memory that is not sent.

　
A timestamp

Some logos

Media type.

The timestamp indicates presentation.
Time, Renderer
Filters arrange the order of render according to the time. A flag is used to indicate whether data is interrupted. A media type provides a way to change the data format midway through.
Sample has no media type, indicating that their media type has not changed.

When a filter is using buffer, it will keep a sample
Allocator determines whether a buffer can be used again through the reference count of sample. In this way, the buffer usage conflict is prevented.
If the filter releases a reference to the sample, the sample is returned to the memory pool of Allocator for reuse.

5. hardware devices
Functions in graph

The following is a document by Lu Qiming, which is marked. I think he has a clear description of the hardware.

　
As you know, to improve system stability, Windows operating systems
Hardware operations are separated
Applications generally cannot directly access the hardware. DirectShow filter works in user mode. The operating system privilege level is ring.
3) while the hardware works in kernel mode (the operating system privilege level is ring 0), how can they work together?

　　
The DirectShow solution is to set
Calculation
Packaging filter; this filter can work in the user mode, the appearance and control method is the same as the ordinary filter, and the packaging filter is complete with the hardware driver.
. This design enables the development of DirectShow applications.
Personnel, freed from the special processing required to support hardware. DirectShow integrated packaging filter, including
Audio capture filter (qcap. dll), VFW capture filter (qcap. dll, filter class
The ID is clsid_vfwcapture), TV tuner filter (kstvtune. Ax, filter class
ID is clsid_ctvtunerfilter), analog video crossbar filter (ksxbar. Ax), TV
Audio filter (filter class
ID is clsid_tvaudiofilter). In addition, DirectShow designs ksproxy for hardware using WDM drivers.
Filter (ksproxy. Ax ,). Let's take a look at the structure: See figure 1.

We can see that ksproxy. Ax, kstune. Ax,
Ksxbar. Ax packaging filters are used with other common DirectShow
The filter is at the same level and can work collaboratively. In user mode, the filter uses stream
Class controls the hardware driver Minidriver (the DLL provided by the hardware vendor to implement the hardware control function); stream
Class and Minidriver provide services at the underlying level of the system together. It is worth noting that the stream here
Class is a driver model that calls the Minidriver of hardware. In addition, stream
The function of class is to coordinate the work between Minidriver, so that some data can be directly stored in
Transmission from one hardware to another (or different functional modules on the same hardware) in mode improves the system efficiency. (For more details about the underlying driver, see
Windows DDK .)

Next, let's take a look at several common hardware types.

VFW video capture card. This type of hardware is already available in the market
It is a trend of elimination. New video capture cards generally adopt WDM driving models. However, DirectShow provides a packaging filter to ensure backward compatibility.
Hardware. Similar to packaging filters of other hardware, the packaging filter does not use cocreateinstance like a common filter, but must be created through the system
And then bindtoobject.

Audio Acquisition Card (Sound Card ). The collection function of the sound card is also implemented by packaging the filter. Most of the current sound cards
Sound mixing is available. This filter generally has several input pins. Each pin represents an input, such as line.
In, microphone, CD, and Midi. It is worth noting that these pins represent the physical input terminals on the sound card.
Graph is never connected to other filters. The sound card output function can have two filters: directsound Renderer
Filter and audio Renderer (waveout)
Filter. Note that these two filters are not packaging filters in the above sense. They can interact with hardware because they use API functions: the former uses
Directsound API, which uses waveout
API. The difference between the two filters is that audio mixing is not supported when the latter outputs audio. (By The Way, video Renderer
The filter can access the video card because it uses GDI, DirectDraw, or direct3d.
Api .) If your machine has a sound card, you can use graphedit
The packaging filter of the sound card is displayed in the sources directory.

WDM driver hardware (including video capture card and hardware Decompression Card ). This type of hardware is used
Ksproxy. Ax: The packaging filter. Ksproxy. Ax has many features, so it is known as the Swiss Army knife. It is also called a "Chameleon filter" because
The filter defines a unified interface, and the implementation of the interface varies with the specific hardware driver. In filter graph, ksproxy
The name displayed by the filter is the hardware friendly name (usually defined in the. inf file of the driver ). We can use graphedit
Find the WDM hardware installed in the Local System in the directory starting with streaming. Because ksproxy. Ax can represent various WDM audio and video devices,
The workflow is a bit complicated. This filter does not know in advance what type of device to represent. It must first access the property set of the driver and then dynamically configure the interface to be implemented on the filter.

　
When ksproxy
When the interface method on the filter is called by the application or other filters, it will pass the call method and parameters to the driver, and the driver will finally complete the specified function. In addition, WDM hard
File also supports kernel stream (Kernel
Streaming), that is, data transmission in kernel mode, without switching to user mode. Because the conversion between the kernel mode and the user mode requires a large amount of computing. If you are using
The core stream not only avoids a large amount of computing, but also avoids the copy process between the kernel data and the host memory. In this case, the user-Mode Filter
In graph, even if the PIN is connected, there is no actual data flow. Typical scenarios, such as video capture cards with video port pin and images displayed during Preview
It is directly transmitted to the video memory of the video card in kernel mode. Therefore, you do not want to intercept data streams after the VP pin.

　
At this point, I think you should have a general understanding of DirectShow's support for hardware. For application developers, this content does not need to be thoroughly researched, but only needs to be
Just take a look at the background knowledge. In fact, a lot of tedious work DirectShow has already helped us.