DES creation process

DES (DirectShow Editing Services) is a set of programming interfaces based on the core technology framework of DiretShow. Its appearance simplifies the video editing task and makes up for the inherent deficiency of DirectShow's support for non-linear editing of media files.

DES Architecture

Timeline: Organizes information sets of various media sources, audio and video effects, and transition effects, which actually represent the final video clip.

XML Parser: converts the Timeline structure to an XML file for saving, or generates the corresponding Timeline from the XML file.

Render Engine: converts Timeline into a DShow Filter to implement the output control Engine.

Media Locator: used to locate a Media file.

Timeline-based DES Model

In DES, Timeline is used to represent a Video Editing Project, which is also called a Video Editing process. The process of editing cannot be performed without a media source. In DES, media sources can be audio and video files or even images. A linear connection of sources forms a Track. Note that when we use DES to create a Timeline, the audio and video must be on different tracks.

For processing multiple audio and video Tracks, we can add Effects and Transitions between videos ). DES provides more than 100 kinds of SMPTE transition effects (Transitions), and we can also use the various audio and video effects (Effect) or Transitions that come with IE ). (If you want to know the specific transition effect, run dxsdk root \ Samples \ C ++ \ DirectShow \ Bin \ TransViewer.exe.

/////

Technical Principles
DES (DirectShow Editing Services) is a set of programming interfaces based on the core framework of DirectShow. The emergence of DES simplifies video editing tasks and makes up for the inherent shortcomings of DirectShow in support for non-linear editing of media files. However, as far as technology is concerned, DES does not go beyond the DirectShow Filter architecture, but is only an enhanced application of DirectShow Filter. We can learn about the position of DES in our entire multimedia processing application.

 
Next, let's take an example to see what DES can bring to us. Suppose we have three files A, B, and C, and use these three files to form A synthetic file. We want to take the 4 seconds of A, the 10 seconds of B, and the 5 seconds of C. If this is the case, it is not difficult to directly use DirectShow Filter. (Generally, the application maintains the editing information of each file. The application dynamically creates/controls a single Filter Graph based on the information, process each file in sequence .) However, if our "idea" is changed at any time, we want C to appear before B, or we want to take 10 seconds of content in A's different positions, or we want to add a wonderful background music to the entire merged file. If we still use DirectShow Filter directly, the situation will become very complicated. However, for DES, this is really a small Case! (All the editing information is sent to DES using the interfaces provided by DES. The creation/control output of other filters such as Filter Graph is completely handed over to DES for responsibility! In this case, the Filter Graph created by DES has various Source output control functions, which are generally complicated .)

If we use DES, we can also get the following convenience:

1. The Timeline structure and the concept of Track make the Organization and editing of multimedia files intuitive and efficient;
2. Supports instant preview;
3. Video Editing projects support saving XML documents;
4. supports video/audio effects and transition between videos;
5. You can directly use more than 100 SMPTE Transition effects provided by DES, as well as various Transform and Transition components provided by ms ie;
6. Supports the synthesis of images through the color tone, brightness, RGB value or alpha value;
7. automatically adjust the frame rate and audio sampling rate of the video output from the source file, directly supporting video scaling.

Next, let's take a look at the DES structure (Timeline model), as shown in:

 

This is a tree structure. In this tree, audio and video files are leaf nodes and are called as Source; one or more sources form a Track, and each Track has a unified media format output; A set of tracks is called a Composition. Each Composition can be used to edit all its compositions or tracks. Top-level compositions or tracks constitute a Group; each Group outputs a media stream in a single format. All groups form a Timeline. Timeline indicates a video editing project, which is the root node of the tree. A Timeline project must contain at least one Group. The most typical case is that it generally contains two groups: Audio Group and Video Group.
Next, let's look at a typical Timeline-based Source Track orchestration. For example:

 

In the figure, the direction of the arrow is the direction of Timeline. This Timeline consists of two groups, each of which contains two Source tracks. In a Group, a Track has a priority (Track 0 has the lowest priority, and so on ). At runtime, the Source content in the high-priority Track is always output. If there is no Source output in the high-priority Track at this time, let the Source output in the low-priority Track. For example, the output sequence of the Video Group is Source A-> Source C-> Source B. For the Audio Group, all its Track outputs are simply merged.
Let's look at a typical Track with a Transition Timeline structure. For example:

 

In the figure, the Video Group contains two tracks and several Source orchestrations on the Track. The Rendered video shows the final output effect of the Group. We can see that there is a Transition on Track 1, which indicates the effect of Transition from Track 0 to Track1 in this time period. Generally, Transition is on a high-priority Track. Transition also has a direction. By default, it is a Transition from a low-priority Track to a high-priority Track. Of course, we can also change the direction of Transition. As shown in, the first Transition is from Track 0 to Track 1, and the second Transition is from Track 1 to Track 0.

 

It is worth noting that the Transition used by DES adopts the technology called DirectX Transform Object. Any DirectX Transform Object with two inputs and one output can be used as a Transition. Unfortunately, Microsoft's current DirectX SDK does not support the development of such components. What we can use is only the effect provided by DES, and the effect of Microsoft Internet Explorer. DES uses similar Effect, except that DES Effect is a DirectX Transform Object output by a single input and a single output.

//////

As shown in the preceding architecture diagram, multiple tracks (or one) constitute Composition, while multiple compositions (or one) constitute a Group, while Group (s) constitutes a Timeline. If the Video editing process includes Audio and Video, the Timeline must have at least two groups (Video Group & Audio Group ).

The above is a time abstract diagram of the example Timeline in the SDK. DES will adopt the following methods in case of Time overlap:

1. For Video groups, Track1 has a higher priority than Track0. If overlapping occurs on the Video Group, the image above Track1 will be displayed.

2. For the Audio Group, DES will adopt the mixed (Mixing) method for processing.

When creating Timeline, we have two concepts of time:

1. Timeline Time (Timeline Time): The Time relative to the entire Timeline project.

2. Media Time: the Time relative to the Media source, that is, the start Time relative to the Media file.

Use Timeline
Create Timeline
Let's take the SDK example to explain how to create Timeline. This example is located in dxsdk root \ Samples \ C ++ \ DirectShow \ Editing \ TimelineTest.

First, we will notice a preprocessing macro at the beginning of TimelineTest. cpp:

#001 # ifdef STRICT

#002 # undef STRICT

#003 # endif

Let's take a look at the STRICT macro. In MSDN, the STRICT macro is used for STRICT type detection. Windows. the h header file defines a series of macros and structures used to make the compiled code run between different Windows versions. When we compile the code in a STRICT environment, the four data types will change (only one of the cases is listed below, and others can be found in MSDN ):

1. when STRICT is not defined, all HANDLE in Windows is regarded as interger, which means that if we pass an HWND to HDC, this situation will be allowed. Otherwise, the compiler reports an error if STRICT is set. This is because the compiler performs strict type detection.

Let's return to the sample program. In this example, the Timeline creation process is in the TimelineTest function.

Variable declaration:

#001 CComPtr <IRenderEngine> pRenderEngine;

#002 CComPtr <IGraphBuilder> pGraph;

#003 CComPtr <IVideoWindow> pVidWindow;

#004 CComPtr <IMediaEvent> pEvent;

#005 CComPtr <IAMTimeline> pTimeline;

#006 CComPtr <IAMTimelineObj> pVideoGroupObj;

#007 CComPtr <IAMTimelineObj> pAudioGroupObj;

#008 CComPtr <IMediaControl> pControl;

#009 CComPtr <IMediaSeeking> canceking;

All variables use CComPtr in ATL Libraries (the macro works like a smart pointer, so you don't have to worry about releasing resources when using it ). The declare IRenderEngine interface of the first variable. This interface is used to create a Filter Graph by creating a Timeline for future preview or output files.

Then how should we create Timeline? First, let's recall the creation process of COM variables. Create a COM object through CoCreateInstance and QueryInterface. So what are the differences between them. CoCreateInstance creates an uninitialized object through a specific CLSID ). The QueryInterface is called to obtain the interface of the called object to provide additional operations.

Therefore, we should first create a Timeline object (all Exception Handling is omitted ):

#001 hr = CoCreateInstance (

#002 CLSID_AMTimeline,

#003 NULL,

#004 CLSCTX_INPROC_SERVER,

#005 IID_IAMTimeline,

#006 (void **) & pTimeline

#007 );

Next, we will create a Video Group.

#001 hr = pTimeline-> CreateEmptyNode (& pVideoGroupObj, TIMELINE_MAJOR_TYPE_GROUP );

#002

#003 CComQIPtr <IAMTimelineGroup, & IID_IAMTimelineGroup> pVideoGroup (pVideoGroupObj );

#004

#005 CMediaType VideoGroupType;

#006

#007 // all we set is the major type. The group will automatically

#008 // use other defaults

#009 VideoGroupType. SetType (& MEDIATYPE_Video );

#010 hr = pVideoGroup-> SetMediaType (& VideoGroupType );

We use IAMTimeline: CreateEmptyNode to create the IAMTimelineObj interface. At this time, we get a pointer to this interface. IAMTimeline: The enumerated type of the object to be created is passed by the second parameter of CreateEmptyNode. The enumerated type is defined as follows:

#001 typedef enum {

#002 TIMELINE_MAJOR_TYPE_COMPOSITE = 1,

#003 TIMELINE_MAJOR_TYPE_TRACK = 2,

#004 TIMELINE_MAJOR_TYPE_SOURCE = 4,

#005 TIMELINE_MAJOR_TYPE_TRANSITION = 8,

#006 TIMELINE_MAJOR_TYPE_EFFECT = 16,

#007 TIMELINE_MAJOR_TYPE_GROUP = 128

#008} TIMELINE_MAJOR_TYPE;

It is worth noting that each DES object implements the IAMTimelineObj interface, and each specific object implements its own special interface, as shown below:

1. Source: IAMTimelineSrc, IAMTimelineEffectable, IAMTimelineSplittable;

2. Track: IAMTimelineTrack, IAMTimelineVirtualTrack, IAMTimelineEffectable, IAMTimelineTransable, AMTimelineSplittable;

3. Composition: IAMTimelineComp, IAMTimelineVirtualTrack, IAMTimelineEffectable, IAMTimelineTransable;

4. Group: IAMTimelineGroup, IAMTimelineComp;

5. Effects: IAMTimelineEffect, IAMTimelineSplittable;

6. Transitions: IAMTimelineTrans, IAMTimelineSplittable;

If it is not created yet, we must add the Group to the Timeline:

#001 hr = pTimeline-> AddGroup (pVideoGroupObj );

In the same process, we create a Track and add it to Timeline:

#001 CComPtr <IAMTimelineObj> pTrack1Obj;

#002 hr = pTimeline-> CreateEmptyNode (& pTrack1Obj, TIMELINE_MAJOR_TYPE_TRACK );

#003

#004 //--------------------------------------------

#005 // tell the composition about the track

#006 //--------------------------------------------

#007

#008 CComQIPtr <IAMTimelineComp, & IID_IAMTimelineComp> pRootComp (pVideoGroupObj );

#009 hr = pRootComp-> VTrackInsBefore (pTrack1Obj,-1 );

We call VTrackInsBefore during the process of adding. The second parameter of this function is the priority of the inserted track in composition. If you want to insert the Track to the end of the priority (that is, the default insertion), use the-1 parameter.

Add Source and set the Source Time in Timeline and Media Time ):

#001 CComPtr <IAMTimelineObj> pSource1Obj;

#002 hr = pTimeline-> CreateEmptyNode (& pSource1Obj, TIMELINE_MAJOR_TYPE_SOURCE );

#003

#004 // set up source right

#005 //

#006 hr = pSource1Obj-> SetStartStop (TLStart, TLStop );

#007 CComQIPtr <IAMTimelineSrc, & IID_IAMTimelineSrc> pSource1Src (pSource1Obj );

#008

#009 hr | = pSource1Src-> SetMediaTimes (MediaStart, MediaStop );

#010 hr | = pSource1Src-> SetMediaName (pClipname );

Timeline has been set up in the above process, and Source, Track, and Composition have been set up. Without adding other Effects and Transitions, we can create a Filter Graph and Preview It. But let's introduce how to add Transition.

Create Transitions
The process of establishing Transitions (including all the IAMTimelineObj creation processes) is actually calling the IAMTimeline interface CreateEmtpyNode (they pass different type parameters ).

#001 CComPtr <IAMTimelineObj> pTrackTransObj;

#002 hr = pTimeline-> CreateEmptyNode (& pTrackTransObj,

#003 TIMELINE_MAJOR_TYPE_TRANSITION );

Then we set the Transitions (DXT_Jpeg here) and the execution time of the transition.

#001 REFERENCE_TIME TransStart = 0 * UNITS;

#002 REFERENCE_TIME TransStop = 4 * UNITS;

#003

#004 // we set the CLSID of the DXT to use instead of

#005 // pointer to the actual object. We let the DXT

#006 // have it's default properties.

#007 //

#008 hr = pTrackTransObj-> SetSubObjectGUID (CLSID_DxtJpeg );

#009 hr | = pTrackTransObj-> SetStartStop (TransStart, TransStop );

#010

#011 CComQIPtr <IAMTimelineTrans, & IID_IAMTimelineTrans> pTrackTrans (pTrackTransObj );

#012 hr | = pTransable-> TransAdd (pTrackTransObj );

But now we can't end it. There are 207 transition methods for DXT_Jpeg. How can we decide which one to use? MSDN has a detailed introduction. After the transition mode is selected, how can we set it? You don't have to worry about it. The following shows the sample code.

#001 CComPtr <IPropertySetter> pTransSetter;

#002 hr = CoCreateInstance (CLSID_PropertySetter, NULL, CLSCTX_INPROC_SERVER,

#003 IID_IPropertySetter, (void **) & pTransSetter );

#004

#005 DEXTER_PARAM Param;

#006 CComBSTR ParamName ("MaskNum"); // the property name

#007 Param. Name = ParamName;

#008 Param. nValues = 1; // how many values we want to set

#009

#010 DEXTER_VALUE Value;

#011 memset (& Value, 0, sizeof (Value ));

#012 VariantClear (& Value. v );

#013 V_I4 (& Value. v) = 128; // mask number 128

#014 V_VT (& Value. v) = VT_I4; // integer

#015

#016 hr = pTransSetter-> AddProp (Param, & Value );

#017 hr | = pTrackTransObj-> SetPropertySetter (pTransSetter );

Audio Processing is similar to video processing. We will not introduce it here. If necessary, open the source code in the SDK and he will explain it to you. Now it's almost J. Let's preview it.

Preview Timeline
The entire process saves me the trouble of exception handling, but here I still need to introduce ValidateSourceNames. This function checks the validity of the Source. When we are between previews or output files, the validity of the file source is necessary (I used to be in an endless loop) L.

#001 //----------------------------------------------

#002 // make sure files are in their correct location

#003 //----------------------------------------------

#004

#005 hr = pTimeline-> ValidateSourceNames (

#006 SFN_VALIDATEF_CHECK | SFN_VALIDATEF_POPUP | SFN_VALIDATEF_REPLACE,

#007 NULL,

#008 0 );

#009 ASSERT (! FAILED (hr ));

After talking about this for a long time, IRenderEngine should also be available. Let's create the IRenderEngine:

#001 hr = CoCreateInstance (

#002 CLSID_RenderEngine,

#003 NULL,

#004 CLSCTX_INPROC_SERVER,

#005 IID_IRenderEngine,

#006 (void **) & pRenderEngine );

Recall the role of IRenderEngine. This interface is used to create a Filter Graph for previewing or output files. So we need to pass the Timeline information to it.

#001 // tell the render engine about the timeline it shoshould look

#002 //

#003 hr = pRenderEngine-> SetTimelineObject (pTimeline );

The next process is simple. Use ConnectFrontEnd to connect to the Filter created in the Timeline part, and RenderOutputPins to preview (the Renderer is automatically created and the Filter is connected for preview ).

#001 //--------------------------------------------

#002 // connect up the front end, then the back end

#003 //--------------------------------------------

#004

#005 hr = pRenderEngine-> ConnectFrontEnd ();

#006 hr | = pRenderEngine-> RenderOutputPins ();

The final process is to run Graph.

#001 hr = pRenderEngine-> GetFilterGraph (& pGraph );

#002 hr | = pGraph-> QueryInterface (IID_IMediaEvent,

#003 (void **) & pEvent );

#004 hr | = pGraph-> QueryInterface (IID_IMediaControl,

#005 (void **) & pControl );

#006 hr | = pGraph-> QueryInterface (IID_IMediaSeeking,

#007 (void **) & pSeeking );

#008 hr | = pGraph-> QueryInterface (IID_IVideoWindow,

#009 (void **) & pVidWindow );

#010

#011 long lStyle = 0;

#012 hr = pVidWindow-> get_WindowStyle (& lStyle );

#013

#014 lStyle & = ~ (WS_MINIMIZEBOX | WS_MAXIMIZEBOX | WS_SYSMENU );

#015 hr = pVidWindow-> put_WindowStyle (lStyle );

#016

#017 //--------------------------------------------

#018 // run it

#019 //--------------------------------------------

#020

#021 hr = pControl-> Run ();

This ends. For the file writing section, see how DES writes files.