Simplifying Linux multimedia development with GStreamer

First, the basic concept

GStreamer is the recommended streaming media application framework for the GNOME desktop environment, with a plug-in (plugin) and pipeline (pipeline) architecture, all functional modules in the framework are implemented as pluggable components (component), And can be easily installed on any pipe when needed, because all the plug-ins through the pipeline mechanism for unified data exchange, it is easy to use the existing various plug-ins "assemble" a fully functional multimedia application.

1.1 Component handling

For programmers who need to apply the GStreamer framework, Gstelement is a must-understand concept because it is the basic component of the pipeline and the basis of all the available components in the framework, which is no wonder that most of the functions in the GStreamer framework involve The operation of the Gstelement object. From the point of view of GStreamer itself, gstelement can be described as a black box with a specific attribute, which interacts with the outside world through the connection point to characterize its own features or functions to the rest of the frame.

According to the differences in their respective functions, GStreamer also divides gstelement into the following categories:

• The source element is only available on the output side, and it can only be used to generate data for pipeline consumption, and it cannot do any processing of the data. An example of a typical data source component is an audio capture unit that is responsible for reading the original audio data from the sound card and then providing it to other modules as a data source.
• The filter element filters elements both input and output, which obtain the corresponding data from the input and are passed to the output after special processing. An example of a typical filter element is an audio coding unit, which first obtains audio data from the outside world, encodes it according to a specific compression algorithm, and then provides the encoded results to other modules for use.
• The Sink element receiver element has only the input, which only has the ability to consume data and is the terminal of the entire media pipeline. An example of a typical receiver element is the audio playback unit, which is responsible for writing the received data to the sound card, which is often the last link in the audio processing process.

Figure 1 will help you better understand the difference between a data source component, a filter element, and a receiver element, and it's not difficult to see how they fit together to form a pipe:

Figure 1

It should be noted that the specific form of the filter element is very flexible, GStreamer does not strictly specify the number of inputs and outputs, in fact, they can be one or more. Figure 2 is the basic structure of an AVI separator, it can separate the input data into separate audio information and video information, the filter element used to implement this function obviously has only one input, but it needs to have two outputs.

Figure 2

The only way to create a Gstelement object in your application is with the factory object gstelementfactory. Because the GStreamer framework provides multiple types of Gstelement objects, several types of Gstelementfactory objects are provided accordingly, which are differentiated by specific factory names. For example, the following code obtains a factory object named Mad through the Gst_element_factory_find () function, which can then be used to create a corresponding MP3 decoder component:

Gstelementfactory *factory;factory = Gst_element_factory_find ("Mad");

Once the factory object has been successfully obtained, it is then possible to create a specific Gstelement object by using the Gst_element_factory_create () function, which has two parameters at the time of invocation, namely the factory object to be used, and the name of the component that will be created. The component name can be obtained by means of a query or by passing in a null pointer (NULL) to generate the default component for the factory object. The following code demonstrates how to use a factory object that has already been obtained to create a MP3 decoder component named decoder:

Gstelement *element;element = Gst_element_factory_create (Factory, "decoder");

When you create a gstelement that is no longer in use, you must also call the Gst_element_unref () function to release the memory resources it occupies:

Gst_element_unref (Element);

GStreamer uses the same mechanism as gobject to manage properties, including query, set, and read (get). All Gstelement objects need to inherit the most basic attribute of name from their parent gstobject, because they are like gst_element_factory_make () and Gst_element_factory_ A function such as create () uses the name attribute when creating the factory object and the Component object, and can set and read the Name property of the Gstelement object by calling the Gst_object_set_name () and Gst_object_get_name () functions.

1.2 Gasket Handling

Padding (PAD) is another basic concept introduced by the GStreamer framework, which refers to the connection channel of the element (element) to the outside world, and for a particular component in the frame, the type of media it can handle is exposed to other components through the liner. Once the Gstelement object has been successfully created, the specified padding for the component can be obtained through Gst_element_get_pad (). For example, the following code returns a gasket named SRC in the element element:

Gstpad *srcpad;srcpad = Gst_element_get_pad (element, "src");

You can also use the gst_element_get_pad_list () function to query all the pads in a specified component, if necessary. For example, the following code outputs the names of all the liners in the element element:

GList *pads;pads = gst_element_get_pad_list (Element), while (pads) {  Gstpad *pad = Gst_pad (pads->data);  G_print ("Pad name is:%s\n", gst_pad_get_name (PAD));  Pads = G_list_next (pads);}

As with components, the name of the liner can be dynamically set or read, which is done by calling the Gst_pad_get_name () and Gst_pad_set_name () functions. The liner of all components can be subdivided into input liner and output liner, wherein the input liner can only receive data but not data, and the output liner is the opposite, can only produce data but not receive data, using the function gst_pad_get_direction () can obtain the type of the specified liner. All pads in the GStreamer frame must be attached to a component, and calling Gst_pad_get_parent () can obtain the component to which the specified gasket belongs, and the return value of the function is a pointer to gstelement. A liner can be seen in some way as a spokesperson for a component, as it is responsible for describing the capabilities of the component to the outside world. The GStreamer framework provides a unified mechanism for the ability of a liner to describe a component (capability), which is achieved by means of a data structure _gstcaps:

struct _gstcaps {  Gchar *name;/* The name of this caps *  /guint16 ID;/* type ID (major type) */  Guint Refcou nt  /* Caps are refcounted */  gstprops *properties;/* Properties for this capability */  gstcaps *next;/* Caps can be Chained together */};

The following is a description of the capability of the mad element, it is not difficult to see that the component actually contains sink and SRC two pads, and each liner has a specific function information. The pad named sink is the input to the mad component, which accepts media data with MIME-type Audio/mp3, and also has layer, bitrate, and framed three properties. The pad named SRC is the output of the mad component, which is responsible for generating MIME-type Audio/raw media data, as well as various properties such as format, depth, rate, and channels.

Pads:  SINK Template: ' SINK '    availability:always    capabilities:    ' mad_sink ':      MIME type: ' audio/ mp3 ':  src Template: ' src '    availability:always    capabilities:      ' mad_src ':        MIME type: ' Audio/raw ':        format:String:int        endianness:integer:1234        width:integer:16        depth:integer:16        Channels : Integer range:1-2        law:integer:0        signed:Boolean:TRUE        rate:integer range:11025-48000

accurately, each liner in the GStreamer framework may correspond to multiple capability descriptions, which can be obtained through function gst_pad_get_caps (). For example, the following code will output the name of all capabilities described in the PAD liner and its MIME type:

Gstcaps *caps;caps = gst_pad_get_caps (PAD) g_print ("Pad name is:%s\n", gst_pad_get_name (PAD)) and while (caps) {  G_PR Int ("Capability name is%s, MIME type is%s\n",  gst_caps_get_name (CAP),  gst_caps_get_mime (CAP));  caps = Caps->next;}

1.3 Case Cabinet

The box cabinet (bin) is a container element in the GStreamer frame, which is often used to accommodate other component objects, but because it is also a Gstelement object, it can actually be used to accommodate other Cabinet objects. By combining multiple components that need to be processed into a single logical element, a cabinet can easily be used to construct more complex piping, since the components in the cabinet are no longer required to operate individually. Another advantage of using cabinets in the GStreamer framework is that it will try to optimize the flow of data, which is attractive for multimedia applications.

Figure 3 depicts the typical structure of the cabinet in the GStreamer frame:

Figure 3

There are two main types of cabinets used in GStreamer applications:

• The Gstpipeline pipeline is the most commonly used container, and for a GStreamer application its top-level cabinet must be a conduit.
• The role of the gstthread thread is to be able to provide the ability to synchronize, if the GStreamer application requires strict audio and video synchronization, it is generally necessary to use this type of cabinet.

The GStreamer framework provides two ways to create a cabinet: one with a factory approach and the other with a specific function. The following code demonstrates how to use the factory method to create a thread object and how to use a specific function to create a pipe object:

Gstelement *thread, *pipeline;//creates a thread object and assigns it a unique name. Thread = Gst_element_factory_make ("thread", NULL);//Creates a specific pipe object based on the given name. Pipeline = gst_pipeline_new ("Pipeline_name");

Once the cabinet has been successfully created, you can call the Gst_bin_add () function to add the existing component to it:

Gstelement *element; Gstelement *bin;bin = gst_bin_new ("Bin_name"), element = Gst_element_factory_make ("mpg123", "decoder"); Gst_bin_add ( Gst_bin (BIN), Element);

It is also easy to find a specific component from the cabinet, which can be achieved with the help of the Gst_bin_get_by_name () function:

Gstelement *element;element = Gst_bin_get_by_name (Gst_bin (BIN), "decoder");

Since one of the cabinets in the GStreamer frame can be added to another cabinet, there may be cases where the enclosure is nested, and the Gst_bin_get_by_name () function will recursively look up nested cabinets when looking for components. After the components have been added to the cabinet, they can also be removed from the container when needed, which is done by calling the Gst_bin_remove () function:

Gstelement *element;gst_bin_remove (Gst_bin (bin), Element);

If you study the cabinets described in 13 carefully, you will find that it does not have its own input pads and output pads, so it is clearly not possible to interact with other components as a logical whole. To solve this problem, GStreamer introduced the concept of Ghost pad, which was chosen from the liner of all the components in the cabinet, and usually the input and output pads were selected, as shown in 4:

Figure 4

A cabinet with a sprite liner behaves exactly the same as a component, with all the properties it has, and all the operations that can be performed on the components are also available for the cabinet, so that they can be used like components in GStreamer applications. The following code demonstrates how to add a sprite pad to a cabinet:

Gstelement *bin; Gstelement *element;element = Gst_element_factory_create ("Mad", "decoder"); bin = gst_bin_new ("Bin_name"); Gst_bin_add (Gst_bin (bin), Element); Gst_element_add_ghost_pad (Bin, Gst_element_get_pad (element, "sink"), "sink");

Back to top of page

Second, the component connection

After introducing the concept of component and liner, the process of processing the multimedia data becomes very clear: by connecting the gaskets of different components to form a media processing pipeline, the data flow through the pipeline can be gstreamer by each component, and finally realize the specific multimedia function.

Figure 1 depicts a very simple pipeline, which consists of three basic components: The data source component is responsible for generating the data, its output liner is connected to the input liner of the filter element, the filter element is responsible for obtaining data from its input liner, and after specific processing, The result is passed to the receiver element connected by the output gasket, the receiver element is responsible for receiving the data, and its input liner is connected with the output liner of the filter element, which is responsible for the corresponding processing of the final result.

The components in the GStreamer frame are connected by their own pads, and the following code demonstrates how to connect two components through a gasket and how to disconnect them when needed:

Gstpad *srcpad, *sinkpad;srcpad = Gst_element_get_pad (element1, "src"); sinpad = Gst_element_get_pad (Element2, "sink") ;//Connect Gst_pad_link (Srcpad, Sinkpad);//Disconnect Gst_pad_unlink (Srcpad, Sinkpad);

If there is only one input liner and one output liner for the components that need to be connected, it is easier to call the Gst_element_link () function to establish a connection directly between them, or call the Gst_element_unlink () function to break the connection between them:

Connection Gst_element_link (element1, element2);//Disconnect Gst_element_unlink (element1, Element2);

Back to top of page

Third, the state of the component

When the components in the GStreamer frame are connected through a pipe, they begin their own process, which typically undergoes multiple state transitions, where each component will be in one of the following four states at a particular moment:

• NULL This is the default state for all components, indicating that it has just been created and has not started to do anything.
• Ready indicates that the component has been prepared and can begin processing processes at any time.
• PAUSED indicates that the component temporarily stops processing data for some reason.
• PLAYING indicates that the component is processing data.

All components start with a null state, which in turn undergoes a transition between null, ready, PAUSED, playing, and so on. The current state of the component can be toggled by calling the Gst_element_set_state () function:

Gstelement *bin;/* Create the component and connect it to the bin */gst_element_set_state (Bin, gst_state_playing) of the cabinet;

By default, the pipeline and all of its contained components will be null after they are created, and they will not do anything at this time. When the pipeline is finished, do not forget to re-switch the state of the pipeline back to the null state, so that all of the components contained in it will have the opportunity to release the resources they are occupying.

The true process of a pipeline begins when it is first switched to the ready state, and the pipeline and all components it contains will be initialized to prepare for the data processing process that is about to be performed. For a typical component, the actions that are required to be in the ready state include opening media files and audio devices, or attempting to establish a connection with a media server located at the far end.

Once a pipeline in the ready state switches to the playing state, the multimedia data that needs to be processed begins to flow through the entire pipeline and is processed sequentially by the individual components contained in the pipeline, which eventually implements some of the multimedia functions that the pipeline has predefined. The GStreamer framework also allows the pipeline to switch directly from a null state to a playing state without having to go through an intermediate ready state.

A pipeline that is in play can switch to the paused state at any time, temporarily stopping all data flow in the pipeline and being able to switch back to the playing state again when needed. If you need to insert or change a component in a pipeline, you must first switch it to the paused or null state, and the component will not release the resources it occupies when it is in the paused state.

Back to top of page

Iv. realization of MP3 player

After understanding some basic concepts and processing processes, here's a look at how to implement a simple MP3 player using the components provided by the GStreamer framework. The structure described in Figure 1 can easily be mapped to a MP3 player, where the data source component is responsible for reading data from the disk, the filter element is responsible for decoding the data, and the receiver component is responsible for writing the decoded data to the sound card.

Like many other GNOME projects, GStreamer is also implemented in C language. If you want to apply the various features provided by GStreamer in your program, you must first call Gst_init () in the main function to do the appropriate initialization to pass the parameters entered by the user from the command line to the GStreamer library. The initialization of a typical GStreamer application is as follows:

#include <gst/gst.h>int main (int argc, char *argv[]) {  gst_init (&ARGC, &argv);  /* ... */}

Next, you need to create three components and connect them to a pipe, since all gstreamer components have the same base class gstelement, so they can be defined as follows:

  Gstelement *pipeline, *filesrc, *decoder, *audiosink;

The pipeline is used to hold and manage components in the GStreamer framework, and the following code creates a new pipeline called Pipeline:

  /* Create a new pipe to hold the component *  /pipeline = Gst_pipeline_new ("Pipeline");

The data source component is responsible for reading the data from the disk file, which has a property named location that indicates where the file is located on disk. Use the standard GObject property mechanism to set the appropriate properties for the symbol:

/* Create data source symbol */FILESRC = Gst_element_factory_make ("Filesrc", "Disk_source"), G_object_set (G_object (FILESRC), "location" , argv[1], NULL);

Filter element is responsible for the completion of the MP3 format data decoding, the simplest way is to install the mad plug-in, using it to complete the corresponding decoding work:

/* Create filter element */decoder = Gst_element_factory_make ("Mad", "decoder");

The receiver component is responsible for playing back the decoded data using the sound card:

/* Create receiver symbol */audiosink = Gst_element_factory_make ("Audiosink", "Play_audio");

All three components that have been created need to be added to the pipeline and connected sequentially:

/* Add component to Pipe */gst_bin_add_many (Gst_bin (pipeline), FILESRC, decoder, Audiosink, NULL);/* Through gasket connection element */GST_ELEMENT_LINK_ Many (filesrc, decoder, Audiosink, NULL);

Once all the preparation is done, you can start the process of processing the entire pipeline by switching the state of the pipeline to the playing state:

/* Start pipe */gst_element_set_state (pipeline, gst_state_playing);

Since no threads are used, it is necessary to determine when the pipeline's processing will end by constantly invoking the Gst_bin_iterate () function:

while (Gst_bin_iterate (Gst_bin (pipeline)));

The Gst_bin_iterate () function returns true as long as there is a new event in the pipeline, and the function returns false only when the entire process is complete, terminating the pipeline and releasing the resources that are consumed:

/* Terminate pipe */gst_element_set_state (pipeline, gst_state_null);/* Release Resource */gst_object_unref (Gst_object (pipeline));

The source code for the MP3 player implemented with GStreamer is as follows:

#include <gst/gst.h>int main (int argc, char *argv[]) {gstelement *pipeline, *filesrc, *decoder, *audiosink;    Gst_init (&AMP;ARGC, &AMP;ARGV);        if (argc! = 2) {g_print ("Usage:%s <mp3 filename>\n", argv[0]);    Exit (-1);    }/* Create a new pipeline */pipeline = Gst_pipeline_new ("Pipeline");    /* Generate a component for reading hard disk data */FILESRC = Gst_element_factory_make ("Filesrc", "Disk_source");    G_object_set (G_object (FILESRC), "location", argv[1], NULL);    /* Create a decoder element */decoder = Gst_element_factory_make ("Mad", "decoder");    /* Create audio playback element */Audiosink = Gst_element_factory_make ("Osssink", "Play_audio");    /* Add the resulting component to the pipeline */Gst_bin_add_many (Gst_bin (pipeline), FILESRC, decoder, Audiosink, NULL);    /* Connect individual components */Gst_element_link_many (FILESRC, decoder, Audiosink, NULL);    /* Start playing */Gst_element_set_state (pipeline, gst_state_playing);    while (Gst_bin_iterate (Gst_bin (pipeline))); /* Stop the pipeline processing process */gst_element_set_state (pipeline, Gst_state_null);    /* Release occupied resources */Gst_object_unref (Gst_object (pipeline));    Exit (0); }

Back to top of page

V. Summary

With the growing popularity of the GNOME desktop environment, GStreamer as a powerful multimedia application development framework, has begun to receive more and more people's attention. GStreamer is designed with a very flexible architecture and offers a number of pre-defined media processing modules, making it much easier to develop multimedia applications under Linux.

Simplifying Linux multimedia development with GStreamer