GStreamer Basic Tutorial 08--pipeline Quick access

Goal

The pipeline created by GStreamer does not need to be completely closed. There are several ways to send data to pipeline at any time, or to remove it from the pipeline. This tutorial shows you:

How to send external data to pipeline

How to get the data out of the pipeline

How to manipulate this data

Introduced

There are several ways to enable an app to interact with the data flow through pipeline. This tutorial covers the simplest of these, because the element created specifically for this is used.

The APPSRC is designed to allow applications to pipeline the element that is passed into the data, and Appsink is the opposite, allowing the application to get data from pipeline. To avoid confusion, we can understand that APPSRC is an ordinary source element, but its data is from outer space, and Appsink is an ordinary sink element, and the data is gone out of the way.

APPSRC and Appsink use a lot of them, so they all provide their own APIs, so you can access them just by connecting to the Gstreamer-app library. In this tutorial, we will use a simple method to implement the signal.

APPSRC can have different modes of operation: in pull mode, request data to the app when needed, and in push mode, the app pushes the data to its own rhythm. Also, in push mode, if enough data is available, the application can be blocked on push, or it can be controlled via enough-data and need-data signals. The examples in this tutorial take the form of this signal control, and other methods that are not mentioned can be found in the APPSRC documentation.

Buffers

Large chunks of data passed through pipeline are called buffers. Because this example produces data and consumes data, we need to understand gstbuffer.

Source Pads is responsible for manufacturing buffer, which is consumed by the sink pad. The GStreamer passes these buffer between each element.

A buffer can simply describe a small piece of data, and don't assume that all of our buffer is the same size. Also, buffer has a timestamp and expiration date, which describes when the data in the buffer needs to be rendered. Time stamping is a very complex and profound topic, but this simple explanation is enough now.

As an example, a FILESRC will provide an "any" attribute of buffers and no timestamp information. After Demux ("GStreamer Basic tutorial 03--Dynamic Pipeline"), buffers will have some specific caps, such as "video/x-h264", after decoding, Each buffer contains a frame with the original caps (for example: VIDEO/X-RAW-YUV), and a very clear timestamp is used to indicate when the frame will be displayed.

Tutorial

This tutorial is an extension of the previous tutorial ("GStreamer Basic tutorial 07--Multithreading and Pad effectiveness") in two ways: the first is to use APPSRC to replace the AUDIOTESTSRC to generate audio data, and the second is to add a new branch in tee so that it flows into audio Sink and waveform display also copied a copy of the data to Appsink. This appsink the information back to the application, and the application notifies the user of the data being received or other more complex work.

A rough waveform generator.

#include <gst/gst.h> #include <string.h> #define CHUNK_SIZE 1024x768/* Amount of bytes We is sending in EAC H buffer */#define SAMPLE_RATE 44100/* Samples per second we are sending */#define AUDIO_CAPS "Audio/x-raw-int,channels =1,rate=%d,signed= (Boolean) true,width=16,depth=16,endianness=byte_order "/* Structure to contain all our information, So we can pass it to callbacks */typedef struct _CUSTOMDATA {gstelement *pipeline, *app_source, *tee, *audio_queue, *a
  Udio_convert1, *audio_resample, *audio_sink;
  Gstelement *video_queue, *audio_convert2, *visual, *video_convert, *video_sink;
  
  Gstelement *app_queue, *app_sink;   Guint64 Num_samples;     /* Number of samples generated so far (for timestamp generation) */Gfloat A, B, C, D;        /* for waveform generation */Guint SourceID;  /* To control the Gsource */Gmainloop *main_loop;
  
/* GLib ' s Main Loop */} CustomData; /* This method was called by the idle Gsource in the Mainloop, to feed CHUNK_size bytes into appsrc. * The idle handler is added-the mainloop when APPSRC requests US-to-start sending data (Need-data signal) * and is REM
 Oved when appsrc have enough data (Enough-data signal).
  */Static Gboolean Push_data (CustomData *data) {Gstbuffer *buffer;
  Gstflowreturn ret;
  int i;
  Gint16 *raw; Gint num_samples = CHUNK_SIZE/2;
  
  /* Because Each sample is a-bits */gfloat freq;
  
  /* Create a new empty buffer */buffer = Gst_buffer_new_and_alloc (chunk_size); /* Set its timestamp and duration */gst_buffer_timestamp (BUFFER) = Gst_util_uint64_scale (Data->num_samples, Gst_se
  COND, sample_rate);
  
  Gst_buffer_duration (BUFFER) = Gst_util_uint64_scale (Chunk_size, Gst_second, sample_rate);
  /* Generate some psychodelic waveforms */raw = (Gint16 *) Gst_buffer_data (BUFFER);
  Data->c + = data->d;
  Data->d-= data->c/1000;
  Freq = 1100 + + * data->d;
  for (i = 0; i < num_samples; i++) {data->a + = data->b;  Data->b-= data->a/freq;
  Raw[i] = (gint16) (* data->a);
  
  } data->num_samples + = Num_samples;
  
  /* Push the buffer into the APPSRC */G_signal_emit_by_name (Data->app_source, "Push-buffer", buffer, &ret);
  
  /* Free, the buffer now, we are doing with it */gst_buffer_unref (buffer);
  if (ret! = GST_FLOW_OK) {/* We got some error, stop sending data */return FALSE;
} return TRUE; }/* This signal callback triggers when APPSRC needs data. Here, we add a idle handler * to the Mainloop-start pushing data into the APPSRC */static void Start_feed (Gstelemen
    T *source, guint size, CustomData *data) {if (Data->sourceid = = 0) {g_print ("Start feeding\n");
  Data->sourceid = G_idle_add ((gsourcefunc) push_data, data);
 }}/* This callback triggers when appsrc have enough data and we can stop sending. * We Remove the idle handler from the Mainloop */static void Stop_feed (Gstelement *source, CustomData *dATA) {if (Data->sourceid! = 0) {g_print ("Stop feeding\n");
    G_source_remove (Data->sourceid);
  Data->sourceid = 0; }}/* The Appsink has received a buffer */static void New_buffer (Gstelement *sink, CustomData *data) {Gstbuffer *
  
  Buffer
  /* Retrieve the buffer */g_signal_emit_by_name (sink, "Pull-buffer", &buffer); 
    if (buffer) {/* The only thing we does in this example are print a * to indicate a received buffer */g_print ("*");
  Gst_buffer_unref (buffer); }}/* This function was called when an error message was posted on the bus */static void Error_cb (Gstbus *bus, gstmess
  Age *msg, CustomData *data) {gerror *err;
  
  Gchar *debug_info;
  /* Print Error details on the screen */Gst_message_parse_error (msg, &err, &debug_info);
  G_printerr ("Error received from element%s:%s\n", Gst_object_name (MSG-&GT;SRC), err->message);
  G_printerr ("Debugging Information:%s\n", Debug_info debug_info: "None"); G_Clear_error (&err);
  
  G_free (Debug_info);
G_main_loop_quit (Data->main_loop);
  } int main (int argc, char *argv[]) {CustomData data;
  Gstpadtemplate *tee_src_pad_template;
  Gstpad *tee_audio_pad, *tee_video_pad, *tee_app_pad;
  Gstpad *queue_audio_pad, *queue_video_pad, *queue_app_pad;
  Gchar *audio_caps_text;
  Gstcaps *audio_caps;
  
  Gstbus *bus;
  /* Initialize CUMSTOM data structure */memset (&data, 0, sizeof (data)); DATA.B = 1;
  
  /* for waveform generation */DATA.D = 1;
  
  /* Initialize GStreamer */Gst_init (&AMP;ARGC, &AMP;ARGV);
  /* Create the elements */Data.app_source = Gst_element_factory_make ("Appsrc", "Audio_source");
  Data.tee = Gst_element_factory_make ("Tee", "tee");
  Data.audio_queue = Gst_element_factory_make ("queue", "Audio_queue");
  Data.audio_convert1 = Gst_element_factory_make ("Audioconvert", "Audio_convert1");
  Data.audio_resample = Gst_element_factory_make ("Audioresample", "audio_resample"); Data.audio_sink = Gst_element_factory_make ("Autoaudiosink", "Audio_sink");
  Data.video_queue = Gst_element_factory_make ("queue", "Video_queue");
  Data.audio_convert2 = Gst_element_factory_make ("Audioconvert", "Audio_convert2");
  Data.visual = Gst_element_factory_make ("Wavescope", "visual");
  Data.video_convert = Gst_element_factory_make ("Ffmpegcolorspace", "CSP");
  Data.video_sink = Gst_element_factory_make ("Autovideosink", "Video_sink");
  Data.app_queue = Gst_element_factory_make ("queue", "App_queue");
  
  Data.app_sink = Gst_element_factory_make ("Appsink", "App_sink");
  
  /* Create the Empty pipeline */Data.pipeline = gst_pipeline_new ("Test-pipeline");
      if (!data.pipeline | |!data.app_source | |!data.tee | |!data.audio_queue | |!data.audio_convert1 | | !data.audio_resample | | !data.audio_sink | | !data.video_queue | | !data.audio_convert2 | |
      !data.visual | | !data.video_convert | | !data.video_sink | | !data.app_queue | | !data.app_sink) {G_printerr ("not all elementsCould be created.\n ");
  return-1;
  
  }/* Configure Wavescope */G_object_set (data.visual, "shader", 0, "style", 0, NULL);
  /* Configure APPSRC */audio_caps_text = g_strdup_printf (Audio_caps, sample_rate);
  Audio_caps = gst_caps_from_string (Audio_caps_text);
  G_object_set (Data.app_source, "caps", audio_caps, NULL);
  G_signal_connect (Data.app_source, "Need-data", G_callback (Start_feed), &data);
  
  G_signal_connect (Data.app_source, "Enough-data", G_callback (Stop_feed), &data);
  /* Configure Appsink */G_object_set (Data.app_sink, "emit-signals", TRUE, "caps", audio_caps, NULL);
  G_signal_connect (Data.app_sink, "New-buffer", G_callback (New_buffer), &data);
  Gst_caps_unref (Audio_caps);
  
  G_free (Audio_caps_text); /* Link All elements this can be automatically linked because they has "always" pads */Gst_bin_add_many (Gst_bin (data 
    . Pipeline), Data.app_source, Data.tee, Data.audio_queue, Data.audio_convert1, Data.audio_resample,  Data.audio_sink, Data.video_queue, Data.audio_convert2, Data.visual, Data.video_convert, Data.video_sink, Data.app_
  Queue, Data.app_sink, NULL);
      if (Gst_element_link_many (Data.app_source, data.tee, NULL)! = TRUE | |
      Gst_element_link_many (Data.audio_queue, Data.audio_convert1, Data.audio_resample, Data.audio_sink, NULL)! = TRUE | | Gst_element_link_many (Data.video_queue, Data.audio_convert2, Data.visual, Data.video_convert, Data.video_sink, NULL
      ) = TRUE | | 
    Gst_element_link_many (Data.app_queue, Data.app_sink, NULL)! = TRUE) {g_printerr ("Elements could not being linked.\n");
    Gst_object_unref (Data.pipeline);
  return-1; }/* Manually link the Tee, which has "Request" pads */tee_src_pad_template = Gst_element_class_get_pad_template (
  Gst_element_get_class (Data.tee), "src%d");
  Tee_audio_pad = Gst_element_request_pad (data.tee, tee_src_pad_template, NULL, NULL); G_print ("obtained request pad%s for audio branch.\n", Gst_pad_get_name (Tee_audio_pad));
  Queue_audio_pad = Gst_element_get_static_pad (Data.audio_queue, "sink");
  Tee_video_pad = Gst_element_request_pad (data.tee, tee_src_pad_template, NULL, NULL);
  G_print ("obtained request pad%s for video branch.\n", Gst_pad_get_name (Tee_video_pad));
  Queue_video_pad = Gst_element_get_static_pad (Data.video_queue, "sink");
  Tee_app_pad = Gst_element_request_pad (data.tee, tee_src_pad_template, NULL, NULL);
  G_print ("obtained request pad%s for app branch.\n", Gst_pad_get_name (Tee_app_pad));
  Queue_app_pad = Gst_element_get_static_pad (Data.app_queue, "sink");
      if (Gst_pad_link (Tee_audio_pad, queue_audio_pad)! = GST_PAD_LINK_OK | |
      Gst_pad_link (Tee_video_pad, queue_video_pad)! = GST_PAD_LINK_OK | |
    Gst_pad_link (Tee_app_pad, queue_app_pad)! = GST_PAD_LINK_OK) {G_printerr ("Tee could not be linked\n");
    Gst_object_unref (Data.pipeline);
  return-1;
  } gst_object_unref (Queue_audio_pad);
  Gst_object_unref (Queue_video_pad); Gst_object_unref (Queue_app_pad); /* Instruct the bus to emit signals-received message, and connect to the interesting signals */bus = Gst_eleme
  Nt_get_bus (Data.pipeline);
  Gst_bus_add_signal_watch (bus);
  G_signal_connect (G_object bus), "Message::error", (gcallback) ERROR_CB, &data);
  
  Gst_object_unref (bus);
  
  /* Start playing the pipeline */gst_element_set_state (data.pipeline, gst_state_playing);
  /* Create a GLib Main Loop and set it to run */Data.main_loop = G_main_loop_new (NULL, FALSE);
  
  G_main_loop_run (Data.main_loop);
  /* Release the request pads from the Tee, and unref them */Gst_element_release_request_pad (data.tee, Tee_audio_pad);
  Gst_element_release_request_pad (Data.tee, Tee_video_pad);
  Gst_element_release_request_pad (Data.tee, Tee_app_pad);
  Gst_object_unref (Tee_audio_pad);
  Gst_object_unref (Tee_video_pad);
  
  Gst_object_unref (Tee_app_pad);
  /* FREE Resources */gst_element_set_state (data.pipeline, gst_state_null);Gst_object_unref (Data.pipeline);
return 0;
 }

Work flow

The code to create the pipeline segment is the extended version of the example in the previous tutorial. Includes the initial or all element, the element connected with always Pad, and then manually connect the request Pad of the tee element.

Let's take a look at the configuration of the two element appsrc and Appsink:

  /* Configure APPSRC *
  /Audio_caps_text = g_strdup_printf (Audio_caps, sample_rate);
  Audio_caps = gst_caps_from_string (audio_caps_text);
  G_object_set (Data.app_source, "caps", audio_caps, NULL);
  G_signal_connect (Data.app_source, "Need-data", G_callback (Start_feed), &data);
  G_signal_connect (Data.app_source, "Enough-data", G_callback (Stop_feed), &data);

The first attribute to be concerned with in APPSRC is caps. It illustrates the type of data the element prepares to generate, so that GStreamer can examine the downstream element to see if it supports it. This property must be a Gstcaps object that can be easily generated by gst_caps_from_string ().

We then connect the Need-data and Enough-data signals and callbacks so that the data in the queue inside the APPSRC is not enough or will send a signal at the ground, and we use these signals to start/stop our signal process.

  /* Configure Appsink *
  /G_object_set (Data.app_sink, "emit-signals", TRUE, "caps", audio_caps, NULL);
  G_signal_connect (Data.app_sink, "New-buffer", G_callback (New_buffer), &data);
  Gst_caps_unref (audio_caps);
  G_free (Audio_caps_text);

With regard to the configuration of the Appsink, we connected the new-buffer signal, which is emitted every time the buffer is received. Of course, this signal needs to be emit-signals the signal property is turned on (by default it is off).

Start pipeline, wait until the message and the last cleanup resources are no different from the previous. Let's focus on the callback we just registered.

/* This signal callback triggers when APPSRC needs data. Here, we add a idle handler
 * to the Mainloop-start pushing data into the APPSRC *
/static void Start_feed (Gs TElement *source, guint size, CustomData *data) {
  if (Data->sourceid = = 0) {
    g_print ("Start feeding\n");
    Data->sourceid = G_idle_add ((gsourcefunc) push_data, data);
  }
}

This function is called when the APPSRC internal queue is about to be empty, and what we do here is simply to register a glib idle function with the G_idle_add () method, which will give the APPSRC input data knowing that the internal queue is full. A glib idle function is a method that glib is called when the main loop is "idle", that is, there is no higher-priority task running at that time.

This is just one of several methods of APPSRC data. It is important to note that buffer is not required to be passed to APPSRC using the glib method in the main thread, and you are not necessarily using need-data and enough-data signals to synchronize APPSRC (which is said to be the most convenient).

We record the return SourceID of G_idle_add () so that it can be turned off later.

/* This callback triggers when appsrc have enough data and we can stop sending.
 * We Remove the idle handler from the Mainloop *
/static void Stop_feed (Gstelement *source, CustomData *data) {
  I F (Data->sourceid! = 0) {
    g_print ("Stop feeding\n");
    G_source_remove (Data->sourceid);
    Data->sourceid = 0;
  }
}

This function is called when the APPSRC internal queue is full, so we need to stop sending the data. Here we simply use G_source_remove () to remove the idle function.

/* This method was called by the idle Gsource in the Mainloop, to feed chunk_size bytes into appsrc. * The idle handler is added-the mainloop when APPSRC requests US-to-start sending data (Need-data signal) * and is REM
 Oved when appsrc have enough data (Enough-data signal).
  */Static Gboolean Push_data (CustomData *data) {Gstbuffer *buffer;
  Gstflowreturn ret;
  int i;
  Gint16 *raw; Gint num_samples = CHUNK_SIZE/2;
  
  /* Because Each sample is a-bits */gfloat freq;
  
  /* Create a new empty buffer */buffer = Gst_buffer_new_and_alloc (chunk_size); /* Set its timestamp and duration */gst_buffer_timestamp (BUFFER) = Gst_util_uint64_scale (Data->num_samples, Gst_se
  COND, sample_rate);
  
  Gst_buffer_duration (BUFFER) = Gst_util_uint64_scale (Chunk_size, Gst_second, sample_rate); /* Generate some psychodelic waveforms */raw = (Gint16 *) Gst_buffer_data (BUFFER); 

This function sends data to APPSRC. It is not controlled by the number and frequency of glib calls, but we will close it when its task is complete (APPSRC internal queue full).

The first step here is to create a new buffer with the Gst_buffer_new_and_alloc () method and the given size (1024 bytes in the example).

We calculate the amount of data we generate from the sampled data, customdata.num_samples the data inside, so that we can use the Gst_buffer_timestamp macro provided by Gstbuffer to generate the BUFFER timestamp.

Gst_util_uint64_scale is a tool function that scales data to ensure it does not overflow.

The data for the BUFFER can be accessed using the Gst_buffer_data macros provided by Gstbuffer.

We will skip the generation of waveforms, because this is not what this tutorial is going to tell you.

  /* Push the buffer into the APPSRC *
  /G_signal_emit_by_name (Data->app_source, "Push-buffer", buffer, &ret); 
  /* free, the buffer now, we are doing with it *
  /gst_buffer_unref (buffer);

Once our buffer is ready, we pass the push-buffer signal with this buffer to APPSRC and then call the Gst_buffer_unref () method, because we will not use it anymore.

/* The Appsink has received a buffer *
/static void New_buffer (Gstelement *sink, CustomData *data) {
  Gstbuffer *b Uffer;
  
  /* Retrieve the buffer *
  /g_signal_emit_by_name (sink, "Pull-buffer", &buffer);
  if (buffer) {/
    * The only thing we does in this example are print a * to indicate a received buffer *
    /g_print ("*");
    gst_buffer_unref (buffer);
  }
}

Finally, this function is called when the Appsink receives a buffer. We used the pull-buffer signal to regain buffer, because it is an example, so we just print some content on the screen. We can use Gstbuffer's Gst_buffer_data macro to get the data pointer and use the GST_BUFFER_SIZE macro to get the data size. Keep in mind that the buffer here does not have to be exactly the same as the buffer we created in the Push_data function, and any element on the transmission path could make some changes to buffer. (This example simply passes through a tee element between APPSRC and Appsink, so there is no change in buffer).

Please do not forget to call Gst_buffer_unref () to release buffer, so much so.