FFMPEG and SDL Study Notes (1)

Learn FFMPEG and SDL programming based on the example program on the FFMPEG official website.

SDL is a cross-platform multimedia development kit. Suitable for game, simulator, player and other application software development. Supports Linux, Win32, and other operating systems.

Main applications:

Video Sets 8 BPP or a higher video mode with any color depth. If the hardware in a mode is not supported, you can switch to another mode. Directly write a linear image frame buffer (framebuffer ). Use the colorkey or Alpha hybrid attribute to create a surface. Surface bits can be automatically converted to the target format. Blit is optimized and supports hardware acceleration. On the X86 platform, there is a blit optimized for MMX. Hardware-accelerated bits and fill operations, if supported.

Event

• Provides the following events:
  • The visibility of the application has changed.
  • Keyboard Input
  • Mouse input
  • User-requested exit
• Each event can be closed or opened through sdl_eventstate.
• Events are added to the internal event queue by using the user-specified filter function.
• Thread-safe event queue.

Audio

• Set 8-bit and 16-bit audio, single-channel or stereo. If the format hardware does not support this function, you can select conversion.
• The audio part is executed by an independent thread and the user callback mechanism is provided.
• The design takes into account the customer's customized soft mix, but in fact the routine contains a complete audio/music output library.

CD audio

• Complete CD audio control API

Thread

• Simple thread creation API
• Simple binary semaphores for synchronization (semaphores)

Timer

• Reads the number of milliseconds that have elapsed.
• The number of milliseconds to wait.
• Set a periodic timer with a precision of 10 milliseconds.

Byte order independence

• Detects the byte sequence of the current system.
• Functions for quick Data Conversion
• Reads and writes data in the specified byte sequence.

Here We Use SDL as the audio and video output object, and FFMPEG completes audio and video decoding.

Like using other software packages or development libraries, you must first initialize the corresponding libraries before using them. The initialization function is as follows:

If (sdl_init (sdl_init_video | sdl_init_audio | sdl_init_timer ))
{
Fprintf (stderr, "cocould not initialize SDL-% s/n", sdl_geterror ());
Return-1;
}

There are many methods for SDL to achieve video output, but YUV overlay is a simple and common method. The specific method is as follows:

First, create a surface to display video data, then create an overlay, and then output the video to the surface through overlay.

The creation process is as follows:

Int init_sdl (INT width, int height)
{

// Create screen for displaying
Screen = sdl_setvideomode (width, height, 0, 0 );
If (! Screen)
{
Fprintf (stderr, "SDL: cocould not set video mode-exiting/N ");
Return-1;
}

// Now we create a YUV overlay on that screen so we can input video to it:
BMP = sdl_createyuvoverlay (width, height,
Sdl_yv12_overlay, screen );

Return 0;
}

 

After the video is created, the video data can be displayed. I have encapsulated the data as follows:

// Display function. This function is displayed after a complete video frame is extracted.
Void sdl_display (avpicture * pict, sdl_overlay * BMP, Enum pixelformat src_fmt, int width, int height)
{
Sdl_rect rect;

Struct swscontext * img_convert_ctx = NULL;
Avpicture P;

Sdl_lockyuvoverlay (BMP );

P. Data [0] = BMP-> pixels [0];
P. Data [1] = BMP-> pixels [2];
P. Data [2] = BMP-> pixels [1];

P. linesize [0] = BMP-> pitches [0];
P. linesize [1] = BMP-> pitches [2];
P. linesize [2] = BMP-> pitches [1];
 

// Convert the video format to yuv420p.
Img_convert_ctx = sws_getcachedcontext (img_convert_ctx, width, height,
Src_fmt, width, height, pix_fmt_yuv420p,
Sws_x, null );
If (img_convert_ctx = NULL)
{

Printf ("can't init convert context! /N ");
Return;
}
Sws_scale (img_convert_ctx, PICT-> data, PICT-> linesize,
0, width, P. Data, P. linesize );


Sdl_unlockyuvoverlay (BMP );
// Set the position and size of the display area
Rect. x = 0;
Rect. Y = 0;
Rect. W = width;
Rect. H = height;
// Display Video Frames
Sdl_displayyuvoverlay (BMP, & rect );

}

In this way, after decoding a frame of data, you can call this function to display the video.

 

After the video display is completed, it is time to output audio.

To output audio, you must first enable the audio device. SDL encapsulates the Enable and initialization of the audio device.Sdl_openaudio
To open and initialize the audio device.
Sdl_audiospec
Set the corresponding parameters, and then pass
Sdl_openaudio
Set the device and encapsulate it as follows:

Sdl_audiospec audio_spec, spec;

Int init_sdl_audio (avcodeccontext * acodecctx)
{
Audio_spec.freq = acodecctx-> sample_rate;
Audio_spec.format = audio_s16sys;
Audio_spec.channels = acodecctx-> channels;
Audio_spec.silence = 0;
Audio_spec.samples = sdl_audio_buffer_size;
Audio_spec.callback = audio_callback;
Audio_spec.userdata = acodecctx;

If (sdl_openaudio (& audio_spec, & spec) <0)
{
Fprintf (stderr, "sdl_openaudio: % s/n", sdl_geterror ());
Return-1;
}

Return 0;
}

The rest is the same as the video. First, extract the audio stream, then find the decoding context based on the audio stream, then find the decoder based on the decoding context, and enable the decoder. Then, you can perform decoding.

However, we cannot decode the audio package directly like decoding the video. We constantly extract the packet from the file, and SDL must constantly call the callback function. The solution is to create a mutex queue, FFmpeg has encapsulated an avpacketlist struct for us. We need to encapsulate it again as follows:

Typedef struct packetqueue {
Avpacketlist * first_pkt, * last_pkt;
Int nb_packets;
Int size;
Sdl_mutex * mutex;
Sdl_cond * cond;
} Packetqueue;
Note: The size here is the packet size, while the nb_packets is the number of packages in the queue.
A queue must first have an initialization function to complete initialization.
Void packet_queue_init (packetqueue * q ){
Memset (Q, 0, sizeof (packetqueue ));
Q-> mutex = sdl_createmutex ();
Q-> cond = sdl_createcond ();
}
Obviously, this initialization function completes the creation of memory allocation, mutex and condition of the queue.
Then there is the function for joining and leaving the team.
// Put audio packet in the queue
Int packet_queue_put (packetqueue * q, avpacket * Pkt ){

Avpacketlist * pkt1;
// If Pkt is not allocated, allocate it
If (av_dup_packet (Pkt) <0 ){
Return-1;
}
// Allocate space for new member of queue
Pkt1 = av_malloc (sizeof (avpacketlist ));
If (! Pkt1)
Return-1;
// Put Pkt in pkt1
Pkt1-> Pkt = * Pkt;
Pkt1-> next = NULL;

// Lock queue and wait until finishing put
Sdl_lockmutex (Q-> mutex );
// If last_pkt is null, it means that the queue is null, so put the packet in the first position
If (! Q-> last_pkt)
Q-> first_pkt = pkt1;
Else
Q-> last_pkt-> next = pkt1;
Q-> last_pkt = pkt1;
Q-> nb_packets ++;
Q-> size + = pkt1-> Pkt. size;
// Send signal of finish
Sdl_condsignal (Q-> Cond );

Sdl_unlockmutex (Q-> mutex );
Return 0;
}

// Put audio packet in the queue
Static int packet_queue_get (packetqueue * q, avpacket * Pkt, int block ){
Avpacketlist * pkt1;
Int ret;

Sdl_lockmutex (Q-> mutex );

For (;;){

If (quit ){
Ret =-1;
Break;
}

Pkt1 = Q-> first_pkt;
If (pkt1 ){
Q-> first_pkt = pkt1-> next;
If (! Q-> first_pkt)
Q-> last_pkt = NULL;
Q-> nb_packets --;
Q-> size-= pkt1-> Pkt. size;
* Pkt = pkt1-> Pkt;
Av_free (pkt1 );
Ret = 1;
Break;
} Else if (! Block ){
Ret = 0;
Break;
} Else {
Sdl_condwait (Q-> cond, Q-> mutex );
}
}
Sdl_unlockmutex (Q-> mutex );
Return ret;
}
Here, we must note that SDL creates a separate thread for audio processing. The thread calls the callback function to complete decoding of audio frames from the package.
Call the decoding function to decode the audio frames!
Void audio_callback (void * userdata, uint8 * stream, int Len ){

Avcodeccontext * acodecctx = (avcodeccontext *) userdata;
Int len1, audio_size;

Static uint8_t audio_buf [(avcodec_max_audio_frame_size * 3)/2];
Static unsigned int audio_buf_size = 0;
Static unsigned int audio_buf_index = 0;

While (LEN> 0 ){
If (audio_buf_index> = audio_buf_size ){
/* We have already sent all our data; get more */
Audio_size = audio_decode_frame (acodecctx, audio_buf,
Sizeof (audio_buf ));
If (audio_size <0 ){
/* If error, output silence */
Audio_buf_size = 1024;
Memset (audio_buf, 0, audio_buf_size );
} Else {
Audio_buf_size = audio_size;
}
Audio_buf_index = 0;
}
Len1 = audio_buf_size-audio_buf_index;
If (len1> Len)
Len1 = Len;
Memcpy (stream, (uint8_t *) audio_buf + audio_buf_index, len1 );
Len-= len1;
Stream + = len1;
Audio_buf_index + = len1;
}
}
Decoding function
// Decode audio Frame
Int audio_decode_frame (avcodeccontext * acodecctx, uint8_t * audio_buf,
Int buf_size ){

Static avpacket Pkt;
Static uint8_t * audio_pkt_data = NULL;
Static int audio_pkt_size = 0;

Int len1, data_size;

For (;;){
While (audio_pkt_size> 0 ){
Data_size = buf_size;
Len1 = avcodec_decode_audio2 (acodecctx, (int16_t *) audio_buf, & data_size,
Audio_pkt_data, audio_pkt_size );
If (len1 <0 ){
/* If error, skip frame */
Audio_pkt_size = 0;
Break;
}
Audio_pkt_data + = len1;
Audio_pkt_size-= len1;
If (data_size <= 0 ){
/* No data yet, get more frames */
Continue;
}
/* We have data, return it and come back for more later */
Return data_size;
}
If (Pkt. Data)
Av_free_packet (& Pkt );

If (quit ){
Return-1;
}

If (packet_queue_get (& audioq, & Pkt, 1) <0 ){
Return-1;
}
Audio_pkt_data = Pkt. Data;
Audio_pkt_size = Pkt. size;
}
}
// Main Function
Int main ()
{
Avformatcontext * pformatctx;
Avcodeccontext * pcodecctx, * acodecctx;
Avcodec * pcodec, * acodec;
Avstream * st;
Avframe * pframe;
Avpacket packet;
Struct swscontext * img_convert_ctx = NULL;
Sdl_event event;
Uint8_t * buffer;
Sdl_rect rect;
Char * filename = "1.asf ";
Int ret, I, videostream, audiostream, numbytes, framefinished;

// Init SDL library with video and audio
If (sdl_init (sdl_init_video | sdl_init_audio | sdl_init_timer ))
{
Fprintf (stderr, "cocould not initialize SDL-% s/n", sdl_geterror ());
Return-1;
}
// Init the format and CODEC Library
Av_register_all ();

Ret = av_open_input_file (& pformatctx, filename, null, 0, null );
If (Ret <0)
{
Printf ("error1: Open input file failed! /N ");
Return-1;
}
// Retrive stream information
Ret = av_find_stream_info (pformatctx );
If (Ret <0)
{
Printf ("error2: Find stream information failed! /N ");
Return-1;
}

// Dump information about file onto standard error
Dump_format (pformatctx, 0, filename, 0 );

Videostream =-1;
Audiostream =-1;
For (I = 0; I <pformatctx-> nb_streams; I ++)
{
If (pformatctx-> streams [I]-> codec-> codec_type = codec_type_video
& Videostream <0)
{
Videostream = I;
}
If (pformatctx-> streams [I]-> codec-> codec_type = codec_type_audio &&
Audiostream <0)
{
Audiostream = I;
}
}

// Check whether find video stream and audio stream
If (videostream =-1)
{
Printf ("error3: Can't find video stream! /N ");
Return-1;
}
If (audiostream =-1)
{
Printf ("error4: Can't Find audio stream! /N ");
Return-1;
}
// Get video codec Context
St = pformatctx-> streams [videostream];
Pcodecctx = ST-> codec;
// Get audio codec contex

Acodecctx = pformatctx-> streams [audiostream]-> codec;

// Find Video Codec
Pcodec = avcodec_find_decoder (pcodecctx-> codec_id );
If (pcodec = NULL)
{
Printf ("error5: Can't find video decoder! /N ");
Return-1;
}
// Open Video Decoder
Ret = avcodec_open (pcodecctx, pcodec );
If (Ret <0)
{
Printf ("Open Video Decoder failed! /N ");
Return-1;
}

// Allocate Video Frame
Pframe = avcodec_alloc_frame ();

// Init audio codec Context
If (init_sdl_audio (acodecctx) <0)
{
Printf ("error6: init SDL audio failed! /N ");
Return-1;
}
// Find Audio Codec
Acodec = avcodec_find_decoder (acodecctx-> codec_id );
If (acodec = NULL)
{
Printf ("error7: Can't Find audio decoder! /N ");
Return-1;
}
// Open audio decoder
Ret = avcodec_open (acodecctx, acodec );
If (Ret <0)
{
Printf ("error8: open audio decoder failed! /N ");
Return-1;
}

// Init audio packet queue
Packet_queue_init (& audioq );
Sdl_pauseaudio (0 );
// Init overalay
If (init_sdl (pcodecctx-> width, pcodecctx-> height) <0)
{
Printf ("error9: init SDL library failed! /N ");
Return-1;
}
// Do not know why
Url_set_interrupt_cb (decode_interrupt_cb );
// Decode video and audio Frame
While (av_read_frame (pformatctx, & Packet)> = 0)
{
If (packet. stream_index = videostream)
{
// Decode a frame
Avcodec_decode_video (pcodecctx, pframe, & framefinished, packet. Data, packet. size );
// Finish or not?
If (framefinished)
{
Sdl_display (avpicture *) pframe, BMP, pcodecctx-> pix_fmt, pcodecctx-> width, pcodecctx-> height );
}
} Else if (packet. stream_index = audiostream)
{
Packet_queue_put (& audioq, & Packet );

} Else
{
Av_free_packet (& Packet );
}
Sdl_pollevent (& event );
Switch (event. type)
{
Case sdl_quit:
Quit = 1;
Sdl_quit ();
Exit (0 );
Break;
Default:
Break;
}
}
// Free the YUV Frame
Av_free (pframe );

// Close the Codec
Avcodec_close (pcodecctx );

// Close the video file
Av_close_input_file (pformatctx );

Return 0;

}

However, this program does not solve problems such as audio and video synchronization, and video data is displayed very quickly!