FFmpeg document 1: screen recording

Tutorial 1: screen recording

Summary

Movie files have many basic components. First, the file itself is called the container iner. The container type determines the location where information is stored in the file. Avi and QuickTime are examples of containers. Next, you have a set of streams. For example, you often have an audio stream and a video stream. (A stream is just an imaginary word used to represent a series of data elements connected by time ). A data element in a stream is called a frame. Each stream is encoded and generated by different encoders. The decoder describes how the actual data is encoded and decoded, so its name is codec. DivX and MP3 are examples of codecs. The packets is read from the stream. A package contains a piece of data that can be decoded to facilitate our final application.ProgramThe data of the original frame. For our purpose, each package contains a complete frame or a complete frame in many audio formats.

Basically, it is easy to process video and audio streams:

10. Open the video stream video_stream from the video. AVI file.

20. Read packets from the video stream to the frame.

30. If the frame is incomplete, skip to 20.

40. perform operations on this frame

50 to 20

It is quite easy to use FFMPEG in this program to process multiple media, although many programs may be very complicated in frame operations. Therefore, in this tutorial, we will open a file, read the video stream in it, and write the frame to a PPM file.

Open a file

First, let's take a look at how we open a file. Through FFMPEG, you must first initialize this library. (Note that <FFMPEG/avcodec. h> and <FFMPEG/avformat. h> must be used in some systems)

# Include <avcodec. h>

# Include <avformat. h>

...

Int main (INT argc, Charg * argv []) {

Av_register_all ();

All file formats and CODEC libraries are registered here, so they will be automatically used on the files in the appropriate opened format. Note that you only need to call av_register_all () once, so we will call it in the main function main. If you like it, you can also register only the specific format and decoder, but you usually do not need to do so.

Now we can really open the file:

Avformatcontext * pformatctx;

// Open video file

If (av_open_input_file (& pformatctx, argv [1], null, 0, null )! = 0)

Return-1; // couldn't open file

We get the file name through the first parameter. This function reads the header of the file and saves the information to the avformatcontext struct we have given. The last three parameters are used to specify special file formats, buffer sizes, and format parameters. However, if they are set to null or 0, libavformat will automatically detect these parameters.

This function only detects the file header, so we need to check the stream information in the file:

// Retrieve stream information

If (av_find_stream_info (pformatctx) <0)

Return-1; // couldn't find stream information

This function fills in the correct information for pformatctx-> streams. Let's introduce a manually debugged function to see what is in it:

// Dump information about file onto standard error

Dump_format (pformatctx, 0, argv [1], 0 );

Now pformatctx-> streams is just a set of pointers with the size of pformatctx-> nb_streams, so let's skip it until we find a video stream.

Int I;

Avcodeccontext * pcodecctx;

// Find the first video stream

Videostream =-1;

For (I = 0; I <pformatctx-> nb_streams; I ++)

If (pformatctx-> streams [I]-> codec-> codec_type = codec_type_video ){

Videostream = I;

Break;

}

If (videostream =-1)

Return-1; // didn't find a video stream

// Get a pointer to the codec context for the video stream

Pcodecctx = pformatctx-> streams [videostream]-> codec;

The information about the decoder in the stream is what we call "codec context. It contains all the information about the decoder used in the stream. Now we have a pointer to it. However, we must find the real decoder and enable it:

Avcodec * pcodec;

// Find the decoder for the video stream

Pcodec = avcodec_find_decoder (pcodecctx-> codec_id );

If (pcodec = NULL ){

Fprintf (stderr, "unsupported codec! \ N ");

Return-1; // codec not found

}

// Open Codec

If (avcodec_open (pcodecctx, pcodec) <0)

Return-1; // cocould not open Codec

Some may remember from the old guidance that there are twoCodeOthers: Add codec_flag_truncated to pcodecctx-> flags and add an hack to roughly correct the frame rate. These two corrections are no longer in ffplay. C. Therefore, I must assume that they are no longer necessary. After removing the code, there is another difference that needs to be pointed out: pcodecctx-> time_base now stores the frame rate information. Time_base is a struct with a molecule and a denominator (avrational ). We use scores to indicate the frame rate because many codecs use non-integer frame rates (for example, NTSC uses 29.97fps ).

Save data

Now we need to find a place to save the frame:

Avframe * pframe;

// Allocate Video Frame

Pframe = avcodec_alloc_frame ();

Because we want to output a 24-bit RGB ppm file, we must convert the frame format from the original to RGB. FFmpeg will perform these conversions for us. In most projects (including ours), we want to convert the original frame into a specific format. Let's apply for a frame of memory for the conversion first.

// Allocate an avframe Structure

Pframergb = avcodec_alloc_frame ();

If (pframergb = NULL)

Return-1;

Even if we applied for a frame of memory, we still need a place to place the original data during the conversion. We use avpicture_get_size to obtain the required size, and then manually apply for memory space:

Uint8_t * buffer;

Int numbytes;

// Determine required buffer size and allocate Buffer

Numbytes = avpicture_get_size (pix_fmt_rgb24, pcodecctx-> width,

Pcodecctx-> height );

Buffer = (uint8_t *) av_malloc (numbytes * sizeof (uint8_t ));

Av_malloc is the malloc of FFMPEG, which is used to implement a simple malloc packaging. This ensures that the memory address is aligned (4-byte or 2-byte ). It does not protect you from memory leakage, repeated release, or other malloc problems.

Now we use avpicture_fill to combine frames with our new memory. About avpicture formation: The avpicture struct is a subset of the avframe struct-the starting part of the avframe struct is the same as the avpicture struct.

// Assign appropriate parts of buffer to image planes in pframergb

// Note that pframergb is an avframe, but avframe is a superset

// Of avpicture

Avpicture_fill (avpicture *) pframergb, buffer, pix_fmt_rgb24,

Pcodecctx-> width, pcodecctx-> height );

Finally, we are ready to read data from the stream.

Read data

What we will do is read the entire video stream by reading the package, and then decode it into a frame. It is best to convert the format and save it later.

Int framefinished;

Avpacket packet;

I = 0;

While (av_read_frame (pformatctx, & Packet)> = 0 ){

// Is this a packet from the video stream?

If (packet. stream_index = videostream ){

// Decode Video Frame

Avcodec_decode_video (pcodecctx, pframe, & framefinished,

Packet. Data, packet. size );

// Did we get a video frame?

If (framefinished ){

// Convert the image from its native format to RGB

Img_convert (avpicture *) pframergb, pix_fmt_rgb24,

(Avpicture *) pframe, pcodecctx-> pix_fmt,

Pcodecctx-> width, pcodecctx-> height );

// Save the frame to disk

If (++ I <= 5)

Saveframe (pframergb, pcodecctx-> width,

Pcodecctx-> height, I );

}

}

// Free the packet that was allocated by av_read_frame

Av_free_packet (& Packet );

}

This loop process is relatively simple: av_read_frame () reads a package and saves it to the avpacket struct. Note that we only applied for the structure of a package-FFMPEG applied for internal data memory for us and directed it through the packet. Data Pointer. The data can be released later through av_free_packet. Function avcodec_decode_video () converts a package to a frame. However, when decoding a packet, we may not get the required frame information. Therefore, when we get the next frame, avcodec_decode_video () sets the frame end mark framefinished for us. Finally, we use the img_convert () function to convert frames from the original format (pcodecctx-> pix_fmt) to the RGB format. Remember, you can convert an avframe struct pointer to an avpicture struct pointer. Finally, we pass the frame and height width information to our saveframe function.

Comments about packets

Technically, a package can contain some or other data, but the FFMPEG interpreter ensures that the packets we get contains either complete or multiple complete frames.

What we need to do now is to allow the saveframe function to final RGB information to a file in ppm format. We will generate a simple file in ppm format. Believe that it can work.

Void saveframe (avframe * pframe, int width, int height, int IFRAME ){

File * pfile;

Char szfilename [32];

Int y;

// Open File

Sprintf (szfilename, "frame % d. ppm", IFRAME );

Pfile = fopen (szfilename, "WB ");

If (pfile = NULL)

Return;

// Write Header

Fprintf (pfile, "P6 \ n % d \ n255 \ n", width, height );

// Write pixel data

For (y = 0; y

Fwrite (pframe-> data [0] + y * pframe-> linesize [0], 1, width * 3, pfile );

// Close file

Fclose (pfile );

}

We made some standard file opening actions and then wrote RGB data. We write a row of data to the file at one time. A file in the PPM format contains a long string of RGB data. If you understand the HTML color representation method, it is similar to expanding the color headers of each pixel, like # ff0000 # ff0000 .... It indicates a red screen. (It is saved in binary format and has no separator, but you know how to separate it ). The header of the file represents the width and height of the image and the size of the maximum RGB value.

Now, let's review our main () function. Once we start to read the video stream, we must clear everything:

// Free the RGB image

Av_free (buffer );

Av_free (pframergb );

// Free the YUV Frame

Av_free (pframe );

// Close the Codec

Avcodec_close (pcodecctx );

// Close the video file

Av_close_input_file (pformatctx );

Return 0;

You will notice that we use av_free to release the memory we use avcode_alloc_fram and av_malloc to allocate.

The above is the code! Next, we will use Linux or other similar platforms, and you will run:

Gcc-O tutorial01 tutorial01.c-lavutil-lavformat-lavcodec-LZ-lavutil-LM

If you are using the old FFMPEG version, you can remove the-lavutil parameter:

Gcc-O tutorial01 tutorial01.c-lavutil-lavformat-lavcodec-LZ-LM

Most image processing functions can open ppm files. Some movie files can be used for testing.