FFmpeg SDK Development Manual

The more important functions and data structures in FFMpeg are as follows:
1. Data structure:
(1) Avformatcontext
(2) Avoutputformat
(3) Avinputformat
(4) Avcodeccontext
(5) Avcodec
(6) Avframe
(7) Avpacket
(8) Avpicture
(9) Avstream
2. Initialize the function:
(1) Av_register_all ()
(2) Avcodec_open ()
(3) Avcodec_close ()
(4) Av_open_input_file ()
(5) Av_find_input_format ()
(6) Av_find_stream_info ()
(7) Av_close_input_file ()
3. Audio and Video codec function:
(1) Avcodec_find_decoder ()
(2) Avcodec_alloc_frame ()
(3) Avpicture_get_size ()
(4) Avpicture_fill ()
(5) Img_convert ()
(6) Avcodec_alloc_context ()
(7) Avcodec_decode_video ()
(8) Av_free_packet ()
(9) Av_free ()
4. File Operation:
(1) Avnew_steam ()
(2) Av_read_frame ()
(3) Av_write_frame ()
(4) Dump_format ()
5. Other functions:
(1) Avpicture_deinterlace ()
(2) Imgresamplecontext ()
The following is based on the above data structure and function in the FFmpeg test code OUTPUT_EXAMPLE.C in the forward and backward analysis. Before that, let's talk about FFmpeg's compilation problem. The compilation under Linux is relatively simple, here is not much to say. Compiling under Windows can refer to the following Web pages:
Http://bbs.chinavideo.org/viewthread.php?tid=1897&extra=page%3D1
It is worth mentioning that when testing with the compiled SDK (using the output_example.c in the FFmpeg directory), the following two issues may be encountered during the compilation process:
1. Output_example.c used the Snprintf.h this header file. However this header file differs under Win and Linux. Specific under win can be solved in the following ways:
http://www.ijs.si/software/snprintf/
2. If you compile using VC6 or the VC6 command line, inline may not be recognized. Errors appear in the Common.h file and can be added to the Common.h
#ifdef _msc_var
#define Inline __inline
#endif
Confessed to get to the point.
A Data structures in the FFMpeg:
I. Avformatcontext
Typically in code that uses the FFmpeg SDK, Avformatcontext is a data structure that is always used, and many functions use it as a parameter. The comment for this data structure in the FFmpeg code is: Format I/o context
This structure contains the format content of a video stream. There are Avinputformat (or Avoutputformat only one of the avformatcontext within the same time), and Avstream, avpacket these important data structures and some other related information, such as Title,author,copyright and so on. There are also some information that may be used in decoding, such as: Duration, file_size, bit_rate, etc. Refer to the Avformat.h header file.
Useage:
Statement:
Avformatcontext *oc; (1)
Initialization: Because the AVFORMATCONEXT structure contains a lot of information, the initialization process is step-through, and some variables are not initialized if no value is available. However, because the general declaration is using pointers, a process of allocating memory is not limited:
OC = Av_alloc_format_context (); (2)
The avinputformat* (or avoutputformat*) in the structure must be initialized, basically this is the basis of what the compiler code should use codec:
Oc->oformat = FMT; or Oc->iformat = FMT; (3)
Wherein avoutputformat* FMT or avinputformat* fmt. (The initialization of Avinputformat and Avoutputformat is described later.) Then there is a line in the reference code output_example.c:
snprintf (oc-filename, sizeof (Oc->filename), "%s", filename); (4)
It is not clear what the effect is, it is estimated to first write some header information in the output file.
After completing the above steps, (Initialize avinputformat* (or avoutputformat*) and Avformatcontext) Next is the second important structure in the avformatcontext that you will start with OC initialization. Avstream (assuming there is already a declaration Avstream *video_st. The reference code uses a function to complete the initialization, and of course it can be done in the main function, and the arguments passed into the function are OC and FMT->VIDEO_CODEC (this is described in the next section (29)):
Vdeo_st = Add_video_stream (OC, FMT->VIDEO_CODEC); (5)
This function is parsed later when it comes to the avstream structure.
Avformatcontext The final setup work is:
if (Av_set_paramters (oc,null) b_streams; i++) {
Av_freep (&oc->streams->codec); (13)
Av_freep (&oc->streams); (14)
}
Close the ouput file
if (! ( Fmt->flags & Avfmt_nofile)) {
Url_fclose (&OC->PB); (15)
}
Av_free (OC); (16)
With a bunch of code above, it is clear that avformatcontex* oc and avstream* Video_st are the two data structures that have been used throughout the development of the FFmpeg SDK. The following is a brief introduction to three red-labeled functions, which are reference code OUTPUT_EXAMPLE.C developers define themselves. This will make the entire code structure clear, of course, you can also use the FFmpeg SDK in the main function to complete the corresponding function. In the following we will be specifically for these three functions to do analysis.
1. Open_video (OC, Video_st);
This function is primarily the initialization process for a video encoder (or decoder). The initialized data structures for avcodec* codec and avcodeccontext* C include the SDK functions that are used:
c = st->codec;
codec = Avcodec_find_encoder (c->codec_id); Encoder (17) for encoding
codec = Avcodec_find_decoder (c->codec_id); Decode, find decoder (18)
Avcodeccontex is a data structure in the structure avstream, so it is directly complex to C after the initialization of the Avstream (5).
Internal Open Video Codec
Avcodec_open (C,CODEC); (19)
Allocate video stream Buffer
Avframe *picture
uint8_t *video_outbuf
video_outbuf_size=200000;
Video_outbuf = Av_maloc (video_outbuf_size); (20)
Allocate video frame Buffer
Picture = Alloc_picture (C->pix_fmt, C->width, c->height); (21)
These three steps are easier to understand, turn on video codec codec, allocate output stream cache size, allocate each frame image cache size. Among them, Avframe is also one of the main data structures in FFmpeg. This step (8) is the initialization process for the codec.
2. Write_video_frame (Avformatcontext *oc, Avstream *st)
This function does a real coding and decoding work, in which the function is more complex first listed to slowly analyze.
The data structure used is Avcodeccontext *c, Swscontext *img_convert_ctx. Where Swscontext is used to transform the image format. For example, yuv422 change to yuv420, of course, also use the function, see the following list.
Fill_yuv_image (Tmp_picture, Frame_count, C->width, c->height); (22)
Sws_scale (Img_convert_ctx, Tmp_picture->, Tmp_picture->linesize,
0, C->height, Picture->data, picture->linesize); (23)
Img_convert_ctx = Sws_getcontxt (C->width, C->height, pix_fmt_yuv420p, (24)
C->width, C->heigth, c->pix_fmt, sws_flags, NULL, NULL, NULL);
As the reference code does is an encoding. Therefore, it always requires the input of the encoder is YUV file, and is in yuv420 format. There will be some of these processes. The next call to encoder encoding, data-avpacket (packaging) used to the FFmpeg, which is a relatively bad understanding of the place.
Out_size = Avcodec_encode_video (c, Video_outbuf, video_outbuf_size, picture); (25)
Avpacket PKT;
Av_init_packet (&PKT); (26)
... handle PKT process, we'll analyze later
ret = Av_write_frame (OC, &PKT); (27)
If there is encode, there will be decode. And FFmpeg is designed for decoding, but why only encoder is used in the reference code. Personal conjecture is because encode just uses yuv420 to encode, such a yuv420 generation is relatively easy, if the use of decoding, but also in the code with a other format of audio and video files. In the source code Libavcodec folder There is a apiexample.c reference code, which is encoded and decoded. I'll analyze it when I'm free.
3. Close_video (Avformatcontext *oc, Avstream *st)
Avcodec_close (ST->CODEC);
Av_free (Picture->data[0]);
Av_free (picture);
Av_free (VIDEO_OUTBUF);
Easy to understand, not much to say.
The above section, although named as Introduction Avformatcontext. But basically go through the frame of the OUPUT_EXAMPLE.C Video Coding section, one is to explain the importance of structural avformatcontext, on the other hand, but also want to use FFmpeg SDK developers have a general framework.
In fact, some of the real coding functions, memory allocation functions in the SDK are already encapsulated, as long as the structure can be used to understand. What developers are going to do is initialize the process, basically the initialization of data structure 1.
II. Avoutputformat
Although simple (initialized) but very important, he is the "indication" of which codec the codec will use. The most important thing in its member data is about the video codec: enum Codecid Video_codec;
Avoutputformat *fmt;
FMT = Guess_format (null, filename, null); (28)
The file format is judged by filename, and what encoder is initialized. Of course, if is using Avinputformat *fmt, is fix with what decoder. (Specify the output sequence->fix encoder, specifying the input sequence->fix decoder.) ）
III. Avstream
Avstream is justified as the second cross-cutting structure following avformatcontext. There are avcodeccontext in his member data, which is basically set up for the parameters of the video codec that are used (including important information such as bit rate, resolution, etc.). As a "stream", it contains some data from the concept of "flow", such as: Frame rate (r_frame_rate), Basic Time Measurement unit (time_base), first frame position (start_time), Duration (duration) , the number of frames (nb_frames), and some IP information. Of course, some of this information is not required to be initialized, but Avcodeccontex must be initialized, and is the most important part of initializing Avstream. We talked about the initialization function of Avstream (5), and now let's see how he does it:
Declaration
Avstream *video_st;
Video_st = Add_video_stream (OC, FMT->VIDEO_CODEC);
Static Avstream *add_video_stream (Avformatcontex *oc, int codec_id) {(29)
Avcodeccontext *c; Member of Avstream, which'll be initialized here
Avstream *st; Temporary data, would be returned
st = Av_new_stream (OC, 0); (30)
c = st->codec;
The following basic is the initialization process for C. Includes bit rate, resolution, GOP size, and so on.
......
The following two lines need to be noted, especially the use of MP4
if (!strcmp (Oc->oformat->name, "mp4") | |!strcmp (oc->oformat->name, "mov") | |!strcmp (oc->oformat-> Name, "3GP"))
C->flags |= Codec_flag_global_header;
Pass St to Video_st;
Return St;
}
In the above code, there are a few points to note. One is (30) and C = St->codec is sure to do, of course, this is the most basic programming problem, (30) is the St this avsteam bound to avformatcontext* OC. The following C = St->codec is the avcodeccontext that binds C to the St. The second is the initialization process of C, the ouput_example.c do is some basic configuration, of course, as a user you also want to add some other codec to codec conditions. You can refer to the avcodec.h about the structure of Avcodeccontext, the comments are more detailed.
The use of Avstream in the previous introduction of Avformatcontext has been involved in the main function in the three codec functions (8), (10) and (11). Observe the relevant code, you can find the main or the Avstream in the Avcodeccontext extract, and then extract the AVCODEC structure as in (8):
Open_video (OC, Video_st);
Avformatcontext *oc, Avstream *st
Avcodec *codec;
Avcodeccontext *c;
c = st->codec;
codec = Avcodec_find_encoder (c->codec_id); (31)
Open the Codec
Avcodec_open (c, codec); (32)
Similarly, we can see that in (Write_video_frame)) Avframe is also used as a carrier for the transfer of AVCODECCONTEXT structures. (one) (Close_video ()) relatively simple, not to boil.
Iv. Avcodeccontext
The comment in this structure in the FFmpeg SDK is that the main external API structure its importance. And in Avcodec its definition, it gives a very detailed introduction to each of its member variables. It should be said that the initialization of Avcodeccontext is the most important link in codec use. Although it has been mentioned in the previous Avstream, it is to be said again. Avcodeccontext as a member structure of the Avstream, it must be initialized (30) after initialization of the Avstream (Avstream initialization is used for Avformatcontex). Although the member variable is more, but here only to say in output_example.c in the use of, others please consult the avcodec.h file described.
Static Avstream *add_video_stream (avformatcontext *oc, int codec_id)
Avcodeccontext *c;
st = Av_new_stream (OC, 0);
c = st->codec;
c->codec_id = codec_id;
C->codec_type = Codec_type_video;
C->bit_rate = 400000; kbits/s
C->width = 352;
C->height = 288; Cif
The frame rate does the denominator, the second molecule, then the time_base is the time frame. (Time base.) ）
C->time_base.den = stream_frame_rate;
C->time_base.num = 1;
C->gop_size = 12;
Here define:
#define STREAM_PIX_FMT pix_fmt_yuv420p
Pixel format, see pix_fmt_xxx
-encoding:set by user.
-decoding:set by LAVC.
C->PIX_FMT = stream_pix_fmt;
In addition to the listed above. There are also: Me_method, such as the specified motion estimation algorithm. Quantization parameter, Max B frame number: Max_b_frames. Code rate control parameters, error masking error_concealment, pattern judgment mode: mb_decision (this parameter is quite interesting, you can see avcodec.h 1566 lines), Lagrange multipler parameters: Lmin & LMAX and macro block-level Lagrange Multipler parameters: Mb_lmin & Mb_lmax, constant quantization parameter rate control METHOD:CQP, etc.
It is worth mentioning that there are two member data structures in Avcodeccontext: Avcodec, Avframe. Avcodec records the codec information to be used and contains 5 functions: init, encoder, close, decode, flush to complete the codec work (see Avcode.h 2072 Row). The avframe contains the encoded frame information, including whether the frame is a key frame, *data[4] defined Y, CB and CR information, etc., followed by detailed description.
After initialization, it can be said that Avcodeccontext in (8) & (10). First initialize the Avcodec *codec and the avframe* picture in (8) Open_video ():
Avcodeccontext *c;
codec = Avcodec_find_encoder (c->codec_id);
......
Picture = Alloc_picture (pix_fmt_yuv420p, C->width, c->height);
The main parameters in Writer_video_frame (Avformatcontext *oc, Avstream *st) are exploited as a codec:
Avcodeccontext *c;
c = st->codec;
......
Out_size = Avcodec_encode_video (c, Video_outbuf, video_outbuf_size, picture);
V. Avcodec
member variables and member functions in struct AVCODEC are relatively small, but important. He contains the codecid, that is, with which codec,
Pixel format information. There are 5 previously mentioned functions (init, encode, close, decoder, flush). By the way, although the encoding function in reference code OUTPUT_EXAMPLE.C is Avcodec_encode_video (), I suspect that the encode function in which Avcodec is called is the same as the arguments and the return values that they pass. Of course not yet confirmed, interested to see FFmpeg source code. In the reference code, the use of Avcodec after initialization is dependent on Avcodeccontex, which is the member of the latter. After the Avcodeccontext is initialized (Add_video_stream ()), the Avcodec can be very well initialized:
Initialization
codec = Avcodec_find_encoder (c->codec_id); (33)
Open codec
Avcodec_open (c, codec) (34)
VI. Avframe
Avframe is a very interesting structure, which is defined by itself:
typedef struct AVFRAME {
Ff_common_frame
}avframe;
Where Ff_common_frame appears as a macro. Because the data in the Avframe is often accessed during the decoding process. In order to speed up, you need to take such a code tool.
Avframe is described as a "raw image" (i.e. YUV or RGB ...). Is there anything else? The structure of his first two member data, uint8_t *data[4],int Linesize[4], the first store is Y, Cb, Cr (YUV format), Linesize is what. It is also possible to extract a different data structure from these two figures:
typedef struct AVPICTURE {
uint8_t *data[4];
int linesize[4]; Number of bytes per line
}avpicture;
In addition, Avframe also contains some other member data, such as. Whether key_frame, encoded image book Coded_picture_number, whether as reference frame reference, macro block type *mb_type, and so on (avcodec.h 446 rows).
The initialization of the avframe is not as simple as it looks on his structure. Because Avframe also has a task that hosts image data (Data[4]) Therefore, allocating memory to him should be done with care. Alloc_picute () is provided in output_example.c to complete the work. Two global variables are defined in the reference code: Avframe *picture,*tmp_picture. (If you use the yuv420 format then only the previous data picture will be used, and the image information will be placed in the pictures.) In other formats, first initialize the yuv420 format and put it into tmp_picture in the go to requirement format and put it in the picture. Initializes the Avframe after Open_video () opens the codec:
Picture = Alloc_picture (C->pix_fmt, C->width, c->height);
Tmp_picture = Alloc_picture (pix_fmt_yuv420p, C->width, c->height);
static Avframe *alloc_picture (int pix_fmt, int width, int height) {
Avframe *picture;
uint8_t *picture_buf; Think what use uint8_t? A byte!
Picture = Avcodec_alloc_frame (); (35)
if (!picture)
return NULL;
Size = Avpicture_get_size (pix_fmt, width, height); (36)
Picture_buf = av_malloc (size); (37)
if (!PICTURE_BUF) {
Av_free (picture); (38)
return NULL;
}
Avpicture_fill ((avpicture *) picture, picture_buf, pix_fmt, width, height); (39)
return picture;
}
As can be seen from the above code, the completion of a avframe initialization (in fact, memory allocation), basically there is such a fixed mode. As for (35) (39) The work is done separately, and why there are two steps, it is not clear, need to look at the original code. My guess is (35) The basic memory allocation is made to Avframe, and the memory allocation to (39) of the first two data that can be extracted from avpicture is preserved.
Here, we observe that there is a (avpicture *) picture,avpicture structure in (39) that is also useful. Basically his size is to be transmitted on the network packet size, we can see in the back Avpacket and Avpicture have a close relationship.
VII. Avpicture
Avpicture does not have its own declaration and initialization process in the reference code. The two occurrences are extracted as coercion type conversions by Avframe:
Open_video () in
Avpicture_fill ((avpicture *) picture, picture_buf, pix_fmt, width, height); (40)
In Write_video_frame
Avpacket PKT;
if (Oc->oformat->flags & avfmt_rawpicture) {
......
pkt.size = sizeof (avpicture); (41)
}
In (40), the memory is actually allocated to Avframe's data[4], linesize[4]. Because of how these two data sizes are allocated, it is really necessary to have pix_fmt, width, and height to determine. If the output file format is raw pictures (such as YUV and RGB), Avpacket as the basic data unit for writing encoded data to a file, his cell size and data are avpacket.
Summed up is that the existence of avpicture for the following reasons, avpicture the concept of picture from the frame to extract, only by the pictures (image) itself information, brightness, chroma and row size. And frame is like whether it's a key frame or something. Such "grading" is the whole concept clearer.
VIII. Avpacket
The existence of a avpacket is a basic unit of writing to a file. We may think that it is not possible to write the encoded bitstream directly to the file, why bother setting up a avpacket structure. In my opinion this kind of coding is very necessary, especially in the video real-time transmission, synchronization, boundary problems can be solved by avpacket. The member data of the Avpacket has two timestamps, data (usually encoded data), size sizes, and so on (see Avformat.h 48 lines). The use of Avpacket has to mention the codec function, because the avpacket of a lot of information only after decoding can be known. In the reference code (OUPUT_EXAMPLE.C from 362 to 394 lines), make a judgment branch. If the output file format is a RAW image (that is, YUV or RGB) Then there is no coding function, write directly to the file (because the program itself generates a YUV file), where the code does not seem to be valuable here, But if the decoding function is to solve the YUV file (or RGB) then the basic write file operation is this:
if (Oc->oformat->flags & avfmt_rawpicture) {
Avpacket PKT; There is no pointer in this case.
Av_init_packet (&PKT);
Pkt.flags |= pkt_flag_key//Raw picture, each frame is a KEY frame?
Pkt.stream_index = st->index;
Pkt.data = (uint8_t *) picture;
pkt.size = sizeof (avpicture);
ret = Av_write_frame (OC, &PKT);
}
Output non-raw picture, after encoding:
else{
VIDEO_OUTBUF & Video_outbuf_size Initialization in Open_video ()
Out_size = Avcodec_encode_video (c, Video_outbuf, video_outbuf_size, picture); (42)
if (Out_size > 0) {
Avpacket PKT;
Av_init_packet (&PKT); (43)
Pkt.pts= av_rescale_q (c->coded_frame->pts, C->time_base, st->time_base); (44)
if (c->coded_frame->key_frame)
Pkt.flags |= Pkt_flag_key;
pkt.stream_index= st->index;
Pkt.data= Video_outbuf;
Pkt.size= out_size;
/* Write the compressed frame in the media file */
ret = Av_write_frame (OC, &PKT); (45)
} else {
ret = 0;
}
if (ret! = 0) {
fprintf (stderr, "Error while writing video frame\n");
Exit (1);
}
Where Video_outbuf and video_outbuf_size are initialized in Open_video ():
Video_outbuf = NULL;
The output is not raw picture, but it does use the encoding codec
if (! ( Oc->oformat->flags & Avfmt_rawpicture)) {
Video_outbuf_size = 200000;
Video_outbuf = Av_malloc (video_outbuf_size);
}
(43) is the initialization function of the avpacket structure. (44) It is difficult to understand, and why there are such a few time stamps I do not understand. Other Avpacket member data assignment is easy to understand, it is important to note that the initialization of video_outbuf and Video_outbuf_size, because in the reference code initialization and use is not in the same function, so it is relatively easy to ignore. (45) is a write file function, avformatcontext* OC contains the file name and other information, the return value RET because it is a total number of written data information, if return 0 indicates that the write failed. (42) and (45) as the more important SDK functions, will also be described later:
IX. Conclusion
The more important data structures in FFmpeg are analyzed. The following generation relationship has a rationale: (-)
Avformatcontext->avstream->avcodeccontext->avcodec
|
Avoutputformat or Avinputformat
Avframe->avpicture....>avpacket
Two Functions in the FFMpeg:
In the previous section of the analysis we have seen that the FFmpeg SDK provides a number of initialization functions and coding functions. All we have to do is initialize the key data structures correctly, and use the appropriate codec functions and read/write (I/O) operation functions correctly. As a monolithic code sdk,ffmpeg has some of his own standardized use procedures. such as function Av_register_all (); is a "registration function" that was called at the very beginning, he initialized the Libavcodec, "registered" all of the codec and video file formats (format). Below, I follow the context of the reference Code (OUPUT_EXAMPLE.C) to introduce the related functions.
/******************************************************************
Main ()
******************************************************************/
1. Av_register_all ();
Usage:initialize ibavcoded, and register all codecs and formats
Every project that uses the FFmpeg SDK must call a function. Codec and format are registered before they can be used. Statements in ALLFORMATS.C, are all macros are interested to see.
2. Avoutputformat Guess_format (const char *short_name, const char *filename, const char *mime_type)
Usage: By file suffix name, guess the file format, in fact, it is to determine what encoder (or decoder).
Avoutputformat *fmt;
FMT = Guess_format (null, filename, null);
3. Avformatcontext *av_alloc_format_context (void)
Usage:allocate the output media context. is actually initializing the Avformatcontext member data avclass:
Avformatcontext *ic;
Ic->av_class = &av_format_context_class;
where
Format_to_name, Options is pointer to function
static const Avclass Av_format_context_class = {"Avformatcontext", Format_to_name, Options};
4. Static Avstream *add_video_stream (avformatcontext *ox, int codec_id);
Avstream *video_st;
Video_st = Add_video_stream (OC, FMT->VIDEO_CODEC);
5. int av_set_parameters (Avformatcontext *s, Avformatparameters *ap)
Usage:set the output parameters (must is done even if no parameters).
Avformatcontext *oc;
If failed, return integer smaller than zero
Av_set_parameters (OC, NULL);
6. Void Dump_format (avformatcontext *ic, int index, const char *url, int is_output);
Usage: This step fills the Avformatcontext watershed (streams field) with useful information. As a debug diagnostic, we will output this information to the standard error output in its entirety, but you do not use it in the product of an application:
Dump_format (OC, 0, filename, 1); That is, to indicate Avformatcontext Avoutputformat, or//Avinputformat
7. static void Open_video (Avformatcontext *oc, Avstream *st)
Open_video (OC, Video_st);
8. int Av_write_header (Avformatcontext *s)
Usage:allocate the stream private data and writer the stream header to an output media file. param s Media file
Handle, return 0 if OK, averror_xxx if error.
Write the stream header, if any
Av_write_header (OC);
9. static void Write_video_frame (Avformatcontext *oc, Avstream *st)
Write_video_frame (OC, Video_st);
static void Close_video (Avformatcontext *oc, Avstream *st)
Close each codec
Close_video (OC, Video_st);
One. int Av_write_trailer (Avformatcontext *s)
Usage:write the trailer, if any. Write the stream trailer to a output media file and free the file private data.
Av_write_trailer (OC);
void Av_freep (void *arg)
Usage:free the streams. Frees memory and sets the pointer to NULL. Arg pointer to the pointer which should is freed.
Av_freep (&OC->STREAMS->CODEC);
Av_freeep (&oc->streams[s]);
int Url_fclose (Byteiocontext *s);
Usage:close the output file
Url_fclose (&OC->PB);
void Av_free (void *ptr)
Usage:free the stream. Free memory which have been allocated with Av_malloc (z) () or Av_realloc ().
Av_free (OC);
/******************************************************************
******************************************************************
Add_video_stream ()
Avcodeccontext *c
Avstream *st
******************************************************************/
****************************************************************