FFmpeg format conversion process

FFmpeg, as an open-source project that supports many video and audio formats, is widely used. Here, we will discuss the reader's understanding about it, which is not a general error. I hope you will criticize and advise me a lot. This is mainly because official documents are scarce. After numerous explorations, I have some experience to share.

1. Overview

There are several important concepts in FFMPEG. After getting familiar with them, things will become much easier.

Avformatcontext indicates an open file or other media. In short, it indicates the data source. Videos and audios are stored in files in a certain format. ThisFormatIt only specifies how to distinguish between a video and an audio stream.Decoding. Normally, the AVI format, that is, the above-mentioned format, allows you to encode videos and audios at will.

The avformat library in FFMPEG can help with the process of "splitting audio and video streams", while avcodec can help to decode videos.

2. Video decoding (1) open a file [CPP]
View plaincopyprint?

1. Avinputformat * inputfmt;
2. Avformatcontext * fmtctx = NULL;
4. Inputfmt = av_find_input_format ("Avi");/* Open the input format of "Avi */
5. If(Inputfmt = NULL ){
6. /* Error Processing */
7. }
8. If(Av_open_input_file (& fmtctx, "/test. Avi", inputfmt, 0, null )! = 0 ){
9. /* Error Processing */
10. }

Avinputformat * inputfmt; avformatcontext * fmtctx = NULL; inputfmt = av_find_input_format ("Avi");/* Open the input format of "Avi" */If (inputfmt = NULL) {/* Error Processing */} If (av_open_input_file (& fmtctx, "/test. avi ", inputfmt, 0, null )! = 0) {/* Error Processing */}

For convenience, we assume that the input file is in AVI format. In many cases, we do not know the file format. FFMPEG provides a probe method, which will be mentioned below.

(2) Find the decoder (video)

[CPP]
View plaincopyprint?

1. IntI, found_stream_index;
2. Avcodeccontext * videodecodectx = NULL;
3. For(I = 0; I <fmtctx-> nb_streams; I ++ ){
4. If(Fmtctx-> streams [I]-> codec-> codec_type = avmedia_video ){
5. Videodecodectx = fmtctx-> streams [I]-> codec;
6. Found_stream_index = I;
7. Break;
8. }
9. }
10. If(Decodectx = NULL ){
11. /* Video Stream not found, error handling */
12. }

Int I, found_stream_index; avcodeccontext * videodecodectx = NULL; for (I = 0; I <fmtctx-> nb_streams; I ++) {If (fmtctx-> streams [I]-> codec-> codec_type = avmedia_video) {videodecodectx = fmtctx-> streams [I]-> codec; found_stream_index = I; break;} If (decodectx = NULL) {/* Video Stream not found, error handling */}

When an avformatcontext is successfully opened, all streams (video streams, audio streams, etc.) contained in the file are stored in its structure. Each stream (avstream) A decoder context is automatically opened, that is, the parameters provided to the decoder,It is not a real decoder, but a decoder parameter!
In the above Code, find the video stream in all the streams contained in the file and obtain a decoder context.

(3) Enable Decoder [CPP]
View plaincopyprint?

1. Avcodec * videodecoder;
2. Videodecoder = avcodec_find_encoder (videodecodectx-> codec_id );
3. If(Videodecoder = NULL ){
4. /* Decoder not found, error handling */
5. }
6. If(Videodecoder-> capabilities & codec_cap_truncated ){
7. Videodecodectx-> flags | = codec_flag_truncated;
8. }
10. If(Avcodec_open (videodecodectx, videodecoder) <0 ){
11. /* Cannot open the decoder and handle errors */
12. }

Avcodec * videodecoder; videodecoder = avcodec_find_encoder (videodecodectx-> codec_id); If (videodecoder = NULL) {/* the decoder cannot be found, error Handling */} If (videodecoder-> capabilities & codec_cap_truncated) {videodecodectx-> flags | = tags;} If (avcodec_open (videodecodectx, videodecoder) <0) {/* cannot open decoder, error handling */}

This step looks for a decoder and uses the previously obtained parameters to open it. Not all encodings are supported, not all parameters are supported by the decoder, so you must handle errors.
Codec_cap_truncated indicates that the decoder can support the so-called "fragment input". Ignore it and wait.
If all the operations are successful, the decoder is successfully enabled. Next we can start decoding.

(4) video decoding [CPP]
View plaincopyprint?

1. Avpacket Pkt;
2. Avframe * frame = avcodec_alloc_frame ();
3. IntGot_picture = 0, Rc = 0;
5. While(1 ){
6. Av_init_packet (& Pkt );
7. Rc = av_read_packet (fmtctx, & Pkt);/* get the data of a package */
8. If(RC! = 0)Break;
9. If(Pkt. stream_index! = Found_stream_index)
10. GotoFree_pkt_continue;/* It is not the data of the video stream of interest. Remove it */
11. If(Avcodec_decode_video2 (videodecodectx, frame, & got_picture, & Pkt) <0 ){
12. /* Core function: decoding. Handle errors. */
13. }
15. If(Got_picture ){
16. {/* Processed image (stored in avframe )*/}
17. Av_free (FRAME );
18. Frame = avcodec_alloc_frame ();
19. }
21. Free_pkt_continue:
22. Av_free_packet (& Pkt );
23. }

Avpacket Pkt; avframe * frame = avcodec_alloc_frame (); int got_picture = 0, Rc = 0; while (1) {av_init_packet (& Pkt); rc = av_read_packet (fmtctx, & Pkt);/* get the data of a package */If (RC! = 0) break; If (Pkt. stream_index! = Found_stream_index) goto free_pkt_continue;/* remove */If (avcodec_decode_video2 (videodecodectx, frame, & got_picture, & Pkt) from data that is not concerned with the video stream. <0) {/* core function: decoding. Handle errors. */} If (got_picture) {/* processed image (stored in avframe) */} av_free (FRAME); frame = avcodec_alloc_frame ();} free_pkt_continue: av_free_packet (& Pkt );}

In fact, this process is very simple:
1. Read data;
2. decoding.
The original data is stored in avpacket, and the resulting image is stored in avframe. A frame of image can be composed of many avpacket, so a got_picture pointer is used to indicate whether to obtain an image.
The question about cap_truncated is whether the decoder supports the avframe boundary and the avpacket boundary. Data can be scattered. This decoder is very important. It can process any segment of data (useful for network data ).

3. Image Conversion

The image conversion here is not similar to the conversion from PNG to JPG, but mainly the conversion of Color Space and size scaling. For example, the image format obtained by MPEG4 decoder is yuv420p. If QT is used to render the image, the RGB format and any size are required.
FFmpeg provides the swscale library to support image conversion. Now let's assume that the data we decoded in the previous step is stored in the avframe * frame. We have:

[CPP]
View plaincopyprint?

1. Swscontext * swsctx;
2. IntDst_width = 320,/* Target width */
3. Dst_height = 240,/* Target Height */
4. Dst_pix_fmt = pix_fmt_rgb24;/* target image format */
5. Avframe * convertedframe = avcodec_alloc_frame ();
6. Swsctx = sws_getcontext (
7. Videodecodectx-> width, videodecodectx-> height, videodecodectx-> pix_fmt,
8. Dst_width, dst_height, dst_pix_fmt, sws_fast_bilinear,
9. Null, null, null );
10. If(Swsctx = NULL ){
11. /* Conversion context initialization failed, error handling */
12. }
14. /* Allocate memory required for conversion */
15. Avpicture_fill (avpicture *) convertedframe, dst_pix_fmt, dst_width, dst_height );
16. If(Sws_scale (swsctx, frame-> data, frame-> linesize, 0,
17. Videodecodectx-> height, convertedframe-> data, convertedframe-> linesize) <= 0 ){
18. /* Conversion failed, error handling */
19. }

Swscontext * swsctx; int dst_width = 320,/* Target width */dst_height = 240,/* Target Height */dst_pix_fmt = pix_fmt_rgb24; /* target image format */avframe * convertedframe = convert (); swsctx = sws_getcontext (videodecodectx-> width, videodecodectx-> height, videodecodectx-> pix_fmt, dst_width, dst_height, expires, sws_fast_bilinear, null); If (swsctx = NULL) {/* failed to initialize the conversion context, error Handling */}/* allocate memory required for conversion */avpicture_fill (avpicture *) convertedframe, dst_pix_fmt, dst_width, dst_height); If (sws_scale (swsctx, frame-> data, frame-> linesize, 0, videodecodectx-> height, convertedframe-> data, convertedframe-> linesize) <= 0) {/* Conversion failed, error handling */}

So convertedframe-> data [0] contains the converted image!