FFmpeg Encapsulation Format processing: video/audio multiplexing (muxer)

Last Update:2018-06-07 Source: Internet

Author: User

Tags file url

Developer on Alibaba Coud: Build your first app with APIs, SDKs, and tutorials on the Alibaba Cloud. Read more ＞

I plan to record examples of FFmpeg-based Encapsulation Format processing. Including separation, reuse, and Encapsulation Format Conversion of audio and video. This is Article 3rd. This document records an FFmpeg-based audio and video multiplexing (SimplestFFmpegmuxer ). The Muxer combines the compressed video data (for example, H.264) with the compressed audio data (for example, AAC ).

I plan to record examples of FFmpeg-based Encapsulation Format processing. Including separation, reuse, and Encapsulation Format Conversion of audio and video. This is Article 3rd. This document records an FFmpeg-based video/audio multiplexing (Simplest FFmpeg muxer ). The Muxer combines the compressed video data (for example, H.264) with the compressed audio data (for example, AAC ).

I plan to record examples of FFmpeg-based Encapsulation Format processing. Including separation, reuse, and Encapsulation Format Conversion of audio and video. This is Article 3rd.

This document records an FFmpeg-based video/audio multiplexing (Simplest FFmpeg muxer ). The Muxer combines the compressed video data (such as H.264) and compressed audio data (such as AAC) into an encapsulated format data (such as MKV .. Encoding and decoding are not involved in this process.

The program recorded in this article combines an H.264 encoded video code stream file and an MP3 encoded audio code stream file into an MP4 encapsulated file.
Three AVFormatContext are initialized, two of which are used for input and one for output. After the three AVFormatContext initialization, you can use the avcodec_copy_context () function to copy the input video/audio parameters to the AVCodecContext structure of the output video/audio. Then, av_read_frame () of the video input stream and the audio input stream are called respectively to retrieve the AVPacket of the video from the video input stream and the AVPacket of the audio input stream, write the retrieved AVPacket to the output file. In this case, av_compare_ts (), a very common function, is used to compare timestamps. You can use this function to determine whether to write the video or audio.

In this article, the input video is not necessarily a H.264 bare stream file, and the audio is not necessarily a pure audio file. You can select two encapsulated audio and video files as the input. The program will "pick" the video stream from the video input file, "pick" the audio stream in the audio input file, and then reuse the "Selected" audio stream. PS1: for H.264 in certain encapsulation formats (for example, MP4/FLV/MKV), bitstream filter named "h1__mp4to1_ B" must be used.
PS2: bitstream filter named "aac_adtstoasc" is required for AAC in certain encapsulation formats (such as MP4/FLV/MKV.

Briefly introduce the significance of each important function in the process:

Avformat_open_input (): Open the input file.
Avcodec_copy_context (): assign a value to the AVCodecContext parameter.
Avformat_alloc_output_context2 (): Initialize the output file.
Avio_open (): Open the output file.
Avformat_write_header (): Write the file header.
Av_compare_ts (): Compares the timestamp to determine whether to write the video or audio. This function is relatively rare.
Av_read_frame (): read an AVPacket from the input file.
Av_interleaved_write_frame (): Write an AVPacket to the output file.
Av_write_trailer (): The end of the written file.

Code

Paste the code below:[Cpp]View plaincopy

```
/**
```

* The simplest FFmpeg-based audio and video multiplexing

```
* Simplest FFmpeg Muxer
```
```
*
```
```
* Lei Xiaohua
```
```
* leixiaohua1020@126.com
```

* China Media University/Digital TV technology

* Communication University of China / Digital TV Technology

```
* http://blog.csdn.net/leixiaohua1020
```
```
*
```

* This program can package the video and audio streams into one Encapsulation Format.

* In the program, package the MP3-encoded audio streams and H.264-encoded video streams (in MPEG2TS encapsulation)

```
* MP4 encapsulated files.
```

* Note that this program does not change the audio/video encoding format.

```
*
```

* This software mux a video bitstream and a audio bitstream

```
* together into a file.
```

* In this example, it mux a H.264 bitstream (in MPEG2TS) and

* a MP3 bitstream file together into MP4 format file.

```
*
```
```
*/
```
```
#include 
   
```
```
extern "C"
```
```
{
```
```
#include "libavformat/avformat.h"
```
```
};
```
```
/*
```

FIX: H.264 in some container format (FLV, MP4, MKV etc.) need

"h264_mp4toannexb" bitstream filter (BSF)

```
*Add SPS,PPS in front of IDR frame
```

*Add start code ("0,0,0,1") in front of NALU

H.264 in some container (MPEG2TS) don't need this BSF.

```
*/
```
```
//'1': Use H.264 Bitstream Filter
```
```
#define USE_H264BSF 0
```
```
/*
```

FIX:AAC in some container format (FLV, MP4, MKV etc.) need

```
"aac_adtstoasc" bitstream filter (BSF)
```
```
*/
```
```
//'1': Use AAC Bitstream Filter
```
```
#define USE_AACBSF 0
```
```
int main(int argc, char* argv[])
```
```
{
```
```
AVOutputFormat *ofmt = NULL;
```

// The input corresponds to an AVFormatContext, and the output corresponds to an AVFormatContext.

//（Input AVFormatContext and Output AVFormatContext）

AVFormatContext *ifmt_ctx_v = NULL, *ifmt_ctx_a = NULL,*ofmt_ctx = NULL;

```
AVPacket pkt;
```
```
int ret, i;
```

Char * in_filename_v = "cuc_ieschool.ts"; // Input file URL)

//char *in_filename_v = "cuc_ieschool.h264";

//char *in_filename_a = "cuc_ieschool.mp3";

```
//char *in_filename_a = "gowest.m4a";
```
```
//char *in_filename_a = "gowest.aac";
```
```
char *in_filename_a = "huoyuanjia.mp3";
```

Char * out_filename = "cuc_ieschool.mp4"; // Output file URL)

```
av_register_all();
```
```
// Input)
```

if ((ret = avformat_open_input(&ifmt_ctx_v, in_filename_v, 0, 0)) < 0) {

```
printf( "Could not open input file.");
```
```
goto end;
```
```
}
```

if ((ret = avformat_find_stream_info(ifmt_ctx_v, 0)) < 0) {

printf( "Failed to retrieve input stream information");

```
goto end;
```
```
}
```

if ((ret = avformat_open_input(&ifmt_ctx_a, in_filename_a, 0, 0)) < 0) {

```
printf( "Could not open input file.");
```
```
goto end;
```
```
}
```

if ((ret = avformat_find_stream_info(ifmt_ctx_a, 0)) < 0) {

printf( "Failed to retrieve input stream information");

```
goto end;
```
```
}
```

printf("Input Information=====================\n");

av_dump_format(ifmt_ctx_v, 0, in_filename_v, 0);

av_dump_format(ifmt_ctx_a, 0, in_filename_a, 0);

printf("======================================\n");

```
// Output)
```

avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, out_filename);

```
if (!ofmt_ctx) {
```

printf( "Could not create output context\n");

```
ret = AVERROR_UNKNOWN;
```
```
goto end;
```
```
}
```
```
ofmt = ofmt_ctx->oformat;
```
```
int videoindex_v=-1,videoindex_out=-1;
```

for (i = 0; i < ifmt_ctx_v->nb_streams; i++) {

// Create output AVStream (according to input AVStream)

if(ifmt_ctx_v->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO){

```
videoindex_v=i;
```

AVStream *in_stream = ifmt_ctx_v->streams[i];

AVStream *out_stream = avformat_new_stream(ofmt_ctx, in_stream->codec->codec);

```
if (!out_stream) {
```

printf( "Failed allocating output stream\n");

```
ret = AVERROR_UNKNOWN;
```
```
goto end;
```
```
}
```
```
videoindex_out=out_stream->index;
```
```
// Copy the settings of AVCodecContext)
```

if (avcodec_copy_context(out_stream->codec, in_stream->codec) < 0) {

printf( "Failed to copy context from input to output stream codec context\n");

```
goto end;
```
```
}
```
```
out_stream->codec->codec_tag = 0;
```

if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)

out_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;

```
break;
```
```
}
```
```
}
```
```
int audioindex_a=-1,audioindex_out=-1;
```

for (i = 0; i < ifmt_ctx_a->nb_streams; i++) {

// Create output AVStream (according to input AVStream)

if(ifmt_ctx_a->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO){

```
audioindex_a=i;
```

AVStream *in_stream = ifmt_ctx_a->streams[i];

AVStream *out_stream = avformat_new_stream(ofmt_ctx, in_stream->codec->codec);

```
if (!out_stream) {
```

printf( "Failed allocating output stream\n");

```
ret = AVERROR_UNKNOWN;
```
```
goto end;
```
```
}
```
```
audioindex_out=out_stream->index;
```
```
// Copy the settings of AVCodecContext)
```

if (avcodec_copy_context(out_stream->codec, in_stream->codec) < 0) {

printf( "Failed to copy context from input to output stream codec context\n");

```
goto end;
```
```
}
```
```
out_stream->codec->codec_tag = 0;
```

if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)

out_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;

```
break;
```
```
}
```
```
}
```

// Output the following format ------------------

printf("Output Information====================\n");

av_dump_format(ofmt_ctx, 0, out_filename, 1);

printf("======================================\n");

```
// Open output file)
```
```
if (!(ofmt->flags & AVFMT_NOFILE)) {
```

if (avio_open(&ofmt_ctx->pb, out_filename, AVIO_FLAG_WRITE) < 0) {

printf( "Could not open output file '%s'", out_filename);

```
goto end;
```
```
}
```
```
}
```
```
// Write file header)
```

if (avformat_write_header(ofmt_ctx, NULL) < 0) {

printf( "Error occurred when opening output file\n");

```
goto end;
```
```
}
```
```
int frame_index=0;
```
```
int64_t cur_pts_v=0,cur_pts_a=0;
```
```
//FIX
```
```
#if USE_H264BSF
```

AVBitStreamFilterContext* h264bsfc = av_bitstream_filter_init("h264_mp4toannexb");

```
#endif
```
```
#if USE_AACBSF
```

AVBitStreamFilterContext* aacbsfc = av_bitstream_filter_init("aac_adtstoasc");

```
#endif
```
```
while (1) {
```
```
AVFormatContext *ifmt_ctx;
```
```
int stream_index=0;
```
```
AVStream *in_stream, *out_stream;
```
```
// Obtain an AVPacket (Get an AVPacket)
```

if(av_compare_ts(cur_pts_v,ifmt_ctx_v->streams[videoindex_v]->time_base,cur_pts_a,ifmt_ctx_a->streams[audioindex_a]->time_base) <= 0){

```
ifmt_ctx=ifmt_ctx_v;
```
```
stream_index=videoindex_out;
```
```
if(av_read_frame(ifmt_ctx, &pkt) >= 0){
```
```
do{
```
```
if(pkt.stream_index==videoindex_v){
```
```
cur_pts_v=pkt.pts;
```
```
break;
```
```
}
```

}while(av_read_frame(ifmt_ctx, &pkt) >= 0);

```
}else{
```
```
break;
```
```
}
```
```
}else{
```
```
ifmt_ctx=ifmt_ctx_a;
```
```
stream_index=audioindex_out;
```
```
if(av_read_frame(ifmt_ctx, &pkt) >= 0){
```
```
do{
```
```
if(pkt.stream_index==audioindex_a){
```
```
cur_pts_a=pkt.pts;
```
```
break;
```
```
}
```

}while(av_read_frame(ifmt_ctx, &pkt) >= 0);

```
}else{
```
```
break;
```
```
}
```
```
}
```

in_stream = ifmt_ctx->streams[pkt.stream_index];

out_stream = ofmt_ctx->streams[stream_index];

```
//FIX
```
```
#if USE_H264BSF
```

av_bitstream_filter_filter(h264bsfc, in_stream->codec, NULL, &pkt.data, &pkt.size, pkt.data, pkt.size, 0);

```
#endif
```
```
#if USE_AACBSF
```

av_bitstream_filter_filter(aacbsfc, in_stream->codec, NULL, &pkt.data, &pkt.size, pkt.data, pkt.size, 0);

```
#endif
```
```
//FIX：No PTS (Example: Raw H.264)
```
```
//Simple Write PTS
```
```
if(pkt.pts==AV_NOPTS_VALUE){
```
```
//Write PTS
```

AVRational time_base1=in_stream->time_base;

```
//Duration between 2 frames (us)
```

int64_t calc_duration=(double)AV_TIME_BASE/av_q2d(in_stream->r_frame_rate);

```
//Parameters
```

pkt.pts=(double)(frame_index*calc_duration)/(double)(av_q2d(time_base1)*AV_TIME_BASE);

```
pkt.dts=pkt.pts;
```

pkt.duration=(double)calc_duration/(double)(av_q2d(time_base1)*AV_TIME_BASE);

```
frame_index++;
```
```
}
```
```
/* copy packet */
```
```
// Convert PTS/DTS (Convert PTS/DTS)
```

pkt.pts = av_rescale_q_rnd(pkt.pts, in_stream->time_base, out_stream->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));

pkt.dts = av_rescale_q_rnd(pkt.dts, in_stream->time_base, out_stream->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));

pkt.duration = av_rescale_q(pkt.duration, in_stream->time_base, out_stream->time_base);

```
pkt.pos = -1;
```
```
pkt.stream_index=stream_index;
```

printf("Write 1 Packet. size:%5d\tpts:%8d\n",pkt.size,pkt.pts);

```
// Write)
```

if (av_interleaved_write_frame(ofmt_ctx, &pkt) < 0) {

```
printf( "Error muxing packet\n");
```
```
break;
```
```
}
```
```
av_free_packet(&pkt);
```
```
}
```
```
// Write file trailer)
```
```
av_write_trailer(ofmt_ctx);
```
```
#if USE_H264BSF
```
```
av_bitstream_filter_close(h264bsfc);
```
```
#endif
```
```
#if USE_AACBSF
```
```
av_bitstream_filter_close(aacbsfc);
```
```
#endif
```
```
end:
```
```
avformat_close_input(&ifmt_ctx_v);
```
```
avformat_close_input(&ifmt_ctx_a);
```
```
/* close output */
```

if (ofmt_ctx && !(ofmt->flags & AVFMT_NOFILE))

```
avio_close(ofmt_ctx->pb);
```
```
avformat_free_context(ofmt_ctx);
```
```
if (ret < 0 && ret != AVERROR_EOF) {
```
```
printf( "Error occurred.\n");
```
```
return -1;
```
```
}
```
```
return 0;
```

This article is an English version of an article which is originally in the Chinese language on aliyun.com and is provided for information purposes only. This website makes no representation or warranty of any kind, either expressed or implied, as to the accuracy, completeness ownership or reliability of the article or any translations thereof. If you have any concerns or complaints relating to the article, please send an email, providing a detailed description of the concern or complaint, to info-contact@alibabacloud.com. A staff member will contact you within 5 working days. Once verified, infringing content will be removed immediately.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

Get Started for Free

Sales Support

1 on 1 presale consultation

Chat Contact Sales
After-Sales Support

24/7 Technical Support 6 Free Tickets per Quarter Faster Response

Open a Ticket
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.

Learn More