"VS Development" "video development" uses FFMPEG+OPENCV to achieve PIP

Requirements: The two-way video synthesis All the way, that is, a picture at the same time show two video, which narrows down into a small video stacked on top of the big video, and the picture of the television effect similar.

Idea: Using H264 encoded video example, the file stored in the ES stream is H264, decoded into YUV,YUV can be converted into RGB format. Copy the RGB of the small video to the location where the big video needs to be overwritten. Convert the re-synthesized RGB to YUV and re-encode the new video with FFmpeg or x264.

Method: Encode or decode or use FFmpeg. FFmpeg decoding two video, after decoding are YUV. Using FFmpeg The Sws_getcontext function changes the size of the small graph. Then use OPENCV to complete the synthesis of two pictures (OpenCV this tall library I used to be so Really ashamed. In fact, the merged RGB here can write their own algorithm implementation, the essence is to copy the small image of the RGB to the corresponding position of the large map. After the synthesis of the RBG into YUV format, the use of x264 re-encoded into H264, saw the big video in the upper left corner there is a small video.

Code ideas:

Two threads, each decoding, the main video of the thread decoding a frame after the notification of the second video of the thread to decode and convert the image size, the sub-video of the thread decoding is completed to notify the main thread to synthesize the video and encode. When synthesizing video, it is very simple to use OPENCV to convert YUV into RGB, then set the sensitive area on the main video and overlay the small video on the line.

The main sub-thread decoding routines, like the basic use of ffmpeg routines. The same is true of the RGB routines that turn YUV into the OpenCV mat type, and the code for the on-video decoding thread is described.

Global variables:

Cv::mat littlergb,bigrgb;//size Video RGB
Cv::mat littleframe,bigframe;//size Video of YUV

The code inside the sub-video decoding thread:

while (Av_read_frame (Pinputformatcontext, &inpack) >=0)
{
Len = Avcodec_decode_video2 (Pinputcodeccontext, &outframe, &ncomplete, &inpack);//decode video
if (ncomplete>0)
{
if (GetMessage (&msg, NULL, 0, 0))
{
Switch (msg.message)
{
Case My_msg_decode:
Sws_scale (M_pswscontext,outframe.data,outframe.linesize, 0,outframe.height,dst->data,dst->linesize);// Convert picture size

memcpy (Littleframe.data,dst->data[0], 640*480); The following is the storage of ffmpeg YUV data into the OpenCV mat type. YUV data stored by OPENCV
memcpy (littleframe.data+640*480,dst->data[1], 640*480/4);
memcpy (littleframe.data+640*480*5/4,dst->data[2], 640*480/4);
SetEvent (hencodeevent);

Break
}
}

}
Av_free_packet (&inpack);
}

To merge pictures directly with OpenCV, the code is as follows:

Cv::cvtcolor (Littleframe, littlergb,cv_yuv2bgr_i420);
Cv::cvtcolor (Bigframe, bigrgb,cv_yuv2bgr_i420); Above YUV turn RGB
Mat ROI (Bigrgb,rect (0,0,640,480));//set sensitive zones on large graphs
Littlergb.copyto (ROI); Copy the small picture in the past
Mat Outframe;
Cv::cvtcolor (Bigrgb, outframe,cv_bgr2yuv_i420); RGB to YUV

This gives you the YUV of the merged picture.

then encode them. The encoded video is a PIP.