FFmpeg av Sync _2

Original address: http://www.jianshu.com/p/27279255f67e

The specific flow of the audio and video player's work is shown in the following figure:
Player Workflow

In a nutshell, it includes: The solution protocol, the solution encapsulation, the audio and the video decoding respectively, the sound and video synchronization play these parts, each part detailed explanation please see the reference material later. Since we decode and play audio and video separately, the rhythm of each play needs to be synchronized, otherwise there will be inconsistencies in the tone and picture. This paper mainly introduces a simple audio and video synchronization scheme. preparatory work

implement FFmpeg audio playback for ffmpeg video playback Related Knowledge
PTS and DTS
Each frame in the audio and video stream has time-related information, where PTS is the playback time, and DTS is the decoding time. The audio pts and DTS are consistent, while some videos may have DTS and pts inconsistent frames, where we control the playback time primarily with PTS. Use Av_frame_get_best_effort_timestamp for decoded avframe to get pts. Time_base
We notice that PTS is a shaping data, Time_base is the unit of PTS, and pts times time_base to get the actual time. Only the time_base obtained in Avstream is right, and other places may have problems. Audio and video synchronization strategy
There are generally three ways to sync audio to video, sync video to audio, and sync video to external time. General parameters set the correct audio will be able to play at normal speed, so the video synchronization to audio in general, is a simple and effective synchronization strategy. This paper mainly adopts this method to synchronize the audio and video to show the relevant basic ideas. sync video to audio
Gets the decoded video frame time.

Avframe vframe;
avstream vstream;
//... Parse video get Vstream, decode video frame get Vframe
... Double timestamp = Av_frame_get_best_effort_timestamp (&vframe) *av_q2d (vstream->time_base);

Gets the decoded audio frame time.

Avframe aframe;
avstream astream;
//... Record the audio points that are being played down as benchmarks
... Audioclock = aframe.pkt_pts * av_q2d (astream->time_base);

Audio and video synchronization logic

/* No AV sync correction is done if below the minimum AV sync threshold */#define AV_SYNC_THRESHOLD_MIN 0.04/* AV sync C Orrection is done if above the maximum AV sync threshold */#define AV_SYNC_THRESHOLD_MAX 0.1/* If a frame duration is lo Nger than this, it'll is not being duplicated to compensate AV sync */#define AV_SYNC_FRAMEDUP_THRESHOLD 0.1 * No AV correct
Ion is done if too big error */#define AV_NOSYNC_THRESHOLD 10.0 double timestamp; Determine if there is a valid pts if (packet.pts = = av_nopts_value) {timestamp = 0;} else {timestamp = Av_frame_get_best_effort_timestamp (
&vframe) *av_q2d (vstream->time_base);
}//Calculate frame rate, average per frame interval double framerate = av_q2d (vstream->avg_frame_rate);
Framerate + = Vframe.repeat_pict * (framerate * 0.5); if (timestamp = = 0.0) {//Play usleep ((unsigned long) (framerate*1000) by default frame rate);} else {if (Fabs (timestamp-audioclock) > Av_sync_threshold_min && fabs (Timestamp-audioclock) < Av_nosync_threshold) {//If the video is faster than audio, the delay difference is played, otherwise direct playback, here is not doneDrop frame processing if (Timestamp > Audioclock) {usleep ((unsigned long) ((Timestamp-audioclock) *1000000)); }
 }
}

Engineering Code

Easyplayer References

AV codec Technology 0 Basic Learning methods Tutorial 05:synching Video