The principle of synchronization of audio and video

1. The principle of audio and video synchronization

2. A more positive method for audio and video synchronization

3. Audio and video synchronization problems encountered

Disclaimer: All content Below is reproduced

1. Original from: http://bbs.chinavideo.org/viewthread.php?tid=1183&extra=page%3D1%26amp%3Bfilter%3Ddigest&page=1

The following sections are posted:

Audio and video synchronization this piece, I have been not very understanding, very want to try, the following is in the QQ group to your predecessors to consult the session record, because to me such a beginner is very helpful, so paste this to make backup, again thank enthusiastic answer the question person, especially (not good people. ) ^_^;

2006-10-25 10:21:06 Dophin
Someone has a real-time encoding when the audio and video recording, how to synchronize the audio and video data. There are related sites can also ha, I really could not find the relevant information ...
2006-10-25 10:25:42 Dophin
Or specific point of the line, such as how the audio and video is synchronized, there are books introduced can ah, first thanks ha ....
(Everyone) 2006-10-25 10:21:45
By Time stamp
(Everyone) 2006-10-25 10:21:49
I'm also studying the problem.
(New left) 2006-10-25 10:21:56
Based on audio time
(Everyone) 2006-10-25 10:22:01
Right
(not good people.) ) 2006-10-25 10:22:08
The New left is right.
(not good people.) ) 2006-10-25 10:22:14
The sound image is sent interleaved.
(New left) 2006-10-25 10:22:43
When video decoding, the corresponding video frame is found according to the current playing time
(not good people.) ) 2006-10-25 10:22:44
Can have an audio packet with n images. The image is displayed at a frame rate during the completion of this audio package.
(Dophin) 2006-10-25 10:28:39
Well, then what's the main thing about audio? After all, the audio playback is also related to the time, audio in the play is to just play on their own.
(not good people.) ) 2006-10-25 10:24:05
The sound card has the time synchronization processing mechanism.
(not good people.) ) 2006-10-25 10:24:32
My previous computer motherboard if the driver is not installed properly. The sound is very fast. The image of the result is also a flash on the past.
(New left) 2006-10-25 10:24:47
Sound normal decoding and playback AH
(not good people.) ) 2006-10-25 10:25:05
is played by Windows MediaPlayer. Only the driver is loaded. The sound card has a clock synchronization function when it is properly driven.
(New left) 2006-10-25 10:25:14
Just the decoding of the video needs to refer to the current sound time
(not good people.) ) 2006-10-25 10:25:16
So the sound and the image are staggered,
(New left) 2006-10-25 10:25:36
When the sound is played, it is not in the tube, it is a separate sound decoding
(Dophin) 2006-10-25 10:32:16
Oh, I'm a little bit of a model in the brain now, that is, audio just play on their own, the video according to their own time stamp with the previous system time and audio time comparison, and then decode play, so realize synchronization, so understand right.
(not good people.) ) 2006-10-25 10:28:00
What system are you using?
(Dophin) 2006-10-25 10:33:26
What is the meaning of interlaced sounds and images. These two do not open two threads. There is no interleaving problem ah, even if it exists, it is the operating system level.
(not good people.) ) 2006-10-25 10:28:35
Avvvvvvvvvvvvvvvavvvvvvvvvvvvvvv
(Dophin) 2006-10-25 10:33:42
I have to, Linux ...
(not good people.) ) 2006-10-25 10:28:52
When you put a, give it directly to the sound card.
(not good people.) ) 2006-10-25 10:29:30
The middle of the V is the difference between the front and back two sound packets, you even the average speed
(Dophin) 2006-10-25 10:35:35
After calculating the average speed, we can draw a few frames per sec and then play V, is that so.
(not good people.) ) 2006-10-25 10:30:54
Right
(not good people.) ) 2006-10-25 10:31:01
Your voice has a sampling rate.
(Dophin) 2006-10-25 10:36:02
Yes, yes,
(not good people.) ) 2006-10-25 10:31:23
It's time to figure out a sound, right?
(not good people.) ) 2006-10-25 10:31:36
Like 44.1k/s.
(Dophin) 2006-10-25 10:36:36
Right
(not good people.) ) 2006-10-25 10:31:56
So you read 22.05K data from the DSP is not 0.5?
(Dophin) 2006-10-25 10:37:05
Is
(not good people.) ) 2006-10-25 10:32:29
You get 15 frames of data in this 0.5-second clock. Then you are not 0.5/15=0.03333 seconds to refresh a picture.
(Dophin) 2006-10-25 10:37:49
That's right.
(not good people.) ) 2006-10-25 10:32:55
Then you read off a voice and a picture pack. Do it again. It's OK. But the prerequisites for your collection must be synchronous.
(not good people.) ) 2006-10-25 10:33:11
That's right. Are you using embedded?
(not good people.) ) 2006-10-25 10:33:25
This is based on sound.
(not good people.) ) 2006-10-25 10:34:04
Another is to set the clock, calculate your frame rate to set the time before and after the frame, there is an error in the middle of delay or jump. The sound is separately administered.
(not good people.) ) 2006-10-25 10:34:21
Look at FFmpeg's fplay.c, the source of the program is very detailed.
(Dophin) 2006-10-25 10:40:12
Mine's not embedded.
(Dophin) 2006-10-25 10:40:24
is the ordinary PC machine,
(not good people.) ) 2006-10-25 10:36:08
Oh. Look at Fplay.c, then. There is a very detailed synchronization of the sound and the image. It's a little simpler, though. But perfectly formed.
(Dophin) 2006-10-25 10:41:54
Oh, great, collect this piece, I basically understand, play I think should also be similar, I try to see, thank you very much ha ... ^_^
(not good people.) ) 2006-10-25 10:37:14
Don't thank me.
(not good people.) ) 2006-10-25 10:37:24
But the way I do it is according to the fplay.c inside.
(Dophin) 2006-10-25 10:42:23
OK, I'll see Fplay in a minute. C, previously did not look because it feels like his playing effect is inferior to mplayer,
(not good people.) ) 2006-10-25 10:38:06
This is a simplified one. Of course not MPlayer 2. The cycle process for mplayer playback is:
while (!mpctx->eof) {
Fill_audio_out_buffers ();//audio stream read, decode, play
Update_video (&blit_frame);//video stream reading, decoding, filtering processing
Sleep_until_update (&time_frame, &aq_sleep_time);//Calculate delay time and sleep wait
Mpctx->video_out->flip_page ();//playback of video
Adjust_sync_and_print_status (frame_time_remaining, time_frame);//synchronous Corrective Treatment According to PTS of the audio and video
}

The audio-video synchronization method is
1) audio Playback playsize = Mpctx->audio_out->play (Sh_audio->a_out_buffer, Playsize, playflags); , the time is calculated based on the data size and accumulated
Mpctx->delay + = playback_speed*playsize/(double) ao_data.bps;
2) before video decoding, use the cumulative delay time to cut off the time of this video mpctx->delay-= Frame_time;
3) Calculate sound delay time *time_frame = delay-mpctx->delay/playback_speed;
where float delay = Mpctx->audio_out->get_delay (), the amount of time it takes to get all the data from the current sound output buf to play out.
4) Playback video synchronization is completed, so the video playback is completely based on the final data output of the sound card to synchronize.
5) Calculate the current audio and video pts difference double av_delay = a_pts-audio_delay-v_pts; then calculate the correction value x = (Av_delay + timing_error * playback_speed) * 0.1F; The correction time is added to the delay accumulation mpctx->delay+=x;. 3.

These days to do file playback, video format is H264, audio is a PCM, using FFmpeg to read audio and video, and then use FFmpeg to decode the display, all of it is smooth, but the audio and video synchronization took me a lot of time, always do not know why it is a lot worse. The principle of audio and video synchronization is, of course, based on the PTS to control the video playback, that is, after the video decoding a frame, whether the display and display how long is through the frame of PTS and the audio is playing simultaneously the PTS comparison, if the audio of the PTS larger, then the video display to prepare the next frame of decoding display , otherwise wait.
The specific implementation of the problem encountered one: there is no way to get the audio frame is playing pts, because the audio playback using the DirectSound, and for DirectSound I can only get the current copy into the DirectSound cached frame of PTS, and cannot get the PTS being played , if you don't get the PTS for the frame you're playing, then synchronization is definitely not possible. Looking for information on the Internet seems to have not found useful, and finally thought that because the size of the audio frame is proportional to the time, then the data in the DirectSound cache can be calculated by the playback time, and then according to the currently copied audio frame pts, you can get the frame being played pts, Then use this to correctly synchronize the display of the video frame.
Question two: According to the above method after processing or there is no synchronization phenomenon, why so I am also baffled its solution, later discovered is the waiting mechanism has a problem, originally I was using sleep () to do the waiting, but actually sleep () the error is very large, online has said there is 15MS, Do audio and video synchronization is certainly not possible, after constant Google, find a code:
void mysleep (int interval)
{
Large_integer litmp;
Longlong Qpart1,qpart2;
Double Dfminus, Dffreq, Dftim;
QueryPerformanceFrequency (&LITMP);
Dffreq = (double) litmp. quadpart;//get the clock frequency of the counter
QueryPerformanceCounter (&LITMP);
QPart1 = litmp. quadpart;//getting the initial value

Do
{
QueryPerformanceCounter (&LITMP);
QPart2 = litmp. quadpart;//Get Abort value
Dfminus = (double) (QPART2-QPART1);
Dftim = dfminus/dffreq;//Gets the corresponding time value in seconds
}while (dftim<0.001 * interval);
}
can achieve a higher accuracy of the wait, from the effect of viewing, can also achieve audio and video synchronization.
This thought that the problem to this end, but the program run time how to find the machine so slow ah, looked at the CPU occupancy rate, reached 100%. Obviously it's not possible to use this as a wait.
So continue Google, online have said timesetevent something, I did not try. I feel a bit of trouble. Later think of a previous read to do with WaitForSingleObject to do a certain piece of code execution, and then try, a try immediately found the effect is obvious, CPU occupancy rate back to the single digit. The changed code is as follows:
void mysleep (int interval)
{
HANDLE evt;
EVT = CreateEvent (null, TRUE, FALSE, NULL);
WaitForSingleObject (evt, interval);
CloseHandle (EVT);
}

Transferred from: http://blog.csdn.net/allentangtg/article/details/6873681