(Conversion) Audio/Video Synchronization-Timestamp

When media content is being played, the biggest headache is that audio and video are not synchronized. Technically, the best solution to the problem of audio/video synchronization is the timestamp: first, select a reference clock (the time on the reference clock must be linearly incrementing ); each data block is timestamp (generally including the start time and End Time) based on the time on the reference clock when data streams are generated. The timestamp on the data block is read during playback, at the same time, refer to the time on the current reference clock to schedule playback. (If the start time of the data block is greater than the time on the current reference clock, do not rush to play the data block, until the reference clock reaches the start time of the data block. If the start time of the data block is less than the time on the current reference clock, play the data block as soon as possible or simply discard the data block ", to make the playback progress catch up with the reference clock ).

Figure 2.8 time stamp solution for Audio/Video Synchronization

It can be seen that there are two key points to avoid audio/video non-synchronization: one is to add the correct timestamp when generating the data stream. If there is a problem with the timestamp played on the data block itself, it is no matter how it is adjusted during playback. 2.8. The video stream content starts from 0 s. Assume that someone starts talking at 10 s and requires audio streams to be matched, the starting time of the audio stream should be 10 s, if the timestamp starts from 0 s or other times, the mixed audio/video stream has a problem in time synchronization. When the time stamp is set, both the video stream and the audio stream refer to the time of the reference clock, and there is no reference relationship between the data streams. That is to say, video Streams and audio streams are synchronized through a neutral third party (that is, the reference clock. The second key point is to control the data stream based on the timestamp during playback, that is, to take different processing methods for data blocks from early to late. In Figure 2.8, when the video stream content is played within 0-10 s, even if the audio stream data block is received, it cannot be played immediately, however, you must wait until the reference clock time reaches 10 s. Otherwise, audio and video will not be synchronized.

In the timestamp-based playback process, it is sometimes not enough to wait for or quickly process data blocks that arrive early or arrive late. If you want to adjust the playback performance more actively and effectively, you need to introduce a feedback mechanism, that is, to feedback the current stream speed to the "Source", which is too fast or too slow ", let the source to slow down or speed up the data flow. Readers familiar with DirectShow know that quality control in DirectShow is such a feedback mechanism. DirectShow provides excellent solutions for audio/video synchronization. However, the wmf sdk is only responsible for reading and decoding the ASF data stream during playback, but not for the final presentation of audio and video content. Therefore, it lacks such a feedback mechanism.

In order to better understand the time stamp-Based Audio/Video Synchronization solution, the following is an example in our daily life. Suppose you have an appointment with a friend of yours to meet at Shanghai square at today, eat together, and play games again. In fact, this is a time point for you and your friends to stay in sync. As a result, you will arrive at Shanghai Plaza at, so you must wait for your friends. After 10 minutes, your friend hasn't arrived yet. At this moment, he called to say that something had been delayed and it would be a little late. You can't help it, because you have already booked a location in the restaurant next to you. If you don't rush in, the reservation will be canceled. So you can tell your friends to meet you directly at the restaurant, ask him to speed up. So at a certain time point in the restaurant, it will become another synchronization point between you and your friends. Although the specific time is not certain (depending on the speed of your friend's arrival), you and your friend will certainly be able to meet in the restaurant. What are the results? Your friend finally arrived at, and you finally "synchronized. After dinner at, you have to deal with the problem temporarily, so you have an appointment with your friend to meet at a nearby game hall. You are not synchronized, but you will synchronize again at a certain time point in the game room.

Why? In fact, synchronization is a dynamic process, a process of waiting and catching up with others. Synchronization is only temporary, but not synchronous. People always oscillate on the synchronized horizontal line, but do not deviate too far from this baseline.

From: http://blog.csdn.net/happydeer/article/details/206765

(Conversion) Audio/Video Synchronization-Timestamp