Audio and video recoding timestamp synchronization problem

In scenes where audio and video are re-encoded and need to be synchronized, there are several basic principles to follow (otherwise the audio and video will be stuck and not smooth.) With audio AAC encoding frequency 44.1k, video H264 encode 25 frame rate for example):

1. Ensure that the audio and video frame arrival interval at the input is basically accurate. The audio AAC is 23.2ms (1000*1024/44100) per frame, and the video frame length is 40ms (1000/25). Therefore, the frequency of arrival of the original audio samples for encoding (or the frequency obtained from buffer) should be 441 samples/per channel/per 10ms (each sample assumes 16bits, or 882 bytes/channel/10ms, If the original audio sample rate is not 44.1k, resampling is required before encoding, the original video frame arrival frequency (or the frequency obtained from buffer) should be 1 frames/per 40ms (video frame rate may need to be resampled). If the output of the audio and video stream is not fluent, you can first check the input audio and video stream input interval situation.

2. Ensure that the audio and video stream timestamps on the output are using the same reference system. For example, audio and video streams use the current system time as the same time reference, but the audio and video streams can have different system time starting points. For example, the audio stream starts at 1492764087000 MS, and the video starts at 1492764087700ms later in 700ms. For example, rtmp in the use of 32-bit time stamp, the audio and video stream can only use relative time stamp, the next example, the time stamp growth to 700ms video streaming time from 0, indicating that the video stream from the audio stream of 700ms start, so as to achieve the effect of synchronization. Another time-stamping scheme is that each audio-video frame grows at a fixed interval, such as the audio-video timestamp starts at 0, each AAC frame is increased by 23.2ms, and each video frame grows by 40ms. Normally, audio and video streams are sent at the corresponding intervals from 0, but there is also a video stream later than the audio stream or audio stream later than the video stream. This situation requires a timestamp synchronization, and a later stream start time is set according to the time of the advance stream.

3. Ensure that the audio and video interval is basically accurate when crossing output. Here the output is exactly the same as a hardware encoder, only to ensure that the cross-output audio and video frame interval is stable, to ensure smooth playback. For example RTMP, each AAC audio frame output interval should be around 23ms, each video frame output interval should be around 40ms, and audio and video frames are cross-output. In other words, each 23MS sends an AAC audio frame, sending a video frame every 40ms (two separate threads can be used to send a stream of audio and video separately). If the above two problems are excluded or not smooth playback, you can check whether this link is normal.

In short, the re-coding and synchronization of this link must be based on mathematical measurement. Where there are errors or additions, please note.

Here is an implementation version of the adjustment based on the above rules in a real project:

Where the hypothetical conditions in the actual environment do not match:

1. The front-end input frame rate changes, x264 cannot stabilize the output fixed 24/25 frames per second.

Timestamp adjustment scheme: The cumulative adjustment of the standard audio and video fixed frame interval to the actual time offset between frames. such as audio, no longer at each frame interval of 23ms to accumulate, the reason is that the system time interval of 1s, the audio timestamp increases less than 1s, the video is no longer at each frame interval 40ms to accumulate, but also to the actual inter-frame interval, such as after each encoding after each frame interval is changed, the frame rate may be less than 24/25. Finally, the synchronization of the audio-video timestamp is realized. It is synchronous to increase the time stamp of the audio and video by the absolute time interval of the system, if using other artificial timestamp growth schemes (such as the above audio and video at fixed interval increment), it is necessary to introduce a synchronization module to deal with the increment migration between audio and video timestamps, which is difficult to control accurately. This method is actually the 2nd principle mentioned above: Audio and video Use the same time reference system.

Scenarios of several errors that have been tried in the project:

1. The above standard audio and video frame interval accumulates the calculation timestamp. In this case, there is a growing problem with the time stamp offset of the audio and video, it is possible that the video is larger than the audio or the audio is bigger than the video.

2. The video stream uses the original frame timestamp as a reference. This method is also undesirable, because the original frame timestamp is based on the original video stream frame rate, you x264 re-encoded, the frame rate has been different. The spacing between frames and frames is also of no referential significance.

Http://www.cnblogs.com/NerdWill/p/6744432.html