Analysis of audio and video synchronization principle

Analysis of audio and video synchronization principle 2013-04-18 15:21:11 Tags: Audio video
what exactly is dts/pts in the video stream? DTS (decoding timestamp) and PTS (display timestamp) are timestamps of the decoder relative to the SCR (System reference) when decoding and displaying the frames, respectively. SCR can be understood as the time when the decoder should start reading data from disk. Each package in an MPEG file has an SCR timestamp and this timestamp is the system time at which the packet was read. Typically, the decoder starts the system clock when it begins to read the MPEG stream (the initial value of the system clock is the SCR value of the first packet, usually 0 but can also start at 0). The dts  timestamp determines that the decoder decodes when the SCR time equals DTS time, and the PTS timestamp is similar. Typically, the dts/pts timestamp indicates a time   between the SCR in the audio and video package. For example, if the SCR of a video packet is 100ms (meaning that the package is read from disk after playing 100ms), then the dts/pts value is almost 200/280ms, Indicates that the video data should be decoded and displayed after 80ms (the video data is saved in a buffer to start decoding) when the SCR is 200ms, and the overflow usually occurs in the set video data stream associated with a high mux rate. If the MUX rate is 1000000bits/sec (meaning that the decoder is going to read the file at 1000000bits/sec rate  ), the video rate is 2000000bits/sec (which means you need to 2000000bits/ SEC rate shows video data), when reading video data from disk   is not fast enough to be able to read enough video data in 1 seconds. In this case, the dts/pts timestamp instructs the video to decode or display before it is read from the hard disk (the dts/pts timestamp will be earlier than the SCR time in the packet containing them). It is no longer a problem to rely on decoders today (although MPEG files are not fully compliant with MPEG because they should not overflow). Some decoders (many well-known PC-based players) read the file as quickly as possible to display the video, and can ignore the SCR directly. Note In the list you provide, the average video stream rate is ~3mbps (3000000bits/sec) but its peak reaches 14Mbps (quite large, DVD is limited to  9.8mbps). This means that the MUX rate needs to be adjusted sufficiently large to handle the 14Mbps portion, and the  bbmpeg calculated mux rate is sometimes too low to cause underflow. Do you plan to make the video stream rate so high? This is more than a description of the DVD, and it's probably not playable in most standalone playback. If you are not so planned, I will increase the value of mquant from 1 andand set the maximum bitstream to 9Mbps in the video settings to keep a smaller stream. If you really want to make the video bitrate so high, you need to increase the MUX rate. From the list provided, you can conclude that Bbmpeg uses 14706800bits/sec or 1838350bytes/sec's mux rate (the total data rate is: 1838350bytes/sec (14706800bits/sec) line). The value you set in the Force Mux rate field should be in  bytes/sec and divisible by 50. So I'll start with 36767 (1838350/50) and increase it until there's no more underflow errors. audio and video synchronization principle [FFmpeg]FFmpeg the approximate process for decoding video files: 1. Register all container formats and Codec:av_register_all () 2. Open file: Av_open_input_file () 3. Extract stream information from file: Av_find_stream_info () 4. Exhaustive all the streams, looking for the kind of codec_type_video 5. Find the corresponding decoder: Avcodec_find_decoder () 6. Open codec: Avcodec_open () 7. Allocate memory for decoded frames: Avcodec_alloc_frame () 8. Constantly extracting the frame data from the stream: Av_read_frame () 9. Determine the type of frame, for video frame invocation: Avcodec_decode_video () 10. After decoding, release the decoder: Avcodec_close () 11. Close the input file: Av_close_input_file () output_example.c in AV Sync code as follows (some of my Code is modified), this implementation is quite simple, but it is very illustrative of the problem. Audio and video synchronization-timestamp When media content is playing, the most annoying thing is that the audio and video are out of sync. Technically speaking, the best solution to the audio and video synchronization problem is timestamp: First select a reference clock (the time on the reference clock is linearly incremented), and the time to generate the data stream is to timestamp each block of data based on the time on the reference clock (typically including the start and end times); Reads the timestamp on the data block while referencing the time on the current reference clock to schedule playback (if the block's start time is greater than the time on the current reference clock, do not rush to play the block until the reference clock reaches the start time of the data block, or if the start time of the block is less than the time on the current reference clock, "ASAP" Play this piece of data or simply "discard" the data to make the playback progress catch up with the reference clock. Visible, avoid the audio and video synchronization phenomenon has two key-one is to generate the data stream with the correct timestamp. If the timestamp on the data block itself is problematic, then how to adjust the playback is not helpful. If the video stream content is starting from 0s, assuming 10s when someone began to speak, asked to match the audio stream, then the audio stream start time should be 10s, if the time stamp from 0s or other time to start playing, then this mixed audio and video stream in time synchronization itself on the problem. When a timestamp is played, the video stream and the audio stream are both reference clock times, and no reference relationship occurs between the data streams, that is, the video stream and the audio stream are synchronized through a neutral third party (that is, the reference clock). The second key point is the control of the data flow based on the timestamp on the playback, that is, the data block early or late to take a different approach. In Figure 2.8, the reference clock time plays the video stream content in the 0-10s, even if the audio stream data block is received, it cannot be played immediately, but must wait until the reference clock reaches 10s before the time, otherwise it will cause the audio and video out of sync problem. During a timestamp-based playback process, it is sometimes not enough to wait or quickly process only the data blocks that are early or late. If you want to be more proactive and effectively adjust playback performance, you need to introduce a feedback mechanism, which is to feed the current data stream too fast or too slowly to the source, allowing the source to slow down or speed up the flow of data. Readers familiar with DirectShow must know that the quality control in DirectShow is such a feedback mechanism (quality). The DirectShow solution for audio and video synchronization is quite impressive. However, the WMF SDK is only responsible for reading and decoding the ASF stream while playing, and is not responsible for the final presentation of the audio and video content, so it lacks such a feedback mechanism. Audio and video synchronization Communication SDK source package sharing: android:http://down.51cto.com/data/711001 windows:http://down.51cto.com/data/715497 linux:http://download.csdn.net/detail/weixiaowenrou/5169796 ios:http:// down.51cto.com/data/715486 web:http://down.51cto.com/data/710983
http://6352513.blog.51cto.com/6342513/1180742