Live Technical Details 1

qq:564702640Basics: I-frame, B-frame, p-frame

I-frames represent keyframes. You can understand the full retention of this frame, and only the data in this frame can be used to complete the decoding. (because full screen is included)

The P-frame indicates the difference between this frame and a previous keyframe (or P-frame). The decoding needs to use the previous cached screen overlay on the definition of this frame to create the final picture. (That is, the difference frame, p frame does not have the complete picture data, only with the previous frame of the picture difference of data)

B-frames are bidirectional differential frames. B-Frame recording is the difference between this frame and the front and back frames (more complex, there are 4 cases). In other words, to decode the B-frame, not only to obtain the previous cache screen, but also to decode after the screen, through the front and back of the picture and the frame of the data overlay to obtain the final picture.

B-Frame compression rate is high, but the codec is CPU-intensive, and in the live stream may increase the live delay, so on the mobile side generally do not use B-frame.

Keyframe Caching Policy

A typical video frame sequence is IBBPBBPBBP ...

For live streaming, to reduce the latency of live streaming, B-frames are usually not used when encoding. P frame B frame has a direct or indirect dependency on the I frame, so the player to decode a video frame sequence, and play, must first decode the I frame, its subsequent B-frame and P-frame can be decoded, so that the server how to cache the key frame, then the broadcast delay and other aspects have a very big impact.

The better strategy is to automatically judge the interval of the key frame by the service, cache the frame sequence according to the business requirement, and ensure that at least two or more key frames are stored in the cache in order to deal with the requirements of low delay, anti-lag and intelligent packet loss.

The tradeoff between delay and lag

The delay and lag of live broadcast is the two indexes which are very concerned when analyzing the quality of the live broadcast service. The scene of the interactive broadcast is very sensitive to the delay, and the news sports live more attention to the smoothness of the play.

However, these two indicators in theory, is a pair of contradictory relationship-the need for a lower delay, it indicates that the server side and the player buffer must be shorter, the abnormal jitter from the network is prone to lag, the business can accept higher latency, the service side and the player can have a longer buffer, To respond to jitter from the network, providing a smoother live experience.

Of course, for the network conditions are very good users, these two can be guaranteed at the same time, here is mainly for the network conditions are not so good users, how to solve the problem of delay and lag.

There are usually two techniques to balance and optimize these two indicators.

First, the service side provides a flexible configuration policy, more sensitive to delay requirements, the service side in the case of guaranteed keyframes, a small buffer queue for each connection, and for the more high-latency broadcast, the appropriate increase in the buffer queue length, to ensure smooth playback.

Second, the service side of all connected network situation Intelligent detection, when the network is in good condition, the server will reduce the size of the buffer queue of the connection, reduce latency, and when the network condition is poor, especially when the jitter is more obvious, the server to increase the buffer queue length for the connection, priority to ensure the smoothness of playback.

Packet loss Policy

When do I need to drop a packet?

For a network connection is good, delay is also relatively small connection, packet loss strategy will never be useful. and the network connection than the poor users, because the download speed is relatively slow or jitter is relatively large, the user's delay will be higher.

In another case, if the live stream key frame interval is longer, then in the case of ensuring that the first packet is a keyframe, viewers watching the program may have a delay in reaching the length of a keyframe sequence. In both cases, you need to enable the packet loss policy to adjust the playback delay.

There are two questions to solve about packet loss:

One is to correctly determine when the need to drop packets;

The second is how to drop the packet to make the audience's playback experience the least impact. A good practice is that the backend cycle monitors the length of all connected buffer queues, so that the queue length and time form a discrete function relationship, and the backend analyzes this discrete function by self-research algorithm to determine if packet loss is required.

The general drop-frame strategy is to discard a complete sequence of video frames directly, which is seemingly simple, but has a great impact on user playback. And should be the background of the gradual drop frame strategy, each video frame sequence, lost the last one to two frames, so that the user's perception of the smallest, smooth gradually reduce the effect of delay.

Live Technical Details 1