H264 in WEBRTC those pits

WebRTC represents the best technology in the field of real-time communication since the date of birth. But for a long time, it supported the video encoder only VP8, and later with H265/VP9 as the representative of the birth of the next generation of video Encoders, WebRTC appeared VP9 Codec. The most widely used H264 has been kept out of sight. Until Cisco announces its H264 codec open source for OpenH264, and has paid H264 royalties for all OpenH264 users, in the event of an IETF WEBRTC meeting, The H264 and VP8 are included in the video encoder required to support WEBRTC. Next, Google finally added support for H264 in WEBRTC, on PC platforms (Windows and Macs), encoders are used with OpenH264, decoders are used with ffmpeg, on iOS platforms, Encoders and decoders can use both OpenH264 and ffmpeg, as well as hardware codecs supported by Apple's Videotoolbox. On the Android platform, you can use OpenH264, Ffmpge, or Mediacodec. This is a great boon for the vast majority of companies that need H264. With a little configuration in the downloaded WEBRTC code, you can use H264.
WEBRTC is known for its excellent QoS, and its VP8 and VP9 videos can be kept smooth under poor network conditions, and its quality is very good relative to the current network bandwidth. If the codec replaced with H264, its quality can be compared with VP8/VP9.
With this problem, we assessed the quality of the H264, although in the bandwidth and set the conditions of packet loss, H264 is still relatively smooth, but the odds of its emergence is significantly higher than the VP8/VP9, the delay is greater than VP8/VP9, sometimes the quality will be relatively poor, there are obvious block effect, It is easy to judge that this kind of block effect is caused by the loss of coding quality. Theoretically, a good QoS design implementation, should not be related to the type of codec, and the recognized H264 Rd performance is better than VP8, why in the WEBRTC H264 quality is worse.
With this problem, we have done an in-depth analysis of WEBRTC. The QoS policies of the WEBRTC are mainly rate evaluation (similar to available bandwidth evaluation), Nack retransmission, FEC (forward error correction), PLI request, etc., these controls should be independent of the codec type. But the truth is not so, in the implementation of WEBRTC, if codec is chosen as H264, FEC is closed. VP8/VP9 has a support time rating, although OpenH264 also supports time ratings, but cannot be opened in WEBRTC.
With regard to FEC, Google's explanation is as follows:
Payload types without picture ID cannot determine this a stream is complete
Without retransmitting FEC, so using FEC + NACK for H. (instance) is
A waste of bandwidth since FEC packets still has to be transmitted. Note
That's the case with FLEXFEC.
The h264 does not have a picture ID, so the flow integrity cannot be judged, so using fec+nack is a waste of bandwidth because FEC needs to be flushed.
Is this really the truth?
The answer is that this is only part of the truth.
The real truth is
1. When H264 FEC is enabled, it causes the retransmission packet to be discarded. The receiving end of the Jitterbuffer has been silly waiting, resulting in a long wait
2. H264 FEC recovery hidden bug, easy to trigger the flow integrity error caused decoding failure, causing video lag.
3. What Google is saying is that the picture ID is the concept of VP8/VP9, which is used to indicate the continuity of the video frame, while the H264 can only rely on the RTP serial number to adjudicate the succession.

The reason why H264 cannot enable time rating is that the H264 RTP packet protocol cannot contain information about the video reference hierarchy, and thus cannot decide whether the different hierarchies are complete, and VP8/VP9 's RTP packet protocol contains abundant information enough to judge the flow integrity.

So, the root cause is that VP8/VP9 is Google's biological, all for their tailor-made, and H264 is Baoyang, only hard to this box set.

But knowing these reasons, it is easy to find a solution, to achieve a quality better than VP8 H264 plan.

Note: The WEBRTC, which is discussed in this article, has a version number of 52 and the latest code may differ

This article is done by Sandfox and myself. Reference www.webrtc.org RFC6386-VP8 Data format and decoding guide RFC6184-RTP Payload format for H264 Video