Live experience in depth optimization program--even wheat interactive live

First, preface

Mobile Live the fire has been burning from 2015 to 2016, and there is no doubt that live broadcasting is one of the hottest areas in the mobile Internet, and it attracts a lot of commercial capital under the guidance of super-heat. It is the real air outlet that the live application faces in the moment when all the big live applications are in full bloom. Standing in this tuyere, live application only grasp the good vane, launched with high user stickiness of the differential function, in this constantly new era to establish a foothold, to obtain an unshakable position.

(Mobile Live Hot)

Most of the current domestic live applications, using a single-host mode, the host and the audience only use words, likes, gifts and other ways to interact. When the host broadcast live, if the audience can interact with the real-time video, give the audience a chance to face the wheat, which will greatly enhance the user's sense of participation and happiness, increase user stickiness. And there are very few applications on the market that can provide this kind of interactive live streaming function, which will also be the main competitive field for live applications in the second half of 2016.

Second, Lian Mai Interactive live what is

For a more intuitive explanation of what interactive live broadcasting is, give a simple example: the traditional live broadcast is like watching "news broadcast", viewers can only watch this program, and occasionally can send messages through mobile phone messages and program groups to interact. Of course, now based on the interconnection of the network is much more advanced, you can use the Internet to send text, like, send gifts, the real-time message is also greatly improved, but in essence, watching the "news broadcast" experience similar.

And the interactive live is like to arrive at the Mango station Happy Camp's record scene, the audience sits in the recording scene Auditorium, May watch the program, also has the opportunity to be invited to the stage and the host interaction, of course the host can invite several audiences to come to the stage to interact, but the interactive content other audience also can see.

Compared with the traditional one-way live broadcast, the interactive live stream gives the audience a more direct sense of participation and the satisfaction of real-time interaction with the main broadcast video, which has a significant effect on enhancing the activity and stickiness of the live broadcast application.

Third, even wheat interactive broadcast function flow

① host normal Start live, the ordinary audience to see the host's single live picture;

② need to make a request for the MAI's audience to enter the application list of the company;

③ the host selects one or more viewers from the list of applications to carry out the operation, and the host interacts with the real-time audio and video viewers, while the interactive live system generates a "synthetic picture";

④ ordinary viewers to see the live screen for the host and even the audience of the "synthetic picture";

⑤ the end of the headset, restore the host single live mode.

Four, even wheat interactive live implementation plan

Next, we discuss the specific implementation of the interactive live streaming, which will mainly describe the interactive real-time high and realistic feasibility of two scenarios. These two schemes NetEase Yunxin has the practice in the project, below will analyze the respective merits and demerits in detail.

In order to achieve interactive real-time high-quality, the first need a set of similar, Skype and FaceTime multi-voice video real-time call system. This set of real-time call system can choose independent research and development or based on open source software such as: Google's WEBRTC do two times development, NetEase Cloud letter independent research and development of a set of private protocol based on a multi-person real-time call system, the following brief introduction of the multi-person real-time call system of some key technical details.

① Multi-person audio and video real-time call system in order to reduce the call delay, we use the UDP protocol as the Transport Layer protocol, it is well known that the UDP protocol is unreliable, in order to improve the real-time audio and video call effect under the weak network, need to use related schemes to do QoS protection, mainly including: a) Using the network state-based audio and video bitrate adaptive algorithm, according to the current network packet loss, delay adaptive reduction or increase audio and video bitrate and frame rate, through this method to reduce network congestion, improve call quality; b) Use the Intelligent JITTERBUF algorithm to smooth network jitter, At the same time, the internal use of audio-coded packet loss compensation (PLC) algorithm to further improve the call quality; c) using a video codec based on multilayer reference to reduce the lag after video packet loss; d) the entire UDP transport layer uses the forward error correction FEC algorithm for intelligent protection to maximize the effectiveness of real-time audio and video calls. Based on our actual testing, audio and video calls can resist 20% drops and 600ms of network jitter after using the appellate QoS protection strategy.

② Multi-person audio and video real-time call system in order to ensure the quality of the premise to minimize the call flow, audio codec mainly to Opus-based, Opus Fusion absorbed the Celt and silk coding advantages, with high sound quality, high compression rate, high anti-loss features, very suitable for mobile networks. Video codec we use openh264,openh264 codec performance is excellent, at the same time, with: dynamic bitrate, dynamic frame rate, such as time-domain layered and other suitable for mobile network real-time call characteristics. At the same time we use the self-developed noise reduction algorithm, with echo cancellation, automatic gain and comfort noise and other audio processing algorithms to further ensure the quality of audio.

③ now users for the clarity of video requirements more and more high, our multi-person real-time call system can support 720p,720p under the pure software codec to the CPU overhead, so on the machine can turn on the hardware codec, for the need for 720p resolution to use hardware codec as far as possible. For Apple phone Hardware codec is basically only related to iOS version, and the Android situation is much more complex, not only with the phone hardware, but also related to the ROM of each phone, in order to solve this problem need to do the adaptation. We have adapted to most Android devices with the help of NetEase's powerful mobile application testing department.

④ no global server Deployment and network topology, it is impossible to frame a perfect multi-person audio and video real-time call system. Rely on NetEase Cloud in the world's computer room node, we have built a number of multi-line access network topology, the deployment of high-availability server cluster, and the use of intelligent allocation algorithm and routing strategy, for the trans-provincial, cross-operator, multi-national real-time calls to provide high-quality transmission channels.

To achieve the ideal effect of the interactive live streaming, a strong and perfect multi-person real-time call system is the premise. After a brief introduction of a powerful multi-person real-time call system needs to have the characteristics, then we can discuss the next-to-even-wheat interactive live broadcast of the specific implementation.

Programme one:

The traditional live streaming process is: After the host client collects and encodes the audio and video data, it pushes the stream to the CDN directly using the RTMP protocol, and the other viewers use the corresponding pull-stream address to pull the audio and video stream from the CDN.

The program uses real-time call system for host and audience of real-time interactive headset, through the real-time call channel host side to receive the audience sent audio and video data, the host side will own voice and the audience's voice to do mix, and will own the picture and the audience's picture to do the video synthesis, Finally, the mixed sounds and images are pushed to the CDN streaming media server.

The architecture diagram is as follows:

Advantages of the scheme:

① hosts and even wheat audiences use real-time audio and video to interact with each other, real-time, and the interaction between the host and the audience in the composite screen seen by the audience is also synchronized in real time.

The ② scheme has little change to the original live streaming client, and the service side does not need to be modified. The overall implementation of the scheme is simple and can be quickly built using existing systems and SDKs.

Our interactive live streaming on the windows side of NetEase Bobo is a program that runs stably after the launch in the second half of 2015. Although this scheme is simple and feasible, there are two more deadly problems for mobile terminals:

① host side of the bandwidth pressure is very large, from the architecture diagram can be seen, the host side must send a copy of the audio and video data through the real-time call system to the viewers, but also need to push the way to the CDN streaming media server. Therefore, compared to single-player live, the upstream stream of the wheat will be twice times the original. This twice times the traffic in the Windows stable wired network environment, the impact is small, but in the upstream bandwidth is limited mobile network, will greatly affect the effect of live broadcast.

② host side of the video codec pressure is very large, and the cause of the bandwidth pressure is large, the host must encode a video to the audience, while the need to synthesize and encode all the way to the CDN, two-time coding for mobile performance pressure is very large, Real-world testing for 720p of the resolution of the interactive live streaming only on the flagship model can barely support, but the heat and power consumption will be greatly increased.

Because of these two problems, we think that scenario one is not quite applicable in mobile scenarios.

Scenario Two:

In order to solve the problem of the solution, our team spent 3 months to do technical research, design and develop an alternative plan.

The architecture diagram is as follows:

In this scheme, the key of the scheme is that the host no longer pushes directly to the CDN streaming server, but is based on the real-time audio and video calling system, which is forwarded to the interactive live server by the relay server of the real-time audio and video, and then pushed to the CDN streaming media server by the interactive live broadcast server.

Multi-person audio and video real-time call system, can realize the real-time interaction of many people, and all the packets in the multiplayer mode are transferred through the audio and Video transit server. Audio and Video Relay server forward to participate in the client at the same time, forwarding a copy to the interactive live server, the interactive live broadcast server to the received voice mix, while the video mix processing, processing completed and then pushed to the CDN streaming media server. Through this scheme, the mix mixing and push-stream operation by the host side of the scheme one is passed on by the interactive live server.

Advantages of the scheme:

① host and even wheat audience use real-time audio and video to carry on the interactive, real-time high, the general audience to see the synthesis of the image of the host and audience interaction is also synchronous real-time;

② can realize the interactive live of multi-connected wheat, the function difference is obvious;

③ upstream push stream of all clients no longer relies on the TCP-based RTMP protocol, but uses NetEase self-developed UDP-based high-performance private protocol, the Transport layer of QoS assurance more intelligent and efficient;

The bandwidth and performance pressure on the host side of the ④ solution is no longer present, and the program is ideal for mobile-connected live streaming.

Of course, although the program has many advantages, but it is the most difficult to achieve. First of all, this architecture involves the integration of the real-time call system and the interactive live Broadcasting system, with high complexity of architecture and code. In particular, the interactive live broadcast system, due to the processing of video mixing, server-side code performance and hardware requirements are very high. In order to solve this problem, we used many high-performance physics machines in NetEase room as the interactive live server, and optimized the server-side code architecture and processing process continuously, and finally satisfied the business demand through continuous optimization. In conclusion, we believe that this program is the most suitable for mobile side of the current interactive live streaming program.

V. Looking to the future

As a mobile live broadcast in 2016, developers and companies around the world are thinking about how to deliver better services. We believe that content is always the king of live development, as a research and development engineer is responsible for the content of the transmission escort, providing high-definition, smooth and low latency live content. and the differentiated function will become the highlight of the live application, in which the live streaming application with the interaction of the wheat will increase the user's participation, happiness and increase the user stickiness, even the importance of the interactive live streaming is self-evident.

Technology is always the first productivity, the current VR virtual reality technology and live as hot. The VR and live broadcast VR live can make the audience immersive, its unique interactive way may in the near future, will be in the broadcast field of a new round of change. The interactive live broadcast in the future will be a pan-live and scene-based live as the theme, fully integrated with VR technology, fully open the news, tourism, education, medical and other full-scene immersive "Live +" era.

Live experience in depth optimization program--even wheat interactive live