Mobile Interactive live Streaming (introductory article)

This article from the NetEase cloud communityPreface

This article is for mobile video live development Novice, in order to quickly get started, using the powerful Google search engine with their own understanding of the "video broadcast background Knowledge".

Background Knowledgenoun explanationPush-Stream ProtocolRTMP

Real Time Messaging Protocol (live messaging protocol)

With Flash Player as the player client, Flash Player is now installed on nearly 99% of the world's PCs, so it is generally not necessary to install plug-ins for viewing audio from rtmp streaming media systems. Users only need to open the Web page, you can directly watch streaming media, very convenient

1. The plaintext protocol that works over TCP, using port 1935;

2. Rtmpt encapsulated in HTTP requests, can traverse firewalls;

3. Rtmps similar to rtmpt, but using an HTTPS connection;

HLS

HTTP Live Streaming

HLS is part of Apple's QuickTime X and iphone software system. It works by dividing the entire stream into small HTTP-based files to download, one at a time. When media streaming is playing, clients can choose to download the same resources at different rates from many different alternative sources, allowing streaming media sessions to accommodate different data rates. When you start a streaming session, the client downloads a extended m3u (m3u8) playlist file that contains metadata to find the available media streams.

Client Support

• iOS starts from 3.0 as a standard feature.

• Adobe Flash Player supports HLS from 11.0 onwards.

• Google's Android has been supporting HLS since Honeycomb (3.0).

• Vodobox HLS player (Android,ios, Adobe Flash player)

• Jwplayer (Adobe Flash)

• FlowPlayer (Adobe Flash, using the HLSJS version does not use Adobe Flash)

• The Windows 10 EDGE browser starts to support HLS.

H264 Encoding

H264 is a high compression rate coding standard, how to compress? The general video collection is 25 frames per second, that is, 25 times a second, in fact, each picture of the content is not small, the method of compression is to use the algorithm, only each picture changes the part of the difference to save, so that the video file is much smaller

MKV

Russian Матроска is Матрёшка (Russian set of baby) misreading, because the Matroska work principle with layers of the Russian doll stack, is the "more tight package," the name. MKV is just one of the file formats of the Matroska Media series.

Related Terms

YUV

YUV is mainly used to optimize the transmission of color video signal, so that it is backward compatible with old black and white. Compared with the RGB video signal transmission, its greatest advantage is that it only takes up a very small bandwidth

PCM

Pulse code Modulation (PCM) is a time continuous, value continuous analog signal transformed into time discrete, the value of discrete digital signal after transmission in the channel. Pulse code modulation is to sample the analog signal first, then quantify the magnitude of the sample value, and encode the process

muxer

Muxer refers to merging files, merging video files, audio files, and subtitle files into one video format. For example, the RMVB format of video, MP3 format audio files and SRT format subtitle files, merged into a new MP4 or MKV format files.

demuxer

Demuxer is the inverse process of muxer, which is to extract different format files from the synthesized files.

Overall ArchitectureArchitecture Diagram

role Functions

app-side push stream

Audio Acquisition Model

Optimization StrategyQuality of service policy

Push the flow end will be based on the current uplink network control audio and video data packet and encoding, in the case of poor network, audio and video data sent out, resulting in data stranded in the local, then, will stop the encoder to prevent further retention of data, at the same time, according to network conditions to choose the appropriate strategy to control audio and video transmission.

For example, if the network is very poor, the push stream will give priority to the audio data, ensure that the user can hear the sound, and in a certain interval to send key frame data, to ensure that users can see some changes in the screen after a certain time interval.

Configuration keyframes

Reasonable control of the keyframe send interval (2 seconds or 1 seconds is recommended), which reduces the back-end processing and sets a smaller creation condition for the back-end buffers.

Cloud ServerOverall structure

service AgreementSRS

• Open source

• 1. Simple architecture and powerful function
  2. Main support RTMP protocol
  3. Cluster support

crtmp

• Open source

• 1.c++ Development
  2. Rich Support Protocol
  3. Not good for cluster support

nginx-rtmp

• Open source

• 1. Fully asynchronous model implementation, performance benefits
  2. Insufficient stability

Red5

• Open source

• 1. Pure Java
  2. Insufficient performance

FMS

• No open source

• 1.adobe Streaming Media Server
  2. Performance and function are good

video transcoding function Problem

Recording, live transcoding, yellow, and distribution.

The bit rate of the video live stream end is determined by the push-flow end, that is, the bitrate of the stream end is consistent with the bitrate of the push-flow end. However, the following scenarios can cause poor live performance:

• The rate of push-stream end is not matched with the bandwidth of the broadcast stream. When the rate of push-stream is high and the client's bandwidth resource is limited, the playback will be stuck, while the low-rate of push-stream is lower but the client is less effective for the broadcast efficiency.

• The player plug-in needs to implement multi-bitrate switching. The front-end player plug-in can often set the bitrate switch, which requires the same-way push stream can provide multiple bitrate at the same time the broadcast stream address.

As a result, live transcoding provides the ability to provide multiple different bitrate multicast addresses at the same push-stream address.

app End Pull Stream

Encoding PolicyPush-stream encoding-hard

Recommended Andorid4.3 (API18) or the use of hard-copy, the following version of the use of soft, ios using a full-hard programming;

Pull Stream Encoding-soft

Andorid, iOS players are using a soft decoding scheme, after we and a large number of customers testing and summary, although the loss of power consumption, but in some detail performance will be better, and controllability, compatibility is strong, error situation is less, recommended use.

Pros and consSoft Coding/decoding

• Advantages

1. Strong compatibility

2. Color is stronger than hard coding

3. Coding class with large operating space and high degree of freedom

• Disadvantages

1. Eat CPU, consume larger

hard-coded/decoded

• Advantages

1. Low power consumption and high execution efficiency

• Disadvantages

1. The difference of the chip

2. Controllability is relatively low

Featured FeaturesInteractive WhiteboardData Encapsulation

Layout Structure

Bomb ScreenLayout Structure

Danmakuflamemaster of the projectile screen

References

• Open source Framework RTMP live streaming client for Android Https://github.com/begeekmyfriend/yasea

• Live architecture Diagram https://netease.im/edu

• rtmp Protocol Introduction Https://en.wikipedia.org/wiki/Real-Time_Messaging_Protocol

• Audio processing Queue Https://developer.apple.com/library/content/documentation/MusicAudio/Conceptual/AudioQueueProgrammingGuide /ABOUTAUDIOQUEUES/ABOUTAUDIOQUEUES.HTML#//APPLE_REF/DOC/UID/TP40005343-CH5-SW1

• Video broadcast technology principles and solutions reference HTTPS://GITHUB.COM/F2E-JOURNEY/XUEQIANBAN/ISSUES/61

NetEase Cloud New User package:https://www.163yun.com/gift

This article from the NetEase cloud community, by the author Keris authorized release

Mobile Interactive live Streaming (introductory article)