Frontend development: H5 live broadcast and h5 live broadcast

Frontend development: H5 live broadcast and h5 live broadcast
Preface

Not long ago, I took the time to study and explore the current popular live video streaming, understand its overall implementation process, and explore feasible solutions for mobile HTML5 live streaming.

It is found that the mainstream live video broadcast solutions on the WEB currently include HLS and RTMP. The mobile WEB end currently focuses on HLS (HLS has a delay problem, you can also use video. js uses RTMP), while the PC uses RTMP for real-time purposes. Next we will share H5 live video themes around these two video stream protocols.

I. Video Stream protocol HLS and RTMP

1. HTTP Live Streaming

HTTP Live Streaming (HLS) is an HTTP-based video stream protocol, which is implemented by Apple. Quick time, Safari, and Safari on Mac OS support HLS well, later versions of Android have also added support for HLS. Common clients such as MPlayerX and VLC also support the HLS protocol.
The HLS protocol is based on HTTP, while a server that provides HLS needs to do two things:

The browser uses the m3u8 file. M3u8 is similar to m3u in audio List format. It can be simply considered that m3u8 is a playback list containing multiple ts files. The player plays the video one by one in sequence. After the video is completely played, request the m3u8 file to obtain the playlist containing the latest ts file and continue playing. The whole live broadcast process is composed of an constantly updated m3u8 and a bunch of small ts files. m3u8 must be dynamically updated, and ts can go through CDN. A typical m3u8 file format is as follows:

#EXTM3U#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000gear1/prog_index.m3u8#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=311111gear2/prog_index.m3u8#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=484444gear3/prog_index.m3u8#EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=737777gear4/prog_index.m3u8

We can see that the HLS protocol is essentially an HTTP request/response, so it is adaptable and will not be affected by the firewall. But it also has a fatal weakness: latency is very obvious. If each ts is split by 5 seconds and a m3u8 table has six ts indexes, a latency of at least 30 seconds will occur. If the length of each ts is reduced, and the number of indexes in m3u8 is reduced, the latency will be reduced, but the buffer will be more frequent, and the request pressure on the server will also increase exponentially. Therefore, you can only find one discount point based on the actual situation.

For a browser that supports HLS, you can directly write it in this way to play the video:

<video src=”./bipbopall.m3u8″ height=”300″ width=”400″  preload=”auto” autoplay=”autoplay” loop=”loop” webkit-playsinline=”true”></video>

Note: HLS only supports safari on the PC side. Similar to chrome, HTML 5 video is used.
Tags cannot be played in m3u8 format. You can directly use some mature online solutions, such as sewise-player, MediaElement, videojs-contrib-hls, and javasplayer.

2. Real Time Messaging Protocol

Real Time Messaging Protocol (RTMP for short) is a live video Protocol developed by Macromedia. It belongs to Adobe. This solution requires the establishment of specialized RTMP streaming Media services such as Adobe Media Server, and the browser can only use Flash to implement the player. Its real-time performance is very good, and the latency is very small, but it is hard to support mobile WEB playback.
Although it cannot be played on the H5 page of iOS, for iOS native apps, you can write and decode it for resolution. RTMP has low latency and good real-time performance.
Browser, HTML5 video
Tags cannot play RTMP videos, which can be achieved through video. js.

<link href=“http://vjs.zencdn.net/5.8.8/video-js.css” rel=“stylesheet”><video id=“example_video_1″ class=“video-js vjs-default-skin” controls preload=“auto” width=“640” height=“264” loop=“loop” webkit-playsinline><source src=“rtmp://10.14.221.17:1935/rtmplive/home” type=‘rtmp/flv’></video><script src=“http://vjs.zencdn.net/5.8.8/video.js”></script><script>videojs.options.flash.swf = ‘video.swf’;videojs(‘example_video_1′).ready(function() {this.play();});</script>

3. Comparison between video stream protocol HLS and RTMP

Ii. Live broadcasting

 

Currently, live video is usually displayed on pages such as YY live and Yingke live. The structure can be divided into three layers:
① Background video Layer
② Attention and comment Module
③ Thumb up animation

The current H5 is similar to the live video page, so it is not difficult to implement the technology. It can be divided:
① Use video tags to play the background of the video at the bottom
② The focus and comment module uses WebScoket to send and receive new messages in real time through DOM and CSS3.
③ Likes with CSS3 Animation

After learning about the live video format, we will learn about the live video process as a whole.
Iii. Overall live broadcasting Process
The overall process of live broadcasting can be roughly divided:

• Video Acquisition terminal: it can be an audio/video input device on a computer, or a camera or microphone on a mobile phone. Currently, mobile video is the main video.
• Live streaming video server: An Nginx server that collects video streams transmitted from the video recording end (H264/ACC encoding) and is parsed and encoded by the server, push RTMP/HLS video streams to the Video Playback End.
• Video Playback End: it can be a computer Player (QuickTime Player, VLC), a mobile native Player, or an H5 video tag. Currently, it is mainly a mobile native Player.

 

(Web Front-end learning and communication group: 328058344 chat prohibited, not welcome !)

Iv. H5 Video Recording

For H5 video recording, webRTC (Web Real-Time Communication) is a technology that supports Real-Time voice conversations or video conversations in Web browsers, the disadvantage is that it only supports PC Chrome well, and mobile terminal support is not ideal.

Note:

 

 

Although Google has been pushing WebRTC, there are already many products, but most mobile Browsers Do not support webRTC (not the latest iOS 10.0 ), therefore, the real video recording still depends on the client (iOS, Android), and the effect will be better.

WebRTC support

2. The iOS native app calls the camera to record video.
   ① The original audio and video data streams can be collected using AVCaptureSession and AVCaptureDevice.
   ② Encode videos with H264 encoding and audio with AAC encoding. In iOS, the encapsulated encoding library (x264 encoding, faac encoding, and ffmpeg encoding) is provided to encode audio and video.
   ③ Pack the encoded audio and video data.
   ④ Establish an RTMP connection and push it to the server.
   

5. Set up Nginx + Rtmp live stream service

brew tap homebrew/nginx

② Install nginx-rtmp-module

brew install nginx-full –with-rtmp-module

2. nginx. conf configuration file: Configure RTMP and HLS

Find the nginx. conf configuration file (path/usr/local/etc/nginx. conf) and configure RTMP and HLS.

① Add the configuration content of rtmp before the http node:

② Add hls configuration in http

3. Restart the nginx service.

Restart the nginx service. Enter http: // localhost: 8080 in the browser. Check whether the welcome page is displayed.

nginx -s reload

Vi. Live Stream Conversion format and encoding streaming

When the server receives the video stream transmitted from the recording end, it needs to parse and encode it and push the RTMP/HLS video stream to the Video Playback End. Common encoding library solutions, such as x264 encoding, faac encoding, and ffmpeg encoding.
Given that the FFmpeg tool integrates multiple audio and video formats for encoding, We can prioritize FFmpeg for conversion and encoding streaming.

1. Install FFmpeg

brew install ffmpeg

2. Streaming MP4 files

Video file address:/Users/gao/Desktop/video/test.mp4
Stream pushing URL: rtmp: // localhost: 1935/rtmplive/home, rtmp: // localhost: 1935/rtmplive/home

// RTMP stream ffmpeg-re-I/Users/gao/Desktop/video/test.mp4-vcodec libx264-acodec aac-f flv rtmp: // 10.14.221.17: 1935/rtmplive/home // HLS protocol stream ffmpeg-re-I/Users/gao/Desktop/video/test.mp4-vcodec libx264-vprofile baseline-acodec aac-ar 44100-strict- 2-ac 1-f flv-q 10 rtmp: // 10.14.221.17: 1935/hls/test

Note:
After streaming, you can install VLC and ffplay (video players supporting rtmp Protocol) to pull streams locally for demonstration.
3. FFmpeg streaming command
① Live broadcast of video files

ffmpeg -re -i /Users/gao/Desktop/video/test.mp4 -vcodec libx264 -vprofile baseline -acodec aac -ar 44100 -strict -2 -ac 1 -f flv -q 10 rtmp://192.168.1.101:1935/hls/testffmpeg -re -i /Users/gao/Desktop/video/test.mp4 -vcodec libx264 -vprofile baseline -acodec aac -ar 44100 -strict -2 -ac 1 -f flv -q 10 rtmp://10.14.221.17:1935/hls/test

② Streaming Camera + desktop + microphone recording for live broadcast

ffmpeg -f avfoundation -framerate 30 -i “1:0″ \-f avfoundation -framerate 30 -video_size 640x480 -i “0” \-c:v libx264 -preset ultrafast \-filter_complex ‘overlay=main_w-overlay_w-10:main_h-overlay_h-10′ -acodec libmp3lame -ar 44100 -ac 1 -f flv rtmp://192.168.1.101:1935/hls/test

For more commands, see:
FFmpeg command for processing RTMP Streaming Media
Frequently Used FFmpeg streaming commands

VII. H5 live video playback

Mobile iOS and Android support the HLS protocol. After the video collection and video stream pushing services are completed, you can directly configure the video tag on the H5 page to play live videos.

<video controls preload=“auto” autoplay=“autoplay” loop=“loop” webkit-playsinline><source src=“http://10.14.221.8/hls/test.m3u8″ type=“application/vnd.apple.mpegurl” /><p class=“warning”>Your browser does not support HTML5 video.</p></video>

VIII. Summary

This article describes the complete process of video collection and uploading, server processing of video streaming, and playback of live videos on H5 pages. It describes the implementation principle of live videos and encounters many performance optimization problems.

① H5 HLS must be H264 + AAC encoded.
② When H5 HLS playback gets stuck, the server can implement a sharding policy to put ts files on the CDN, and the front-end can implement DNS caching as much as possible.
③ In order to achieve better real-time interaction, H5 live broadcast can also adopt the RTMP protocol through video. js
Implement playback.