An analysis of IOS Live app development process _ios

This article for everyone to share the iOS live Class app development process, for your reference, the specific content as follows

One. The general process of audio and video processing:

data collection → code → transmission (streaming media server) → decoding data → playback display
1. Data collection:
Video and audio data are collected by the camera and pickup, which is the original data.
relating to technology or agreement:
Camera: CCD, CMOS
Pickup: acousto-Electric conversion device (microphone), audio amplification circuit
2. Data coding:
Use related hardware or software to encode and process audio and video raw data (digital) and processing (such as audio and video mixing, packaging and packaging, etc.) to get available audio and video data
relating to technology or agreement:
Encoding mode: CBR, VBR
Encoding format
Video: h.265, H.264, MPEG-4, packaging containers have TS, MKV, AVI, MP4 and so on
Audio: g.711μ, AAC, Opus and so on, the package has MP3, OGG, AAC, etc.
3. Data transmission:
After the completion of the coding of audio and video data transmission, the early audio and video through coaxial cable, such as transmission, IP network development, the use of IP network excellent transmission
relating to technology or agreement:
Transport protocol: RTP and RTCP, RTSP, RTMP, HTTP, HLS (HTTP Live streaming), etc.
Control signaling: SIP and SDP, SNMP, etc.
4, decoding the data:
Using the relevant hardware or software to decode the received encoded audio and video data and get the image/sound that can be directly displayed
relating to technology or agreement:
The general corresponding encoder will have the corresponding decoder, there are some Third-party decoder plug-ins and so on
5, Play display:
Display the corresponding image or sound in the monitor (TV, monitor screen, etc.) or speakers (headphones, horn, etc.)
relating to technology or agreement:
Monitors, speakers, 3D glasses, etc.

(ii) video streaming and video streaming work process analysis:
1. Video Push Stream end
Push stream is to send the collected audio, video data through streaming media protocol to the streaming media server.
First, select Streaming media protocol
Now live broadcast application, adopt RTMP agreement mostly, also have partial use HLS agreement.
Using the RTMP protocol, it is necessary to look at its interaction with the streaming media server process, the default port of the RTMP protocol is 1935, using the TCP protocol. And you need to understand the FLV encapsulation format.
Using the HLS protocol, because it involves slicing, the delay will be relatively large, need to understand TS flow.

second, the acquisition of audio and video data
Do live, the source of data is indispensable, is to collect camera, microphone data.
To capture audio and video data on the iOS platform, you need to use the Avfoundation.framework framework to get the voice and video data from the callback of the capturesession session.

Three, hard coded, soft coded audio and video data

Soft coding is the use of CPU resources to compress audio and video data, hard coding is the opposite.
Soft code, now widely used FFmpeg library combined with the code library to achieve, ffmpeg+x624 to encode video data Yuv/rgb output H264 data,
FFMPEG+FDK_AAC to encode the audio data PCM output AAC data.

Four, according to the selected streaming media protocol envelope audio and video data
pack audio and video into packet.

v. Send packet data to server interactively
according to the selected streaming media protocol, send the corresponding instruction to connect to the server, after the successful connection to the server, you can send packet data. &NBSP
Part 2.  
Pull stream, which is to obtain audio and video data from a streaming media server. &NBSP
I), parsing protocol  ,
the player end resolves the streaming media Protocol (RTMP,HLS) based on the URL. &NBSP
II), Solution Encapsulation  ,
Solution Encapsulation, is the DEMUX process, from the container format (flv,ts), the separation of audio and video data. &NBSP
III), decoding  ,
Decoding, is to extract the data extracted to restore the original data. Decoding is to convert H264 into YUV,AAC into PCM. &NBSP
Decoding can use soft decoding, hard decoding. &NBSP
Soft decoding is the use of CPU resources to extract data, the use of the way is ffmpeg decoding. &NBSP
Hard decoding, for iOS platforms, you can use Videotoolbox.framework (the framework can only be used in iOS 8.0 and above)
to hard decode video data. On the Android platform, you can use Mediacodec to hard decode video data. &NBSP
IV), rendering data  ,
renders YUV data using OpenGL rendering, rendering the video screen. The PCM is fed into the device's hardware resources to play, generating sound. &NBSP
iOS plays streaming audio, using the audio Queue, that is, using the audiotoolbox.framework framework.

Iii. useful third-party resource sharing for iOS-developed iOS Live platform:  
1.  real-time beauty reference URL: http://altitudelabs.com/blog/ REAL-TIME-FILTER/&NBSP
2. Live test address:  
hls:http://live.3gv.ifeng.com/live/hongkong.m3u8 Phoenix Satellite television Hong Kong  
Smtp:rtmp://live.hkstv.hk.lxdns.com/live/hks Hong Kong satellite TV  
rstp:rtsp://rtsp.vdowowza.tvb.com/tvblive/ Mobileinews200.stream Interactive news station 1&NBSP
3.  HLS streaming tools download, for processing m3u8 audio, video source HLS streaming Tool Download
4. Live related SDK
1). Videocore in the Sina Cloud has documented, can read the iOS push Stream SDK using the document
2). NetEase Cloud Letter Live SDK seems to be charged, the price is not cheap, the general company may not use, but you can try to see, to understand, to help learn!
3). Storm clouds Live    http://www.baofengcloud.com/apisdk/sdk/live/iossdk.html
4. Cloud Live       Http://www.rongcloud.cn/live
5). Tencent Live sdk  https://www.qcloud.com/solution/video.html

5. Broadcast related open Source Library
1). Beautifyfacedemo based on gpuimage image processing, do beauty aspects can refer to the
2. Lf.swift Open source rtmp Swift version, you can learn! As if the author is Japanese, or is working in Japan, wrote a lot of Japanese!
3). Plplayerkit a player that does not rely on FFmpeg to support rtmp
4). Hls-demo Https://github.com/yangchao0033/HLS-Demo HLS Play Demo
5. Smarterstreaming https://github.com/daniulive/SmarterStreaming Live SDK (support for private protocols and rtmp push stream, such as Windows push stream/android push stream/ios push stream /windows player/android player/ios player)

Four, iOS development of the iOS live platform common video broadcast related protocols detailed

1, RTMP (real time messaging Protocol, real-time messaging protocol)
RTMP is an open protocol for the development of audio, video and data transmission between the Flash Player and the server by Adobe Systems Company. It has three varieties:
1), working on TCP above the plaintext protocol, using port 1935;
2), Rtmpt package in the HTTP request, can traverse the firewall;
3), rtmps similar rtmpt, but using HTTPS connection;
The RTMP protocol is used by Flash for object, video, audio transmission. This protocol is based on the TCP protocol or polling HTTP protocol. The RTMP protocol is like a container for loading packets, which can be either an AMF format or a video-audio data in FLV. A single connection can transmit multiple network flows through different channels, and the packets in these channels are transmitted in a fixed size packet.

2, RTSP (real time streaming Protocol, real-time streaming transport protocol)
RTSP defines how a one-to-many application can efficiently transfer multimedia data over an IP network. RTSP provides an extensible framework in which data sources can include real-time data and existing stored data. The purpose of this protocol is to control multiple data sending connections, to provide a way to select Send channel such as UDP, multicast UDP and TCP, and to provide a method for choosing based on RTP forwarding mechanism.
RTSP syntax and operation are similar to http/1.1, but do not specifically emphasize time synchronization, so it is more tolerant of network latency. The proxy server's caching function also applies to RTSP, and because RTSP has a redirect function, the service-providing server can be switched based on actual load conditions to avoid delays caused by too large a load to be concentrated on the same server.

3, RTP (Real-time transport Protocol, real-time transmission protocol)
RTP is a Transport layer protocol for multimedia data streams, which details the standard packet format for transmitting audio and video on the Internet. RTP is often used in streaming media systems (with RTCP protocol), video conferencing and one-click System (with H.323 or SIP), making it the Technology foundation of IP telephony industry.
RTP, which is built on a UDP protocol and often used with RTCP, does not itself provide a timely delivery mechanism or other quality of service (QoS) assurance, and relies on low-level services to implement this process.
RTP does not guarantee the transmission or prevention of disorderly transmission, and is not sure of the reliability of the underlying network, just send, regardless of whether the transmission dropped packets, and whether the receiver received the packet. RTP implementation of orderly transmission, RTP serial number allows the receiver to reorganize the sender's packet sequence, while the serial number can also be used to determine the appropriate packet location, such as in video decoding, do not need sequential decoding.

4, RTCP (Real-time transport control Protocol, real-time Transmission Controller protocol)
RTCP is a matching protocol for RTP, which provides the channel control for RTP media streaming. RTCP and RTP collaborate to package and send multimedia data, and periodically transfer control data between multimedia streaming session participants.
The main function of RTCP is to provide feedback to the quality of service (QoS) provided by RTP, collect statistics on relevant media connections, such as bytes transmitted, transfer packets, lost packets, one-way and bidirectional network latency, and so on. Network applications can use the information provided by RTCP to improve service quality, such as limiting traffic or using a codec with a smaller compression ratio.

The above is the entire content of this article, I hope to help you learn, but also hope that we support the cloud habitat community.