WebRTC Past Life

WebRTC's Past Life

This article is translated by RWEBRTC

WebRTC 技术是激烈的开放的 Web 战争中一大突破。-Brendan Eich, inventor of JavaScript

No plug-in real-time communication

Imagine mobile phones, TVs, and computers communicating through a unified platform. Just imagine, it's easy to add video chat and peer-to data sharing to your site. This is the vision of WebRTC technology.

Want to have a try? WEBRT c is available in Chrome, Opera, and Firefox. In apprtc.appspot.com, you can try these examples:

1. Open apprtc.appspot.com in Chrome, Opera, or Firefox.
2. Click the Allow button to let the app use your webcam (Web app).
3. Open the URL at the bottom in a new tab, preferably on another computer.

There is an example of this application in this article.

Quick Start

Don't have time to read this article, or just want to write code as soon as possible?

1. Take a look at the overview about WebRTC at the Google I/O conference. (Slideshow here)
2. If you have not used Getusermedia, take a look at this article in the project, where you can see the sample code.
3. To learn about Rtcpeerconnection's API, click here and here, this example implements the WebRTC on a simple interface.
4. Learn more about the server, firewall, and NAT penetration associated with WebRTC. Read here to see the debug log.
5. Wait, you want to try WebRTC? Here are more than 20 Demos, try out these javaScript APIs.
6. Do you have a problem with your machine? Test cases can be found here.

Or, jump straight to our WebRTC Codelab: Step-by-step instructions to build a complete video chat application, including a simple signaling server.

A brief history of WebRTC technology

Making audio and video communication through the network is the ultimate challenge for the network: real-time communication (RTC). Real-time communication is as natural as typing text into a text box on a network, without which it restricts the way we develop new ways of interacting with people.

Historically, RTC has been a big and complex change, requiring expensive audio and video technology licensing or costly development, and the integration of RTC technology with existing content, data and services has been difficult and time-consuming, especially on the web.

Gmail video chat began to pop in 2008, and in 2011 Google launched a video group chat, which uses the Googletalk service, just like Gmail. Google has acquired Gips, a company that has developed many components for RTC, such as codec and echo-cancellation technology. Google Open Source Gips developed technology, with the relevant agencies IET and the world market to develop industry standards. In May 2011, Ericsson implemented the first WEBRTC application.

WebRTC has implemented standards for real-time communication, free plug-in audio data transmission. Requirements are:
-Many Web services already use RTC, but need to download, local application or plug-in. Includes Skype, Facebook, Google Hangouts.
-Download Install upgrade plugin is complex, can be wrong, annoying.
-plugins can be difficult to deploy, debug, troubleshoot, etc.-may require technical authorization, complex integration, and expensive technology. It's hard to persuade people to install plugins.

The guiding principle of the WebRTC project is that APIs should be open source, free, standardized, and built-in to the browser, and more efficient than existing technologies.

Where are we

WebRTC is used in a variety of apps, including WhatsApp, Facebook Manager, appear.in and TokBox platforms. Even the experimental WebRTC on the IOS browser. WebRTC is also used in webkitgtk+ and QT.

Microsoft added the MediaCapture and Stream APIs to the Edge.

The WebRTC implements three APIs:

• Meidastream (aka Getusermedia)
• Rtcpeerconnection
• Rtcdatachannel

Getusermedia is implemented in Chrome, Opera, FireFox, and Edge. Take a look at the cross-browser Demo and Amazon example, using Getusermedia as input to the audio.

Rtcpeerconnection is implemented in Chrome, Opera, and FireFox. Explanation of the name: After several iterations, Rtcpeerconnection was implemented by Chrome and Opera as Webkitrtcpeerconnection and was implemented by FireFox as Mozrtcpeerconnection. Other names are obsolete, and when the standard is stable, the prefixes are deleted. There is a super simple demo project on GitHub, built on the apprtc.appspot.com. This app uses Adapter.ja, a JS library, to help WebRTC cross-browser.

Rtcdatachannel is supported by Chrome, Opera and FireFox. Check out the Demo on Github to see how it's used.

Warning

Some platforms claim that supporting WebRTC may actually support only Getusermedia, but not other RTC components.

My first WebRTC.

WebRTC need to do a few things:

• Get audio or video streams or other data streams.
• Get network information such as IP address and port number, exchange this information with other clients to connect, including through firewalls.
• Report an error to signaling or start a session to close the connection.
• Exchange information with other clients for resolution decoders, and so on.
• Transmit audio or video or other data information.

To obtain and transmit data, WebRTC implements the following APIs:

• MediaStream: Gets the data stream from the customer's camera or microphone.
• Rtcpeerconnection: Audio and video calls, including encryption and bandwidth management.
• RTCDATACHANNEL:P2P data transmission pipeline.

(Next detailed discussion of WebRTC networks and channels)

MediaStream (aka Getusermedia)

The MediaStream API represents synchronous streaming media. For example, a streaming input from a camera and a microphone has synchronized audio and video. (Don't confuse MediaStream and track tags, they're completely different).

Perhaps the easy way to understand MediaStream is to look at its performance:

1. Open demo in Chrome and Opera:
   https://webrtc.github.io/samples/src/content/getusermedia/gum。
2. Open the debugger.
3. Check the global scope for flow variables.

Each mediastream has an input, Navigator.getusermedia (), and an output that is output to the video tag or rtcpeerconnection.

The Getusermedia () method has three parameters:

• A constraint object.
• A successful callback, if called, passes a mediastream.
• The failure callback, if called, passes an Error object.

Each mediastream has a label, such as
"as‘Xk7EuLhsuHKbnjLWkW4yYGNJJ8ONsgwHBvLQ" 
A mediastreamtracks array is returned by Getaudiotracks () and Getvideotracks ().

For
https://webrtc.github.io/samples/src/content/getusermedia/gum/ 
For example Stream.getaudiotracks () returns an empty array because there is no audio, assuming a camera is connected, stream.getvideotracks () returns a Mediastreamtrack array that represents the stream of this camera. Each mediastreamtrack has a type (audio or video), a label (sometimes like a FaceTime HD camera) that represents one or more audio and video channels. In this example, only the video has no audio. It's easy to imagine more examples: for example, a chat app, a front-facing camera and a microphone, a screen sharing app.

In Chrome or Opera, the Url.createobjecturl () method can convert a mediastream to a Blog URL that can be set as the source of the video. (In FireFox and opera, this video source can be created via the stream itself), since the M25 version, Chromium-based browsers (Chrome and opera) allow audio data obtained through Getusermedia to be placed And on the Video tab (but notice that it is muted by default).

Getusermedia can be used as a Web audio input API:

function gotStream(stream) {    window.AudioContext = window.AudioContext || window.webkitAudioContext;    var audioContext = new AudioContext();    // Create an AudioNode from the stream    var mediaStreamSource = audioContext.createMediaStreamSource(stream); // Connect it to destination to hear yourself // or any other node for processing! mediaStreamSource.connect(audioContext.destination);}navigator.getUserMedia({audio:true}, gotStream);

Chromium-based applications or extensions can also use Getusermedia. Add audiocapture and/or videocapture permissions only once at the time of installation. Thereafter the user is not asked about the camera and microphone permissions.

The same is true for Web pages that use HTTPS: for Getusermedia, you only need to grant permissions once. The first time, the Allow button is displayed on the InfoBar of the label.

TODO at the same time, Chrome will not encourage Getusermedia () to access HTTP, at the end of 2015, in the M44 version, you might see a warning when you visit http.

Eventually, more than just cameras and microphones, any other data stream can be put into the mediastream. You can get the data stream from your hard disk, or get it from any other sensor or other input.

Note that Getusermedia () must be used on the server and cannot be used in a local file. Otherwise you will get an error.

Getusermedia () can be used with other JavaScript class libraries:

• Webcam Toy is a photo-like APP that uses WebGL to add strange effects to your photos that can be shared and saved locally.
• Facekat is a face-tracking game that uses the headtrackr.js.
• ASCII Camera uses the Ganvas API to generate ASCII photos.

Constraints

has been implemented by Chrome, FireFox, and Opera. These can be used to set the video resolution obtained by Getusermedia () and Rtcpeerconnection Addstream (). The goal is to implement other constraints, such as the face mode (front and rear camera), frame rate, height, and width, using the Applyconstraints () method.

In
https://webrtc.github.io/samples/src/content/getusermedia/resolution/ 
There is an example.

A difficult problem to solve: in the browser, a tab in the Getusermedia settings constraint affects the tabs that are subsequently opened. An error message is given when setting a value that is not allowed:

navigator.getUserMedia error:NavigatorUserMediaError {code: 1, PERMISSION_DENIED: 1}

Screen and label capture

The Chrome app also makes it possible to share a video tag in a single browser tag, or the entire desktop through chrome.tabcapture and Chrome.desktopcapture APIs. Here you can find an example of a desktop capture. For screenshots of video, code and more information see here.

TODO using Csreen capture as a source of mediastream in Chrome is also possible, where chromemediasource constraints are used, see Demo here, notice that screen fetching requires HTTPS and should be tagged with command line, be WebRTC discuss the explanation.

Signaling: Session control, network and media information

WebRTC uses rtcpeerconnection to pass data streams between browsers, but it also requires a machine to coordinate communication to send control I information, which is called the signaling process. Signaling methods and protocols are not WebRTC defined: Signaling is not part of the rtcpeerconnection API.

Instead, developers of WebRTC apps can choose any message protocol they like, such as SIP or XMPP, or any full-duplex communication channel. This apprtc.appspot.com example uses the XHR and Channel APIs as the signaling. This codelab uses the Socket.io, which is a node server.

Signaling is used to exchange three kinds of information:

• Connection control information: initializing or closing the connection report error.
• Network configuration: For the external network, our computer's IP address and port?
• Multimedia data: What codec is used and what information can the browser process?

This information must be exchanged all through signaling before the peer-to data transfer.

For example, suppose Alice wants to communicate with Bob. This is the example that shows the signaling information in this process. This code assumes the existence of certain signals, which are created in the Createsignaling () method. Also notice that there are prefixes on Chrome and Opera.

var signalingchannel = Createsignalingchannel (); var pc;var configuration = ...;//Run Start (true) to initiate a callfuncti On Start (Iscaller) {PC = new Rtcpeerconnection (configuration);Send any ice candidates to the other peerPc.onicecandidate = function (evt) {Signalingchannel.send (json.stringify ({"Candidate": Evt.candidate});};Once remote stream arrives, show it in the remote video elementPc.onaddstream = function (evt) {REMOTEVIEW.SRC = Url.createobjecturl (Evt.stream);};Get the local stream, show it in the local video element and send itNavigator.getusermedia ({"Audio": True, "video": True}, Function (stream) {SELFVIEW.SRC = Url.createobjecturl (stream);Pc.addstream (stream); if (iscaller)  Pc.createoffer (gotdescription);  Pc.createanswer (pc.remotedescription, gotdescription);  Pc.setlocaldescription (desc); }}); Signalingchannel.onmessage = function (evt) { if (!pc)  start (false); Span class= "Hljs-code" > var signal = Json.parse (evt.data);  if (SIGNAL.SDP)  Pc.setremotedescription (new rtcsessiondescription (SIGNAL.SDP));  Else Pc.addicecandidate (new Rtcicecandidate (Signal.candidate));};  

First, Alice and Bob Exchange network information. ("Finding candidates" means looking for a network port using the ICE frame)

1. Alice uses the onicecandidate handle to create a rtcpeerconnection.
2. When a network handler is available, the handle runs.
3. Alice sends the serialized data to Bob, either way: WebSocket or otherwise.
4. When Bob gets the data from Alice, call Addicecandidate to add the remote node description.

WebRTC clients (known as peers, such as Alice and BOB) also need to exchange local and remote audio and video media information, such as protocols and encoders used. TODO signaling to exchange information.

1. Alice executes the Rtcpeerconnection Createoffer () method, which is described by the rtcsessiondescription callback: Alice local session.
2. In this callback, Alice sets the local description by Setlocationdescription () and then sends Bob the stage description through their channel. Note that Rtcpeerconnection will not start sending data until Setremotedescription is called: it is determined here.
3. Bob sets the message Alice sends over as a description of the remote information, through the Setremotedescription () method.
4. Bob runs Rtcpeerconnection's Createanswer () method, and the local session can be adapted to her by describing the information obtained from Alice. This createanswer () callback is sent to Alice by a rtcsessiondescription callback: Bob sets a local description.
5. When Alice gets the local description of Bob, it sets this description information through Setremotedescription.
6. Start communication.

The Rtcsessiondescription object conforms to the SDP, an SDP similar to the following:
```
V=0
o=-3883943731 1 in IP4 127.0.0.1
s=
T=0 0
A=group:bundle Audio Video
M=audio 1 RTP/SAVPF 103 104 0 8 106 105 13 126

// ...

a=ssrc:2223794119 label:h4fjnmzxy3dpigq7hxuctlb4wlllerhnfxh810
```

The collection and exchange of network and media information can be done simultaneously, but both processes must be completed before the audio and video streams between peers can begin.

The collection and exchange of network information can be done at the same time, but it needs to be completed before sending audio and video.

This offer/andwer is called JSEP (JavaScript Session establishment Protocol) and has a good WebRTC implementation explanation in [here].

Once the signaling is complete, data can be transferred directly from end to end. If it fails, it is forwarded through the Mediation Server Relay service. Streaming is the work of rtcpeerconnection.

Rtcpeerconnection

Rtcpeerconnection is part of the WebRTC, which is a stable and efficient handle to the end-to-end transfer of data.

Below is a WebRTC architecture that shows the role of Rtcpeerconnection, as you can see, the green part and the complexity!

From the JavaScript point of view, the main need to understand from the chart rtcpeerconnection to the developer to block the underlying complex things. WebRTC uses protocols and decoders to do a lot of work to make real-time communication possible, even in the case of unreliable networks:

• Packet packet Drop hiding
• Echo Cancellation
• Broadband Adaptive
• Dynamic jitter buffering
• TODO Automatic Gain Control
• Noise reduction
• Picture Erase

The code of the signaling shows a simplified example from the point of view. Here are two of the two applications that work with WebRTC, the first being an example of a simple rtcpeerconnection; the second is a fully operable video client.

Rtcpeerconnection without a server

The following code is from the Demo here. Includes local and remote rtcpeerconnection on the Web page. This is no use, the call and the call are all on the same page, but it does make it clearer to understand the rtcpeerconnection API, because the rtcpeerconnection of this interface can exchange information directly without signaling.

Other questions: TODO see http://www.w3.org/TR/webrtc/#constraints for more information.

In the example here, PC1 represents the local, PC2 represents the remote.

Local

1. Creates a new rtcpeerconnection add Getusermedia gets to the data stream.

1. Create an offer and set the local description of the PC1 as and remote PC2. This can be done directly without using signaling, because both local and remote are on one interface.

pc1.createOffer(gotDescription1);//...function gotDescription1(desc){  pc1.setLocalDescription(desc);  trace("Offer from pc1 \n" + desc.sdp);  pc2.setRemoteDescription(desc);  pc2.createAnswer(gotDescription2);}

Remote

Create the PC2, add the PC1 stream, and display it in the video node:

pc2 = new webkitRTCPeerConnection(servers);pc2.onaddstream = gotRemoteStream;//...function gotRemoteStream(e){  vid2.src = URL.createObjectURL(e.stream);}

Rtcpeerconnection and servers

In the real world, WebRTC needs servers, but when it's simple, the following may be true:

• You can find each other by user name.
• WebRTC clients to Exchange network information.
• Exchange media data information, such as resolution and video formats, between end-to-end.
• WebRTC the client for NAT and firewall penetration.

In other words, WebRTC needs four types of service information.

• User discovery and communication.
• Signaling.
• nat/firewall penetrates.
• When end-to-end communication fails, the Relay service is used for communication.

NAT penetration, end-to-end connectivity, user discovery, and the creation of signaling servers are beyond the scope of this article. What I want to say is that the STUN protocol and its extended ICE ice framework can use rtcpeerconnection to handle NAT and other network changes.

ICE is a connected end-to-end framework, such as a video client. Initially, ICE tried to connect directly, using UDP for less latency, and in the process, the STUN service had a simple task: to get a device behind the firewall to find the public IP and port. Google has two STUN servers, one of which is used in a apprtc.appspot.com example.

If UDP fails, ICE tries TCP: First HTTP, then HTTPS. If the direct connection fails-especially because the enterprise's NAT and firewall--ice use a relay relay TURN server. In other words, ICE first uses UDP's STUN server to connect directly to the peer, and if it fails, back to the TURN server. This expresses the process of looking for networks and ports.

WebRTC Engineer Jastin provides more information about ICE, STUN and TURN at Google's IO conference in 2013. (There are examples of TURN and STUN on the slide.)

A simple Chat client

The following signaling server uses the: https://apprtc.appspot.com.

If you think it's a bit confusing, you might prefer our WebRTC codelab. This guide describes how to build a complete video chat application step-by-step, including using Socket.io to build a simple signaling server.

Apprtc.appspot.com is a great place to try WebRTC, where signaling and nat/firewall are used as STUN servers. This application uses Adapter.js to handle the implementation of different rtcpeerconnection and Getusermedia ().

This code is deliberately written a little longer output log: View the console to understand the order of event execution. A detailed code is given below.

Next

This demo starts with the Initialize () function:

function initialize() {    console.log("Initializing; room=99688636.");    card = document.getElementById("card");    localVideo = document.getElementById("localVideo");    miniVideo = document.getElementById("miniVideo"); remoteVideo = document.getElementById("remoteVideo"); resetStatus(); openChannel(‘AHRlWrqvgCpvbd9B-Gl5vZ2F1BlpwFv0xBUwRgLF/* ...*/‘); doGetUserMedia();}

Note that the room and token variables used by Openchannel () are provided by Google APP itself: look here and see what values have been added to this library.

This code initializes the video variable to show the local webcam video stream and the remote client stream. Resetstatus () Simple setup state information.

The Openchannel () function sets information between WebRTC clients.

function openChannel(channelToken) {  console.log("Opening channel.");  var channel = new goog.appengine.Channel(channelToken);  var handler = {    ‘onopen‘: onChannelOpened,    ‘onmessage‘: onChannelMessage,    ‘onerror‘: onChannelError,    ‘onclose‘: onChannelClosed  };  socket = channel.open(handler);}

For signaling, this Demo is using the Channel API, which makes it possible for JavaScript to communicate without polling between clients. (WebRTC signaling is described in detail above).

Use the channel API to create a tunnel like this:

1. The Client generates a unique ID.
2. Client A requests a data pipeline from the APP Engine to send this ID.
3. APP Engine requests a data pipeline for this ID and Token through the Channel API.
4. The APP returns this token to A.
5. The Client opens the Socket connection to monitor the server's pipeline.

Send a message like this:

1. Client B sends a POST request to the APP Engine.
2. APP Engine passes a request to the Channel.
3. The Channel carries information to Client A.
4. The Accept message callback for Client A is called.

WebRTC Past Life