Capturing audio and video in HTML5

Brief introduction

Audio/video capture has long been the "Holy Grail" in web development. Over the years, we've always relied on browser plugins (Flash or Silverlight) to achieve this. Come and see it!

Now it's HTML5 's turn to do it. It may not seem obvious, but the rise of HTML5 has led to a surge in access to hardware devices. Geolocation (GPS), Orientation API (Accelerometer), WebGL (GPU), and Web Audio API (video hardware) are good examples. These features are powerful and showcase advanced JavaScript APIs based on the underlying hardware capabilities of the system.

This tutorial introduces a new API navigator.getUserMedia() that allows a network app to access the user's camera and microphone.

The history of Getusermedia ()

If you don't know, getUserMedia() the history is an interesting story.

Several variants of the Media Capture API have appeared in the past few years. Many people realize that it is necessary to be able to access local devices on the network, but it is up to everyone to develop a new specification. The situation was so chaotic that the company finally decided to set up a working group. They have only one purpose: to clarify the chaotic situation! The device API policy (DAP) team is responsible for unifying and standardizing the excess proposals.

I'll try to summarize what happened in 2011 ...

1th Round: HTML Media capture

HTML Media capture is the first step in DAP's standardization of network media capture. The method is overloaded <input type="file"> and accept adds a new value to the parameter.

If you want users to take their own snapshots from a webcam, you can use capture=camera :

<input type= "File" accept= "Image/*;capture=camera" >

　　

Recording video or audio is similar:

<input type= "File" accept= "Video/*;capture=camcorder" ><input type= "file" accept= "audio/*;capture= Microphone ">

　　

Pretty good, huh? It can re-use the file input, which I particularly like. This is semantically meaningful. The disadvantage of this particular API is the inability to handle instant effects such as rendering live webcam data to <canvas> and applying WebGL filters. HTML Media capture only lets you record media files or take snapshots in a timely manner.

Support :

• Android 3.0 Browser-an example of the first implementation. Watch this video to see how it's actually used.
• Android Chrome Browser (0.16)

Unless you are using one of these mobile browsers, I recommend that you do not use the API. Suppliers are turning getUserMedia() . It is unlikely that anyone else will be able to implement HTML media capture for long.

2nd round: Device elements

Many people think that the limitations of HTML media capture are too great, so a new specification comes into being that can support any type of (future) device. Not surprisingly, the design requires new <device> elements, namely getUserMedia() the predecessor.

Opera is <device> one of the first browsers to create the initial implementation of video capture based on elements. Shortly after (exactly the same day), WhatWG decided to repeal the <device> tag to support navigator.getUserMedia() The new JavaScript API called. A week later, the new version of Opera was added with support for the updated getUserMedia() specification. At the end of the year, Microsoft joined the ranks, releasing the IE9 lab to support the new specification.

<device>The effect is as follows:

<device type= "Media" onchange= "Update (this.data)" ></device><video autoplay></video>< script>  function Update (stream) {    document.queryselector (' video '). src = stream.url;  } </script>

　　

Support :

Unfortunately, none of the published browsers have ever been included <device> . I guess this is an API that you don't have to worry about. But <device> there are two big advantages: the semantic side, and the ease with which you can expand, not just the audio/video devices.

Now breathe a deep breath. This thing is fast!

3rd round: WebRTC

<device>The elements eventually disappeared like the dodo.

Relying heavily on WebRTC (network instant Messaging), the pace of finding the right capture API in recent months has accelerated a lot. The specification is supervised by the WebRTC Working Group. Google, Opera, Mozilla and other companies are currently working on implementing the API in their own browsers.

getUserMedia()is related to WebRTC because it is the gateway to this set of APIs. It provides the means to access the user's local camera/microphone media stream.

Support :

In Chrome browser 18.0.1008 and later versions, WebRTC can be about:flags enabled under.

Getting Started with

Using navigator.getUserMedia() , we finally achieved access to the webcam and microphone input without plugins. The camera access rights are now related to the call, not to the installation. It is embedded directly in the browser. Are you feeling excited?

Enable

getUserMedia()The API is also new and only Google and Opera have added it to the developer version. In Chrome 18 and later, this API can be enabled through access about:flags .

Enabled in the Chrome browser about:flags page getUserMedia() .

For Opera, download an experimental version of Android and desktop computers.

function detection

Functional detection is simply to check for presence navigator.getUserMedia :

function Hasgetusermedia () {  //Note:opera builds is unprefixed.  Return!! (Navigator.getusermedia | | navigator.webkitgetusermedia | |            Navigator.mozgetusermedia | | Navigator.msgetusermedia);} if (Hasgetusermedia ()) {  //good to go!} else {  alert (' Getusermedia () was not supported in your browser ');}

　　

To get access to the input device:

To use a webcam or microphone, we need to request permission. getUserMedia()The first parameter is used to specify the type of media you want to access. For example, if you want to request access to a webcam, the first parameter should be "video" . To use both the microphone and the camera, pass "video, audio" :

<video autoplay></video><script>  var onfailsohard=function (e) {    console.log (' reeeejected! ', e);  };  Not showing vendor prefixes.  Navigator.getusermedia (' video, audio ', function (localmediastream) {    var video = document.queryselector (' video ');    video.src = window. Url.createobjecturl (localmediastream);    Note:onloadedmetadata doesn ' t fire in Chrome if using it with Getusermedia.    See crbug.com/110938.    Video.onloadedmetadata = function (e) {      //ready to go. Do some stuff    };  }, Onfailsohard);</script>

　　

Well, what the hell is going on here? Media capture is a great example of the collaboration of a variety of new HTML5 APIs. There are other HTML elements involved in collaboration, such as <audio> and <video> . Note that we are not setting src properties or <video> adding elements to the element <source> . Instead of feeding the video to the URL of the media file, we feed into the Blob URL obtained from the object representing the webcam LocalMediaStream .

I'll also <video> set it to autoplay , otherwise it will stop at the first frame. Adding controls can also achieve the results you expect.

Please note : There is an error in the Chrome browser that simply passes "audio" invalid: crbug.com/112367. I can't use it properly in Opera either <audio> .

Opera and Chrome are implementing different versions of the specification. This leads to a more "challenging" use than expected.

in the Chrome browser :

This code snippet applies to Chrome 18 and later ( about:flags enabled in):

Navigator.webkitgetusermedia (' Audio, video ', function (localmediastream) {  var video = Document.queryselector (' Video ');  VIDEO.SRC = Window.webkitURL.createObjectURL (Localmediastream);}, Onfailsohard);

　　

in Opera :

The Opera developer version does not support an updated version of this specification. This code snippet applies to Opera:

Navigator.getusermedia ({audio:true, video:true}, function (Localmediastream) {  video.src = Localmediastream;}, Onfailsohard);

　　

The key difference is that:

• getUserMedia()are not prefixed.
• Object is passed as the first parameter instead of a list of strings.
• will be video.src set directly to the LocalMediaStream object instead of the Blob URL. As far as I know, Opera will eventually update this setting to require Blob URLs instead.

for both :

If you want to be common across browsers (but this is a problem), try the following:

var video = document.queryselector (' video '), if (Navigator.getusermedia) {  Navigator.getusermedia ({audio:true, Video:true}, function (stream) {    video.src = stream;  }, Onfailsohard);} else if (Navigator.webkitgetusermedia) {  Navigator.webkitgetusermedia (' Audio, video ', function (stream) {    video.src = Window.webkitURL.createObjectURL (stream);  }, Onfailsohard);} else {  video.src = ' SOMEVIDEO.WEBM ';//Fallback.}

　　

Be sure to check out the GUM Shield of Mike Taylor and Mike Robinson. It is a good way to "standardize" inconsistencies between browser implementations.

Safety

In the future, the browser getUserMedia() may pop up the InfoBar when it calls, giving the user the option to grant or deny access to their camera/microphone. Unfortunately, the specification is very weak in terms of security. Currently, no browser has implemented the permission bar.

Provide fallback

For getUserMedia() users who cannot get support, if the API is not supported and/or fails for some reason, you can choose to fallback to an existing video file:

Not showing vendor prefixes or code that works cross-browser:function fallback (e) {  video.src = ' FALLBACKVIDEO.WEBM ‘;} Function success (stream) {  video.src = window. Url.createobjecturl (stream);} if (!navigator.getusermedia) {  fallback ();} else {  Navigator.getusermedia ({video:true}, success, fallback);}

<canvas>The API's ctx.drawImage(video, 0, 0) method can easily <video> draw frames to the <canvas> top. Of course, since we've got getUserMedia() video input, it's easy to create a kiosk app with Instant Video.

<video autoplay></video><canvas style= "Display:none;" ></canvas>var video = document.queryselector (' video '); var canvas = document.queryselector (' canvas '); var ctx = Canvas.getcontext (' 2d '); var localmediastream = Null;function Snapshot () {  if (localmediastream) {    Ctx.drawimage (video, 0, 0);    "IMAGE/WEBP" works in Chrome 18. In the other browsers, this would fall back to Image/png.    Document.queryselector (' img '). src = Canvas.todataurl (' Image/webp ');}  } Video.addeventlistener (' Click ', snapshot, false);//not showing vendor prefixes or code that works Cross-browser.navigato R.getusermedia ({video:true}, function (stream) {  video.src = window. Url.createobjecturl (stream);  Localmediastream = stream;}, Onfailsohard);

　　

Summarize

On the whole, device access on the network has always been a big problem. Many people have tried, but no one has succeeded. Most of the early ideas have never been dominated outside the proprietary environment and have never been widely adopted.

The real problem is that the security model of the network is vastly different from the local system. For example, I might not want any website to have access to my camera, but this problem is difficult to solve.

Bridging frameworks such as PHONEGAP can help overcome this limitation, but this temporary solution is far from enough for the underlying problem. In order for the network application to have a higher power than desktop computer applications, we need to be able to access local devices.

getUserMedia()is just the first wave of access to the new device type. I hope to see more in the near future.

Other resources

• Specifications
• Brousse Lausen (Bruce Lawson) 's Html5doctor article
• Brousse Lausen's dev.opera.com article

Demonstrate

• Real-time Photo booth
• Paul Niff (Paul Neave) 's WebGL camera effect
• Snapster
• Live video in WebGL

Excerpt from: https://www.html5rocks.com/zh/tutorials/getusermedia/intro/

Capturing audio and video in HTML5