Web Audio Living

Summarize the Web site audio live program and the problems encountered.

Code: (GitHub, to be sorted)

Results: With opus Audio encoding, the Web Audio API plays, which can reach up to 100ms latency, high-quality, low-flow audio live.

Background: VDI (Virtual Desktop) h264 Web-site pre-research, after the H264 video live solution after the resolution of a delay has a high demand for audio live program (interactive, audio and video synchronization).

Premise: The FLEXVDI open source project supports audio only for PCM audio data that is uncompressed. And the effect is not good, either lag, or delay, traffic in 2~3mbps (depending on the size of the buffer).

Solution: On the spice server side of the audio using opus encoding, FLEXVDI playback channel to the Opus packet data, call the Opus JS decoding library decoded into PCM data, feed audiocontext to play.

Process brief: FLEXVDI Palyback channel receives opus audio data, calls Libopus.js decode to get PCM data, save to buffer. Create Scriptprocessornode, get PCM data from buffer inside the onaudioprocess function,

Fill the OutputBuffer by the channel and connect the Scriptprocessornode to the audiocontext.destination for playback. The specific code is described later or GitHub.

Opus Codec Interface Description:

Reference: http://opus-codec.org/docs/opus_api-1.2/index.html

One, below is I use Opus C library decoding opus Audio, and then play PCM data with Ffplay a demo, you can see how the Opus decoding interface is used:

#include <stdio.h> #include<stdlib.h>#include<string.h>#include"Opus.h" /*static void Int_to_char (Opus_uint32 i, unsigned char ch[4]) {ch[0] = i>>24;    CH[1] = (i>>16) &0xFF;    CH[2] = (i>>8) &0xFF; CH[3] = i&0xff;}*/ StaticOpus_uint32 Char_to_int (unsignedCharch[4]){    return((Opus_uint32) ch[0]<< -) | ((Opus_uint32) ch[1]<< -)         | ((Opus_uint32) ch[2]<<8) | (Opus_uint32) ch[3];} intMainintargcChar**argv) {Opus_int32 samplerate=0; intChannels =0, err =0, Len =0; intMax_payload_bytes = the; intMax_frame_size =48000*2; Opusdecoder* Dec =NULL; Samplerate= (Opus_int32) atol (argv[1]); Channels= Atoi (argv[2]); FILE* fin = fopen (argv[3],"RB"); FILE* Fout = fopen (argv[4],"wb+");  Short* out; unsignedChar* Fbytes, *data; //In = (short*) malloc (max_frame_size*channels*sizeof (short));     out= ( Short*)malloc(max_frame_size*channels*sizeof( Short)); /*we need to allocate for 16-bit PCM data, but we store it as unsigned char.*/fbytes= (unsignedChar*)malloc(max_frame_size*channels*sizeof( Short)); Data= (unsignedChar*)calloc(Max_payload_bytes,sizeof(unsignedChar)); Dec= Opus_decoder_create (samplerate, channels, &err); intNbytesread =0; Opus_uint64 Tot_out=0;  while(1) {unsignedCharch[4] = {0}; Nbytesread= Fread (CH,1,4, Fin); if(Nbytesread! =4)             Break; Len=char_to_int (CH); Nbytesread= Fread (data,1, Len, Fin); if(Nbytesread! =len) Break; Opus_int32 Output_samples=max_frame_size; Output_samples= Opus_decode (Dec, data, Len, out, Output_samples,0); inti;  for(i=0; i < output_samples*channels; i++)        {             Shorts; S= out[i]; fbytes[2*i]=s&0xFF; fbytes[2*i+1]= (s>>8) &0xFF; }        if(Fwrite (Fbytes,sizeof( Short) *channels, Output_samples, fout)! =(unsigned) output_samples) {fprintf (stderr,"Error writing.\n"); returnexit_failure; } tot_out+=Output_samples; } printf ("tot_out:%llu \ n", tot_out); return 0;}

This program for opus packets composed of files (simple Length+packet format) decoded after the PCM data, and then use Ffplay playback PCM data to see if it can play normally:

Ffplay-f f32le-ac 1-ar 48000 input_audio//Play float32 type PCM data

Ffplay-f s16le-ac 1-ar 48000 input_audio//Play short16 type PCM data

AC indicates the number of channels, AR represents the sample rate, and Input_audio is a PCM audio file.

Second, to obtain the PCM data file, the first to get Opus packet binary files, so here is how the browser to save the binary files to the local problem:

Reference code:

varSaveFile = (function(){        varA = Document.createelement ("a");        Document.body.appendChild (a); A.style= "Display:none"; return function(data, name) {varBlob =NewBlob ([data]); varURL =window.                Url.createobjecturl (BLOB); A.href=URL; A.download=name;                A.click (); Window.        Url.revokeobjecturl (URL); };} ()); SaveFile (data,' TEST.PCM ');

Description: First write the binary data to Typedarray, then use this buffer to construct the Blob object, generate the URL, and then use a tag to download the blob to local.

Third, the use of Audiocontext playback PCM audio data two scenarios:

(1) Realization of FLEXVDI

Reference: Https://github.com/flexVDI/spice-web-client

 functionPlay (buffer, Datatimestamp) {//Each of the data packet is a. Being left channel data and the second being right channel data (LR-LR-LR-LR. .)        //var audio = new Int16array (buffer);        varAudio =Newfloat32array (buffer); //We split the audio buffer in the channels. Float32array is the type required by Web Audio API        varleft =NewFloat32array (AUDIO.LENGTH/2); varright =NewFloat32array (AUDIO.LENGTH/2); varChannelcounter = 0; varAudiocontext = This. Audiocontext; varLen =audio.length;  for(vari = 0; I <Len;) {          //because the audio data spice gives us is a-bits signed int (32768) and we wont to get a float out of it (between-1. 0 and 1.0)Left[channelcounter] = audio[i++]/32768; Right[channelcounter]= audio[i++]/32768; Channelcounter++; }        varSource = audiocontext[' Createbuffersource '] ();//creates a sound source        varAudiobuffer = audiocontext[' Createbuffer ' (2, Channelcounter, This. Frequency); audiobuffer[' Getchanneldata ' (0) [' Set '] (left); audiobuffer[' Getchanneldata ' (1) [' Set '] (right); source[' Buffer '] =Audiobuffer; source[' Connect ' ( This. audiocontext[' Destination ']); source[' Start ' (0);}

Note: The short PCM data is saved in buffer, where the processing of timestamps is removed for simplicity, because Source.start (0) indicates immediate playback. If it is float type, you do not need to divide by 32768.

(2) Realization of WS-AUDIO-API

Reference: Https://github.com/Ivan-Feofanov/ws-audio-api

varBUFL =NewFloat32array ( This. config.codec.bufferSize);varBufR =NewFloat32array ( This. config.codec.bufferSize); This. Scriptnode = Audiocontext.createscriptprocessor ( This. config.codec.bufferSize, 0, 2);if(typeofAudioBuffer.prototype.copyToChannel = = = "function") {      This. scriptnode.onaudioprocess =function(e) {varBUF =E.outputbuffer;　　_this.process (BUFL, BufR); //get PCM data to BUFL, BufRBuf.copytochannel (BUFL, 0); Buf.copytochannel (BufR,1); };} Else {      This. scriptnode.onaudioprocess =function(e) {varBUF =E.outputbuffer;          _this.process (BUFL, BufR); Buf.getchanneldata (0). Set (BUFL); Buf.getchanneldata (1). Set (BufR); };} This. Scriptnode.connect (Audiocontext.destination);

Delay lag problem: Audiocontext Some browser default is 48000 sample rate, some browsers default is 44100 sample rate, if feed to audiocontext PCM data sampling rate mismatch, will produce delay and lag problem.

Web Audio Living