JavaScript binary Data Processing learning--NODEJS environment and browser environment analysis separately

Before using JavaScript mainly to deal with the regular numbers, strings, array objects and other data, basically did not try to use JavaScript to deal with binary data blocks, recent projects involved in this aspect, took a period of time to learn the next API, in this summary. First, the browser is not actively read the local file permissions, so the JavaScript processing binary data ability to learn, should be run on the server side of the Nodejs look. Buffer in the NodejsTo facilitate the processing of binary data, Nodejs specifically encapsulates a buffer module. Document Address: http://nodejs.cn/doc/node/ The Buffer.htmlbuffer module's underlying API can initialize a buffer object in the following way, passing in parameter 50, so that a 50byte,400bit area is requested in memory, and the size of this area cannot be changed once the application is made, and then through the buffer object's fil L method to populate this area of memory. The incoming "ABC" string, which is decoded as binary data by default in UTF8 encoding, is stored in the corresponding location in the memory area. As you can see, the buffer object is displayed in the Debug tool as a Uint8array array of length 50, and this Uint8array object is explained behind what it is. Each of the arrays is stored with an unsigned 8-bit integer, which is a byte, 0~255. Buffer Object Initialization method also includes: Buffer object also have some comparison, splicing, copy, fill, and so on, in the above document has Nodejs FS module not only manual initialization can get a buffer object, through the FS module to read the file, You can also get a buffer object. Dnf.exe is a 31733096-byte file stored on the hard disk
By reading this file through the Readfilesync method of the FS module, you can get a buffer object with this buffer object After running, the result is as follows: so that the Dnf.exe file is read into memory, Nodejs has a buffer object, which occupies a memory space of 31733096byte. The buffer object can be written to the local file system, either through the network to the remote machine, or converted to a string to do more, or no processing. In the example above, we can write a new file in the current directory with a secondary buffer after running: this example can work, but things will not always be so simple. My computer's memory size is 8G, what happens when I want to copy a 12G file with this code? When we read the file in the first step, we need to create a buffer object that takes up 12G of memory space, which is obviously not working and the memory will explode. What about that? Nodejs Stream What is stream, why should there be a stream in the previous section of the example, we encounter the memory is not enough to use the case, obviously we needThe data into a small piece of a small piece, a piece of the memory to be processed, so that the memory will not be blown off~ Here I would like to compare, easy to understand: CPUEquivalent to a WorkersWorkers need to operate tools to heat water (CPU needs to run code to perform calculations) HDDEquivalent to a PoolThe pool can be stored in a large capacity, but can not be directly heated in the water tank, the need to put water into the pot (hard disk can store data, large capacity, but not directly in the hard disk using data to perform calculations, need to read the data into memory) MemoryEquivalent to a mouthful PotThe pot can be filled with water and heated, but the capacity is small (memory can be stored in the data for calculation, but the capacity is small) Heating-Import water (calculate-import data)So the problem we have in the above section is: There is too much water in the pond, it spills out once in the pan and cannot be heated. So we took a measure: to connect a pipe from the sink to the pot, the rootPipeline (stream. Readable class)The water can be imported from the pond. The tube is closed at first, and we can turn the switch on (Bind event: ' Data ', which is triggered immediately after the event is bound) or suspend import (Readable.pause ()) to resume the import (Readable.resume () ) can also be imported manually (Readable.read ()) Finish Heating-export water (calculated-export data)When the imported pot of water is cooked, it is necessary to pour out the pot of water to handle the next pot, then connect one from the potPipeline (stream. Writable Class)To another container (possibly another area of the hard drive, or another remote machine), like the readstream above, the pipe can be exported to another place (Writable.write) do not calculate only transitWhen the pot is just a relay function, you can put the import tube into the export tube (Readable.pipe (destination[, Options]) can also be separated from them (readable.unpipe ([destination])) you can export or import a streamSome pipelines can be either imported or exported (stream. Transform) stream module basic usage the underlying API is, of course, the Official document: http://nodejs.cn/doc/node/stream.html rewrite the preceding example:, or simpler, or the content in Readstream is the buffer object by default, the character encoding is set before reading, and the string is fetched when read. The stream's application is used where the stream is. The request object of the file Systemhttp module and the data transfer of the Response object Net module when we need to continuously write binary data to an address or read binary data from an address We can also implement our own read-write stream according to the API provided by Nodejs. When we use Nodejs as an HTTP proxy, for the client, Nodejs is the server, the response object is a writestream, and for the target server, Nodejs is the client, the Res object is a readstream, so the direct res.pipe (response) forwards the target server's data to the client. When the HTTP server writes the file to the client, it can also direct the readstream of the file to the response in the browser Arraybuffer above the buffer object in the Nodejs in the WS console to show the Uint8array, For this I checked, found that es5 in fact there is a binary processing API, but only in the browser side is not much, so before the attention. Here is an explanation of Arraybuffer in the MSDN documentation:

The ArrayBuffer object represents the raw data buffer used to store data for different typed arrays. cannot read or write directly to ArrayBuffer , but you can pass it to a typed array or DataView object to interpret the original buffer. You can use ArrayBuffer to store any type of data (or mixed-type data).

, store binary data, the definition of length in bytes to count, it looks like the NODEJS environment buffer is very similar? But look at the API this thing does not directly write and read data, so, then look down! arraybuffer View and edit views--typed arrays and DataViewWhat is the typed array just said Arraybuffer, looks a bit like the buffer object in the Nodejs, but does not have the buffer object's read-write API, this is not very scientific, so definitely need a way to manipulate it. In fact, the Arraybuffer object is a static binary data store: 00000001 00000010 000000011 00000100 When you need to write or manipulate this memory area, if you open a few APIs directly to write 01010101 of this data, I think your heart must be broken. So ES5 provides some views to represent and manipulate these binary data, such as converting each 8-bit binary number into a 10-binary, 8-bit unsigned integer, in a special array, Uint8array [1,2,3,4] The prototype of this array does not point to Array.prototype, which does not have the operations of JavaScript ordinary arrays, and all elements inside this array must be an 0~255 integer. When we manipulate Arr[3]=1, Uint8array becomes [1,2,3,1] and Arraybuffer's content becomes 00000001 00000010 000000011 00000001 Note that Arraybuffer is the raw data, you can create several different types of data views based on the original data, and when any one view changes the original data, the data seen by the other views will change, as shown in the example below.  Does this uint8array look familiar? There seems to be some kind of connection between the buffer in Nodejs and the Uint8array in the browser environment! Typed array types There are many types of views: The data types in each array are different, and the names can understand the type of data they store, such as Float32array, which converts every 32 binary numbers into an unsigned floating-point number, which is stored in this typed array. Typed array Operations API initialization method: attribute interpretation: How to: Dataviewdataview is an object that can get Arraybuffer data and edit Arraybuffer data, taking a fragment from Arraybuffer. Provides methods to get or edit the data in these fragments, instead of putting the data into an array structure like a typed array. Method

The following table lists the methods for DataView objects.

Method	Describe
GetInt8 method	Gets the Int8 value at the specified byte offset position relative to the beginning of the view.
GetUint8 Method (DataView)	Gets the Uint8 value at the specified byte offset position relative to the beginning of the view.
GetInt16 Method (DataView)	Gets the Int16 value at the specified byte offset position relative to the beginning of the view.
GetUint16 Method (DataView)	Gets the Uint16 value at the specified byte offset position relative to the beginning of the view.
GetInt32 Method (DataView)	Gets the Int32 value at the specified byte offset position relative to the beginning of the view.
GetUint32 Method (DataView)	Gets the Uint32 value at the specified byte offset position relative to the beginning of the view.
GetFloat32 Method (DataView)	Gets the Float32 value at the specified byte offset position relative to the beginning of the view.
GetFloat64 Method (DataView)	Gets the Float64 value at the specified byte offset position relative to the beginning of the view.
SetInt8 Method (DataView)	Stores the Int8 value at the specified byte offset position relative to the beginning of the view.
SetUint8 Method (DataView)	Stores the Uint8 value at the specified byte offset position relative to the beginning of the view.
SetInt16 Method (DataView)	Stores the Int16 value at the specified byte offset position relative to the beginning of the view.
SetUint16 Method (DataView)	Stores the Uint16 value at the specified byte offset position relative to the beginning of the view.
SetInt32 Method (DataView)	Stores the Int32 value at the specified byte offset position relative to the beginning of the view.
SetUint32 Method (DataView)	Stores the Uint32 value at the specified byte offset position relative to the beginning of the view.
SetFloat32 Method (DataView)	Stores the Float32 value at the specified byte offset position relative to the beginning of the view.
SetFloat64 Method (DataView)	Stores the Float64 value at the specified byte offset position relative to the beginning of the view.

a special typed arrayUint8clampedarray and Uint8array are similar, the only difference is that when the value of the insertion array is not between 0~255, the value policy is different, Uint8array is to take the input number modulo, And Uint8clampedarray is greater than 255 to take 255, less than 0 0uint8clampedarray in the canvas getimagedata when used, because the color value #ffffff, just 3 bytes to represent, more appropriate ~ Characteristics to be aware of when typed arrays are understood byte value range when initializing

The length of the Arraybuffer initialization is in bytes (byte) instead of bit (bit),

When a typed array is initialized through a buffer, the offerset represents the offset in the buffer, and

The length of the typed array initialization is the length of the typed array, which is not the number of bytes in the buffer to be used for this initialization, but rather how many elements in the typed array.

Uint32array = new Uint32array (buffer, byteoffset, length);

So for example uint32array the bytes actually used in the buffer should be from Byteoffset to byteoffset+4*length bytes Instead of byteoffset bytes from the first byte to the Byteoffset+length bytes


Typed arrays are just views, raw data exists in Arraybuffer, raw data changes, all views change

Use instance 1.ajax in the browser to receive Arraybuffer data

varreq =NewXMLHttpRequest (); Req.open (' GET ', ' http://www.example.com '); Req.responsetype= "Arraybuffer";    Req.send (); Req.onreadystatechange=function () {        if(Req.readystate = = 4) {            varBuffer =Req.response; varDataView =NewDataView (buffer); varINTs =NewUint8array (buffer.bytelength);  for(vari = 0; i < ints.length; i++) {Ints[i]=dataview.getuint8 (i); } alert (ints[10]); }    }

2.websocket also supports arraybuffer type of data reception

var New WebSocket (SUri);  

3. Used to create a BLOB object on the front end, file object download or upload 4. Although the browser does not have permission to read local files, but has permission to write to the local file, writing to the file locally may use 5. Video audio playback (write files and play AV audio I did not verify)later I will learn these forms of use, then do a summary ~The relationship between buffer in Nodejs and Uint8array in the browser is visible, and the buffer implemented in Nodejs is actually a subclass of the Uint8arr array, is a class encapsulated by Nodejs to further enhance the binary data processing capabilities of JavaScript. In fact, arraybuffer and typed arrays are not unique in the browser environment, they are content within the ES5 specification, and can be used in nodejs environments, for example: ———————————————— end ——————————————— above, If there are errors, welcome treatise!

JavaScript binary Data Processing learning--NODEJS environment and browser environment analysis separately