Nodejs in the buffer module detailed _node.js

First, the opening analysis

Buffer buffers, which means "temporary storage area", is a memory that temporarily holds input and output data.

The JS language itself has only string data types and no binary data types, so Nodejs provides a global constructor buffer that is equivalent to string to provide operations on binary data. In addition to being able to read an instance of the file to get buffer, it can be constructed directly, for example:

Copy Code code as follows:

var buffer = new buffer ([0x68, 0x65, 0x6c, 0x6c, 0x6f]);

Buffer, like a string, can read bytes at the specified position in the form of [index], in addition to the length of a byte that can be obtained with the. Length property, for example:

Copy Code code as follows:

BUFFER[0]; 0x68;

Buffer and string can be converted to each other, for example, the binary data can be converted to a string using the specified encoding:

Copy Code code as follows:

var str = buffer.tostring ("Utf-8"); Hello

Converts a string to binary data under the specified encoding:

Copy Code code as follows:

var buffer= new buffer ("Hello", "utf-8"); <buffer 6c 6c 6f>

A little bit different:

There is an important difference between buffer and string. The string is read-only, and any modification to the string gets a new string, and the original string remains unchanged.

As for buffer, it is more like an array of C languages that can be manipulated by pointers. For example, you can modify the byte of a location directly by using the [index] method.

--------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------- -------

Instead of returning a new buffer, the slice method is more like returning a pointer to a position in the middle of the original buffer, as shown below.

[0x68, 0x65, 0x6c, 0x6c, 0x6f]
^           ^
| |
Bin Bin.slice (2)
Therefore, the modification of the buffer returned by the slice method is used for the original buffer, for example:

Copy Code code as follows:

var buffer= new Buffer ([0x68, 0x65, 0x6c, 0x6c, 0x6f]);
var sub = Bin.slice (2);
Sub[0] = 0x65;
Console.log (buffer); <buffer 6c 6f>

If you want to copy a buffer, you first create a new buffer and copy the data in the original buffer by using the. Copy method.

This is similar to applying for a new piece of memory and copying the data in the existing memory. Here is an example.

Copy Code code as follows:

var buffer= new Buffer ([0x68, 0x65, 0x6c, 0x6c, 0x6f]);
var dup = new Buffer (bin.length);
Buffer.copy (DUP);
Dup[0] = 0x48;
Console.log (buffer); <buffer 6c 6c 6f>
Console.log (DUP); <buffer 6c 6f>

In a word, buffer extends the data processing ability of JS from string to any binary data.

The above simple lets everybody understand what is the buffer, below specifically says how uses and the concrete use scene.

Two, talk about buffer

JavaScript is very friendly to string processing, whether it is a wide byte or a single-byte string, and is considered a string. node in the need to deal with network protocols, operating databases, processing pictures, file Upload, and also need to deal with a large number of binary data, the string is far from meet these requirements, so buffer came into being.

Buffer structure

Buffer is a typical combination of JavaScript and C + + modules, performance-related parts are implemented in C + +, the non-performance related parts are implemented in JavaScript.

node is loaded into memory when the process starts and puts it into the global object, so there is no need to require

Buffer object: Similar to an array whose elements are binary digits of 16.

Buffer memory allocation

The memory allocation of the buffer object is not in the V8 heap memory, which implements the memory application at node's C + + level.

In order to efficiently use the memory of application, the slab allocation mechanism is adopted in node, slab is a dynamic memory management mechanism, and various *nix operating systems are applied. There are three states of slab:

(1) Full: Fully allocated state

(2) Partial: Partial distribution status

(3) Empty: No assigned state

Conversion of buffer

The buffer object can be converted to and from the string, supported by the following encoding types:

ASCII, UTF-8, Utf-16le/ucs-2, Base64, Binary, Hex

String buffer

New Buffer (str, [encoding]), default UTF-8
Buf.write (string, [offset], [length], [encoding])

Buffer Turn string

Buf.tostring ([encoding], [start], [end])

Type of encoding not supported by buffer

Buffer.isencoding (encoding) to determine whether to support

Iconv-lite: pure JavaScript implementation, lighter, better performance without C + + to JavaScript conversion

Iconv: Call C + + Libiconv library complete

The concatenation of the buffer

Note "Res.on" (' Data ', function (chunk) {}), where the parameter chunk is a buffer object, directly with + stitching will automatically convert to a string, for wide-byte characters may lead to garbled generation,

Workaround:

(1) through the Setencoding () method in the readable flow, this method can let the data event pass no longer is the buffer object, but is the encoded string, its inside uses the Stringencoder module.

(2) The buffer object is staged into the array, and finally the encoding is converted into a string output after assembling into a large buffer.

Buffer is widely used in file I/O and network I/O, its performance is very important, much higher than ordinary string performance.

The use of buffer, in addition to the conversion of the string has a performance loss, in the file reading, there is a highwatermark setting is critical to the performance impact.

A,highwatermark settings have a certain impact on the allocation and use of buffer memory.

B, highwatermark settings are too small, which can cause too many system calls.

When to use buffer, when not to use------pure JavaScript to support Unicode code is not very support for binary, when the TCP stream or file flow is resolved, the processing flow is necessary, we save the Utf-8 string, 2 and so on in other formats, we have to use "Buffer".

Third, the example introduces

Copy Code code as follows:

var buf = new Buffer ("This was text concat test!"), str = "This is Text concat test!";
Console.time ("Buffer concat test!");
var list = [];
var len = 100000 * buf.length;
for (Var i=0;i<100000;i++) {
List.push (BUF);
Len + + buf.length;
}
var S1 = buffer.concat (list, Len). toString ();
Console.timeend ("Buffer concat test!");
Console.time ("String concat test!");
var list = [];
for (var i = 100000 i >= 0; i--) {
List.push (str);
}
var s2 = list.join ("");
Console.timeend ("String concat test!");

The following are the results of the operation:

The read speed is sure that the string is faster, and the buffer requires the operation of ToString (). So when we save the string, we should use string or string, even if the large string concatenation string is not slower than the buffer.

So when do we need to use buffer? When we have no choice, we have to use it when we save the Utf-8 string, 2, and so on in other formats.

Four, sum up

(1) JavaScript is suitable for processing Unicode encoded data, but processing binary data is not user-friendly.
(2) The processing of a eight-bit byte stream is necessary when handling TCP streams or file systems.
(3), node has several methods for processing, creating, and consuming eight-bit byte streams.
(4) The original data is stored in a buffer instance, and a buffer is similar to an array of integers, but its memory is allocated outside the V8 stack. The size of a buffer cannot be changed.
(5), the encoding types of processing are: ASCII,UTF8,UTF16LE,UCS2 (utf16le alias), Base64,binary,hex.
(6), the buffer is a global element, direct new buffer () to get a buffer instance.