Detailed description of the Buffer module in NodeJS _ node. js

This article mainly introduces the Buffer module in NodeJS, which is very detailed. For more information, see I. Opening Analysis

The so-called Buffer refers to the "temporary storage zone", which is a memory block that temporarily stores input and output data.

The JS language only has the String data type and does not have the binary data type. Therefore, NodeJS provides a global constructor Buffer equivalent to the String type to operate binary data. In addition to the Buffer entity that can be read from the file, it can also be directly constructed, for example:

The Code is as follows:

Var buffer = new Buffer ([0x68, 0x65, 0x6c, 0x6c, 0x6f]);

Buffer is similar to a string. In addition to the. length attribute, you can use the [index] method to read the bytes at the specified position. For example:

The Code is as follows:

Buffer [0]; // 0x68;

Buffer and string can be converted to each other. For example, you can use the specified encoding to convert binary data to a string:

The Code is as follows:

Var str = buffer. toString ("UTF-8"); // hello

Convert the string to the binary data of the specified encoding:

The Code is as follows:

Var buffer = new Buffer ("hello", "UTF-8 ");//

A little difference:

There is an important difference between Buffer and string. The string is read-only, and any modification to the string is a new string, the original string remains unchanged.

As for the Buffer, it is more like a C language array that can be used for pointer operations. For example, you can use the [index] method to directly modify the bytes at a certain position.

Zookeeper -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

The slice method does not return a new Buffer, but returns a pointer to a position in the middle of the original Buffer, as shown below.

[0x68, 0x65, 0x6c, 0x6c, 0x6f]
^
|
Bin. slice (2)
Therefore, changes to the Buffer returned by the slice method will apply to the original Buffer, for example:

The Code is as follows:

Var buffer = new Buffer ([0x68, 0x65, 0x6c, 0x6c, 0x6f]);
Var sub = bin. slice (2 );
Sub [0] = 0x65;
Console. log (buffer );//

If you want to copy a Buffer, you must first create a new Buffer and use the. copy method to copy the data in the original Buffer.

This is similar to applying for a new memory and copying the data in the existing memory. The following is an example.

The Code is as follows:

Var buffer = new Buffer ([0x68, 0x65, 0x6c, 0x6c, 0x6f]);
Var dup = new Buffer (bin. length );
Buffer. copy (dup );
Dup [0] = 0x48;
Console. log (buffer );//
Console. log (dup );//

In short, Buffer extends JS data processing capabilities from strings to any binary data.

The above is a simple example to show you what Buffer is. The following describes how to use it and how to use it in specific scenarios.

Ii. Talk about Buffer

JavaScript is very friendly to string processing. Both wide-byte and single-byte strings are considered as a string. Node needs to process network protocols, operate databases, process images, upload files, and so on. It also needs to process a large amount of binary data. The built-in strings are far from meeting these requirements, so the Buffer came into being.

Buffer Structure

Buffer is a typical module that combines Javascript and C ++. The performance-related module is implemented using C ++, and the non-performance-related module is implemented using javascript.

When the Node starts, the Buffer has been installed into the memory and placed into the global object. Therefore, require is not required.

Buffer object: similar to an array, the element is a hexadecimal two-digit number.

Buffer Memory Allocation

The Buffer object memory allocation is not in the V8 heap memory, and the memory application is implemented at the Node C ++ layer.

To efficiently use the applied memory, Node uses the slab allocation mechanism. slab is a dynamic memory management mechanism that applies various * nix operating systems. Slab has three statuses:

(1) full: full allocation status

(2) partial: partial allocation status

(3) empty: not allocated

Buffer Conversion

The Buffer object can be converted to a string. The supported encoding types are as follows:

ASCII, UTF-8, UTF-16LE/UCS-2, Base64, Binary, Hex

String to Buffer

New Buffer (str, [encoding]), default UTF-8
Buf. write (string, [offset], [length], [encoding])

Buffer to string

Buf. toString ([encoding], [start], [end])

Encoding types not supported by Buffer

Determine whether Buffer. isEncoding (encoding) is supported

Iconv-lite: Pure JavaScript implementation, lighter, better performance, no conversion from C ++ to javascript

Iconv: completed by calling the libiconv library of C ++

Buffer Splicing

Note "res. on ('data', function (chunk) {}) ", where the chunk parameter is a Buffer object. Directly splicing with + will be automatically converted to a string, the wide-byte characters may cause garbled characters,

Solution:

(1) by using the setEncoding () method in a readable stream, This method enables data event transmission to be no longer a Buffer object, but an encoded string. The StringEncoder module is used internally.

(2) Save the Buffer object to the array, and finally assemble it into a large Buffer to convert the encoding into string output.

Buffer is widely used in file I/O and network I/O, and its performance is more important than that of common strings.

In addition to the performance loss of String Conversion, the use of Buffer has a highWaterMark setting that is critical to the performance impact during file reading.

A And highWaterMark settings have certain impact on the allocation and use of Buffer memory.

B. The value of highWaterMark is too small, which may lead to too many system calls.

When should buffer be used and when should not be used ------ pure javascript supports unicode code but not binary is very supported. When solving TCP stream or file stream, it is necessary to process the stream, when we save non-UTF-8 strings, binary, and other formats, we must use "Buffer".

Iii. instance Introduction

The Code is as follows:

Var buf = new Buffer ("this is text concat test! "), Str =" this is text concat test! ";
Console. time ("buffer concat test! ");
Var list = [];
Var len = 100000 * buf. length;
For (var I = 0; I <100000; I ++ ){
List. push (buf );
Len + = buf. length;
}
Var s1 = Buffer. concat (list, len). toString ();
Console. timeEnd ("buffer concat test! ");
Console. time ("string concat test! ");
Var list = [];
For (var I = 100000; I> = 0; I --){
List. push (str );
}
Var s2 = list. join ("");
Console. timeEnd ("string concat test! ");

The running result is as follows:

The read speed must be faster than string, and the buffer also needs toString () operations. Therefore, when saving the string, we should use the string, even if the large string is not slower than the buffer.

So when do we need to use a buffer? When we save non-UTF-8 strings, binary, and other formats, we must use them.

Iv. Summary

(1) JavaScript is suitable for processing Unicode encoded data, but it is not friendly to binary data processing.
(2). Therefore, when processing a TCP stream or file system, it is necessary to process an eight-bit byte stream.
(3) Node has several methods for processing, creating, and consuming eight-byte streams.
(4) raw data is stored in a Buffer instance. A Buffer is similar to an integer array, but its memory is allocated outside the V8 stack. The size of a Buffer cannot be changed.
(5) The encoding types include: ascii, utf8, utf16le, ucs2 (alias of utf16le), base64, binary, and hex.
(6), Buffer is a global element, and a Buffer instance is obtained directly using new Buffer.