[Reprint] Buffer module BUFFER--NODEJS Middleware Series

The zero-based NODEJS series article will show you how to benefit JavaScript as a server-side script through NODEJS Framework web development. The NODEJS framework is a V8-based engine and is the fastest JavaScript engine available today. The Chrome browser is based on V8, and opening 20-30 pages is a smooth one. The NODEJS Standard Web Development Framework Express helps us quickly build web sites that are more efficient than PHP and have a lower learning curve. Very suitable for small websites, personalization sites, our own Geek website!!

About the author

• Zhang Dan (Conan), entrepreneur, programmer (JAVA,R,JAVASCRIPT/NODEJS)
• Weibo: @Conan_Z
• Blog:http://blog.fens.me
• Email: [Email protected]

Reprint please specify the source:
http://blog.fens.me/nodejs-buffer/

Objective

JavaScript is designed for browsers and handles Unicode-encoded strings well, but for binary or non-Unicode encoded data it seems powerless. Node. JS inherits the language features of JavaScript, while extending the JavaScript language, providing the buffer class for binary data processing, so that node. JS can handle all types of data like any other programming language.

There are many articles on the internet, mostly speaking of the principle, how to use almost can not find, the article will focus on the use of buffer.

Directory

1. Buffer Introduction
2. Basic use of buffer
3. Performance test for Buffer

1. Buffer Introduction

In node. js, the buffer class is the core library that is published with the node kernel. The buffer library provides a way for node. js to store raw data, allowing Nodejs to process binary data, and it is possible to use the buffer library whenever data moved in an I/O operation needs to be handled in Nodejs. The raw data is stored in an instance of the Buffer class. A Buffer is similar to an array of integers, but it corresponds to a piece of raw memory outside of the V8 heap memory.

The conversion between the Buffer and the Javascript string object requires an explicit call to the encoding method to complete. Here are a couple of different string encodings:

• ' ASCII ' – only for 7-bit ASCII characters. This encoding method is very fast and discards high-level data.
• ' UTF8 ' – multibyte-encoded Unicode characters. Many Web pages and other file formats use UTF-8.
• ' Ucs2 ' – a Unicode character encoded in a small tail byte order (Little-endian) of two bytes. It can encode only the characters in the range of BMP (Basic Multilingual Plane, u+0000–u+ffff).
• ' Base64 ' –base64 string encoding.
• ' Binary ' – an encoding that converts raw binary data into strings, using only the first 8 bits of each character. This encoding method is obsolete and should be used as much as possible with the Buffer object. Subsequent versions of Node will delete this encoding.

Buffer Official Document: http://nodejs.org/api/buffer.html

2. Basic use of buffer

The basic use of buffer, mainly is the operation provided by the API, mainly includes 3 parts to create buffer class, read buffer, write buffer. As the basic operation is described in detail in the official documentation, I'll simply list it.

System environment

• Win7 64bit
• nodejs:v0.10.31
• npm:1.4.23

Create a project

~ cd D:\workspace\javascript>~ D:\workspace\javascript>mkdir nodejs-buffer && cd nodejs-buffer

2.1 Creating a Buffer class

To create an instance of buffer, we create it through new buffer. New file Buffer_new.js.

~ vi buffer_new.js// 长度为0的Buffer实例var a = new Buffer(0);console.log(a);> <Buffer >// 长度为0的Buffer实例相同，a1,a2是一个实例var a2 = new Buffer(0);console.log(a2);> <Buffer >// 长度为10的Buffer实例var a10 = new Buffer(10);console.log(a10);> <Buffer 22 37 02 00 00 00 00 04 00 00>// 数组var b = new Buffer([‘a‘,‘b‘,12])console.log(b);> <Buffer 00 00 0c>// 字符编码var b2 = new Buffer(‘你好‘,‘utf-8‘);console.log(b2);> <Buffer e4 bd a0 e5 a5 bd>

The buffer class has 5 class methods for the secondary operation of the buffer class.

1) encoding check, the above mentioned buffer and JavaScript string conversion, you need to explicitly set the encoding, then these kinds of encoding is supported by buffer. Like Chinese processing can only use UTF-8 encoding, for a few years ago, the usual gbk,gb2312 encoding is unresolved.

// 支持的编码console.log(Buffer.isEncoding(‘utf-8‘))console.log(Buffer.isEncoding(‘binary‘))console.log(Buffer.isEncoding(‘ascii‘))console.log(Buffer.isEncoding(‘ucs2‘))console.log(Buffer.isEncoding(‘base64‘))console.log(Buffer.isEncoding(‘hex‘))  # 16制进> true//不支持的编码console.log(Buffer.isEncoding(‘gbk‘))console.log(Buffer.isEncoding(‘gb2312‘))> false

2) buffer check, many times we need to determine the type of data, corresponding to subsequent operations.

// 是Buffer类console.log(Buffer.isBuffer(new Buffer(‘a‘)))> true// 不是Bufferconsole.log(Buffer.isBuffer(‘adfd‘))console.log(Buffer.isBuffer(‘\u00bd\u00bd‘))> false

3) The byte length of the string, because the string encoding is different, so the length of the string and the length of the byte is sometimes not the same. For example, 1 Chinese characters are 3 bytes, and the output via UTF-8 encoding is 4 Chinese characters, accounting for 12 bytes.

var str2 = ‘粉丝日志‘;console.log(str2 + ": " + str2.length + " characters, " + Buffer.byteLength(str2, ‘utf8‘) + " bytes");> 粉丝日志: 4 characters, 12 bytesconsole.log(str2 + ": " + str2.length + " characters, " + Buffer.byteLength(str2, ‘ascii‘) + " bytes");> 粉丝日志: 4 characters, 4 bytes

4) The connection to buffer, which is used to connect an array of buffer. We can manually allocate the buffer space size after the buffer object is merged, and if the buffer space is not enough, the data will be truncated.

var b1 = new Buffer("abcd");var b2 = new Buffer("1234");var b3 = Buffer.concat([b1,b2],8);console.log(b3.toString());> abcd1234var b4 = Buffer.concat([b1,b2],32);console.log(b4.toString());console.log(b4.toString(‘hex‘));//16进制输出> abcd1234 乱码....> 616263643132333404000000000000000000000000000000082a330200000000var b5 = Buffer.concat([b1,b2],4);console.log(b5.toString());> abcd

Program to run

5) buffer comparison, used to sort the contents of the buffer, in the order of the strings.

var a1 = new Buffer(‘10‘);var a2 = new Buffer(‘50‘);var a3 = new Buffer(‘123‘);// a1小于a2console.log(Buffer.compare(a1,a2));> -1// a2小于a3console.log(Buffer.compare(a2,a3));> 1// a1,a2,a3排序输出console.log([a1,a2,a3].sort(Buffer.compare));> [ <Buffer 31 30>, <Buffer 31 32 33>, <Buffer 35 30> ]// a1,a2,a3排序输出，以utf-8的编码输出console.log([a1,a2,a3].sort(Buffer.compare).toString());> 10,123,50

2.2 Write Buffer

Writes the data to the buffer operation, creating a new file Buffer_write.js.

~ vi buffer_write.js//////////////////////////////// Buffer写入//////////////////////////////// 创建空间大小为64字节的Buffervar buf = new Buffer(64);// 从开始写入Buffer，偏移0var len1 = buf.write(‘从开始写入‘);// 打印数据的长度，打印Buffer的0到len1位置的数据console.log(len1 + " bytes: " + buf.toString(‘utf8‘, 0, len1));// 重新写入Buffer，偏移0，将覆盖之前的Buffer内存len1 = buf.write(‘重新写入‘);console.log(len1 + " bytes: " + buf.toString(‘utf8‘, 0, len1));// 继续写入Buffer，偏移len1，写入unicode的字符串var len2 = buf.write(‘\u00bd + \u00bc = \u00be‘,len1);console.log(len2 + " bytes: " + buf.toString(‘utf8‘, 0, len1+len2));// 继续写入Buffer，偏移30var len3 = buf.write(‘从第30位写入‘, 30);console.log(len3 + " bytes: " + buf.toString(‘utf8‘, 0, 30+len3));// Buffer总长度和数据console.log(buf.length + " bytes: " + buf.toString(‘utf8‘, 0, buf.length));// 继续写入Buffer，偏移30+len3var len4 = buf.write(‘写入的数据长度超过Buffer的总长度！‘,30+len3);// 超过Buffer空间的数据，没有被写入到Buffer中console.log(buf.length + " bytes: " + buf.toString(‘utf8‘, 0, buf.length));

Node. js's buffer, depending on the range of read and write integers, provides support for different widths, so that integers from 1 to 8 bytes (8-bit, 16-bit, 32-bit), floating-point number (float), double-precision floating-point number (double) can be accessed, respectively, corresponding to different writexxx ( ) function, using a method similar to Buf.write ().

buf.write(string[, offset][, length][, encoding])buf.writeUIntLE(value, offset, byteLength[, noAssert])buf.writeUIntBE(value, offset, byteLength[, noAssert])buf.writeIntLE(value, offset, byteLength[, noAssert])buf.writeIntBE(value, offset, byteLength[, noAssert])buf.writeUInt8(value, offset[, noAssert])buf.writeUInt16LE(value, offset[, noAssert])buf.writeUInt16BE(value, offset[, noAssert])buf.writeUInt32LE(value, offset[, noAssert])buf.writeUInt32BE(value, offset[, noAssert])buf.writeInt8(value, offset[, noAssert])buf.writeInt16LE(value, offset[, noAssert])buf.writeInt16BE(value, offset[, noAssert])buf.writeInt32LE(value, offset[, noAssert])buf.writeInt32BE(value, offset[, noAssert])buf.writeFloatLE(value, offset[, noAssert])buf.writeFloatBE(value, offset[, noAssert])buf.writeDoubleLE(value, offset[, noAssert])buf.writeDoubleBE(value, offset[, noAssert])

In addition, there are some prototype functions of the buffer class that can be manipulated in the buffer write operation.

Buffer copy function buf.copy (targetbuffer[, targetstart][, sourcestart][, Sourceend]).

// 新建两个Buffer实例var buf1 = new Buffer(26);var buf2 = new Buffer(26);// 分别向2个实例中写入数据for (var i = 0 ; i < 26 ; i++) {    buf1[i] = i + 97; // 97是ASCII的a    buf2[i] = 50; // 50是ASCII的2}// 把buf1的内存复制给buf2buf1.copy(buf2, 5, 0, 10); // 从buf2的第5个字节位置开始插入，复制buf1的从0-10字节的数据到buf2中console.log(buf2.toString(‘ascii‘, 0, 25)); // 输入buf2的0-25字节> 22222abcdefghij2222222222

Buffer fill function Buf.fill (value[, offset][, end]).

// 新建Buffer实例，长度20节节var buf = new Buffer(20);// 向Buffer中填充数据buf.fill("h");console.log(buf)> <Buffer 68 68 68 68 68 68 68 68 68 68 68 68 68 68 68 68 68 68 68 68>console.log("buf:"+buf.toString())> buf:hhhhhhhhhhhhhhhhhhhh// 清空Buffer中的数据buf.fill();console.log("buf:"+buf.toString())> buf:

Buffer clipping, buf.slice ([start][, end]). Returns a new buffer that points to the same piece of memory as the old buffer, but is clipped from the index start to end.

var buf1 = new Buffer(26);for (var i = 0 ; i < 26 ; i++) {    buf1[i] = i + 97;}// 从剪切buf1中的0-3的位置的字节，新生成的buf2是buf1的一个切片。var buf2 = buf1.slice(0, 3);console.log(buf2.toString(‘ascii‘, 0, buf2.length));> abc// 当修改buf1时，buf2同时也会发生改变buf1[0] = 33;console.log(buf2.toString(‘ascii‘, 0, buf2.length));> !bc

2.3 Reading Buffer

After we write the data to buffer, we also need to read the data out of buffer and create a new file buffer_read.js. We can use the readxxx () function to get the index value that should be encoded at the write-in, and then convert the original value out, there is a way to manipulate the Chinese characters will become troublesome, the most common method of reading buffer, in fact, is ToString ().

~ vi buffer_read.js//////////////////////////////// Buffer 读取//////////////////////////////var buf = new Buffer(10);for (var i = 0 ; i < 10 ; i++) {    buf[i] = i + 97;}console.log(buf.length + " bytes: " + buf.toString(‘utf-8‘));> 10 bytes: abcdefghij// 读取数据for (ii = 0; ii < buf.length; ii++) {    var ch = buf.readUInt8(ii); // 获得ASCII索引    console.log(ch + ":"+ String.fromCharCode(ch));}> 97:a98:b99:c100:d101:e102:f103:g104:h105:i106:j

Writes the Chinese data, reads with READXXX, 3 bytes to represent a medium text.

var buf = new Buffer(10);buf.write(‘abcd‘)buf.write(‘数据‘,4)for (var i = 0; i < buf.length; i++) {    console.log(buf.readUInt8(i));}>979899100230  // 230,149,176 代表“数”149176230  

If you want to output the correct Chinese, then we can use the ToString (' utf-8 ') function to operate.

console.log("buffer :"+buf); // 默认调用了toString()的函数> buffer :abcd数据console.log("utf-8  :"+buf.toString(‘utf-8‘));> utf-8  :abcd数据console.log("ascii  :"+buf.toString(‘ascii‘));//有乱码，中文不能被正确解析> ascii  :abcdf0f.console.log("hex    :"+buf.toString(‘hex‘)); //16进制> hex    :61626364e695b0e68dae

For the output of buffer, the most common operation we use is ToString (), which is read according to the encoded encoding. In addition to the ToString () function, the Tojson () direct buffer can be used to parse the JSON object.

var buf = new Buffer(‘test‘);console.log(buf.toJSON());> { type: ‘Buffer‘, data: [ 116, 101, 115, 116 ] }

3. Buffer Performance Test

With the introduction to buffer above, we've learned about the basic use of buffer, and then we'll start doing some testing with buffer.

3.1 8K's creation test

Every time we create a new buffer instance, we check that the memory pool of the current buffer is full, the current memory pool is shared with the new buffer instance, and the size of the memory pool is 8K.

If the newly created buffer instance is larger than 8K, the buffer is given to the Slowbuffer instance store, and if the newly created buffer instance is less than 8K and less than the remaining space in the current memory pool, the buffer is deposited into the current memory pool If the buffer instance is less than 0, the default Zerobuffer instance is returned uniformly.

Below we create 2 buffer instances, the first one is in 4k space, the second in 4.001k space, the loop is created 100,000 times.

var num = 100*1000;console.time("test1");for(var i=0;i<num;i++){    new Buffer(1024*4);}console.timeEnd("test1");> test1: 132msconsole.time("test2");for(var j=0;j<num;j++){    new Buffer(1024*4+1);}console.timeEnd("test2");> test2: 163ms

The second 4.001k space is 23% more time, which means that the second, every two cycles, should reapply for the memory pool space. This is what we need to be very careful about.

3.2 Multiple buffer or single buffer

When we need to cache the data, is it good to create multiple small buffer instances, or to create a large buffer instance? For example, we want to create 10,000 strings that range in length from 1-2048 to eight.

var max = 2048;     //最大长度var time = 10*1000; //循环1万次// 根据长度创建字符串function getString(size){    var ret = ""    for(var i=0;i<size;i++) ret += "a";    return ret;}// 生成字符串数组，1万条记录var arr1=[];for(var i=0;i<time;i++){    var size = Math.ceil(Math.random()*max)    arr1.push(getString(size));}//console.log(arr1);// 创建1万个小Buffer实例console.time(‘test3‘);var arr_3 = [];for(var i=0;i<time;i++){    arr_3.push(new Buffer(arr1[i]));}console.timeEnd(‘test3‘);> test3: 217ms// 创建一个大实例，和一个offset数组用于读取数据。console.time(‘test4‘);var buf = new Buffer(time*max);var offset=0;var arr_4=[];for(var i=0;i<time;i++){    arr_4[i]=offset;    buf.write(arr1[i],offset,arr1[i].length);    offset=offset+arr1[i].length;}console.timeEnd(‘test4‘);> test4: 12ms

Reads the data indexed to 2.

console.log("src:[2]="+arr1[2]);console.log("test3:[2]="+arr_3[2].toString());console.log("test4:[2]="+buf.toString(‘utf-8‘,arr_4[2],arr_4[3]));

Run the results.

For this type of requirement, it is more efficient to generate a large buffer instance ahead of time than it is to generate a small buffer instance, which can increase the computational efficiency of an order of magnitude. Therefore, it is very important to understand and use good buffer!!

3.3 String VS Buffer

With buffer, do we need to replace all strings with a buffer connection? So we need to test, which is faster when string and buffer are concatenated?

Below we make a string connection, looping 300,000 times.

//测试三，Buffer VS stringvar time = 300*1000;var txt = "aaa"var str = "";console.time(‘test5‘)for(var i=0;i<time;i++){    str += txt;}console.timeEnd(‘test5‘)> test5: 24msconsole.time(‘test6‘)var buf = new Buffer(time * txt.length)var offset = 0;for(var i=0;i<time;i++){    var end = offset + txt.length;    buf.write(txt,offset,end);    offset=end;}console.timeEnd(‘test6‘)> test6: 85ms

From the test results, we can obviously see that string connection operation to the strings, much faster than the connection operation buffer. So when we save the string, we should use string or string. Then we only use buffer if we save the non-utf-8 string and the binary data.

6. Program code

The program code of this article, you can download the source code of this project directly from GitHub, and follow the instructions in the article to learn Buffer,:https://github.com/bsspirit/nodejs-buffer

You can also download it directly from the GitHub command line:

~ git clone [email protected]:bsspirit/nodejs-buffer.git   # 下载github项目~ cd nodejs-buffer                                      # 进入下载目录

About the bottom of node. js, I do not have much contact, failed to do more in-depth research from V8 (c + +), just on the use level, write my summary. If the error or description in the text is unclear, Daniel should correct it!!

Reference article:

Analysis on the buffer class of Nodejs

Reprint please specify the source:
Http://blog.fens.me/nodejs-buffer

[Reprint] Buffer module BUFFER--NODEJS Middleware Series