Detailed descriptions of some ignored Buffer usage in Node. js

About Node. there are many articles on Buffer usage in js, but I still feel that it is not detailed enough. Therefore, this article mainly introduces Node. some Usage of Buffer in js that you may not know are described in detail in this article. If you need it, you can refer to it for reference. Let's take a look at it. For more information about # wiki/1498.html "target =" _ blank "> Node. there are many articles on Buffer usage in js, but I still feel that it is not detailed enough. Therefore, this article mainly introduces Node. some Usage of Buffer in js that you may not know are described in detail in this article. If you need it, you can refer to it for reference. Let's take a look at it.

Preface

In most articles about Buffer, we mainly focus on data splicing and memory allocation. For example, when we use the fs module to read the file content, a Buffer is returned:

fs.readFile('filename', function (err, buf) { // 
 
  });
 

When the net or http module is used to receive network data, the data event parameter is also a Buffer. In this case, we need to useBuffer.concat()To splice data:

Var bufs = []; conn. on ('data', function (buf) {bufs. push (buf) ;}); conn. on ('end', function () {// After receiving data, splice all received Buffer objects var buf = Buffer. concat (bufs );});

You can also useBuffer.toString()To convert base64 or hexadecimal characters, for example:

Console. log (new Buffer ('hello, world! '). ToString ('base64'); // convert to a base64 string: aGVsbG8sIHdvcmxkIQ = console. log (new Buffer ('agvsbg8sihdvcmxkiq = ', 'base64 '). toString (); // restore the base64 string: hello, world! Console. log (new Buffer ('hello, world! '). ToString ('hex'); // convert to a hexadecimal string: 68656c6c6f2c20776f726c6421console. log (new Buffer ('68656c6c6f2c20776f0000c6421 ', 'hex '). toString (); // restore the hexadecimal string: hello, world!

Generally, a single Node. js process has a maximum memory limit. The following is a description from the official documentation:

What is the memory limit on a node process?

Currently, by default v8 has a memory limit of 512 MB on 32-bit systems, and 1.4 GB on 64-bit systems. the limit can be raised by setting -- max_old_space_size to a maximum ~ 1024 (~ 1 GB) (32-bit) and ~ 4096 (~ 4 GB) (64-bit), but it is recommended that you split your single process into several workers if you are hitting memory limits.

Since the memory space occupied by the Buffer object is not counted in the memory space limit of the Node. js process, we often use the Buffer to store data that requires a large amount of memory:

// Allocate a 2 g-1 byte of data // if the memory size exceeds this value at a time, an exception RangeError will be thrown: Invalid typed array lengthvar buf = new Buffer (1024*1024*1024-1 );

The above are several common usage of Buffer. However, when reading the Buffer API documentation, we will find thatreadXXX()AndwriteXXX()The API at the beginning is as follows:

• Buf. readUIntLE (offset, byteLength [, noAssert])

• Buf. readUIntBE (offset, byteLength [, noAssert])

• Buf. readIntLE (offset, byteLength [, noAssert])

• Buf. readIntBE (offset, byteLength [, noAssert])

• Buf. readUInt8 (offset [, noAssert])

• Buf. readUInt16LE (offset [, noAssert])

• Buf. readUInt16BE (offset [, noAssert])

• Buf. readUInt32LE (offset [, noAssert])

• Buf. readUInt32BE (offset [, noAssert])

• Buf. readInt8 (offset [, noAssert])

• Buf. readInt16LE (offset [, noAssert])

• Buf. readInt16BE (offset [, noAssert])

• Buf. readInt32LE (offset [, noAssert])

• Buf. readInt32BE (offset [, noAssert])

• Buf. readFloatLE (offset [, noAssert])

• Buf. readFloatBE (offset [, noAssert])

• Buf. readDoubleLE (offset [, noAssert])

• Buf. readDoubleBE (offset [, noAssert])

• Buf. write (string [, offset] [, length] [, encoding])

• Buf. writeUIntLE (value, offset, byteLength [, noAssert])

• Buf. writeUIntBE (value, offset, byteLength [, noAssert])

• Buf. writeIntLE (value, offset, byteLength [, noAssert])

• Buf. writeIntBE (value, offset, byteLength [, noAssert])

• Buf. writeUInt8 (value, offset [, noAssert])

• Buf. writeUInt16LE (value, offset [, noAssert])

• Buf. writeUInt16BE (value, offset [, noAssert])

• Buf. writeUInt32LE (value, offset [, noAssert])

• Buf. writeUInt32BE (value, offset [, noAssert])

• Buf. writeInt8 (value, offset [, noAssert])

• Buf. writeInt16LE (value, offset [, noAssert])

• Buf. writeInt16BE (value, offset [, noAssert])

• Buf. writeInt32LE (value, offset [, noAssert])

• Buf. writeInt32BE (value, offset [, noAssert])

• Buf. writeFloatLE (value, offset [, noAssert])

• Buf. writeFloatBE (value, offset [, noAssert])

• Buf. writeDoubleLE (value, offset [, noAssert])

• Buf. writeDoubleBE (value, offset [, noAssert])
  

These APIs provide great convenience for data operations in Node. js. Suppose we want to store an integer value in a file. For example, if the current timestamp is 1447656645380 and it is stored as a string, 11 bytes of space is required, to convert it to binary storage, only 6 bytes of space is required:

var buf = new Buffer(6);buf.writeUIntBE(1447656645380, 0, 6);// 
 
  buf.readUIntBE(0, 6);// 1447656645380
 

When using Node. when js compiles some underlying functions, such as a network communication module, a database client module, or a large amount of structured data needs to be operated from a file, the APIS provided by the above Buffer objects are essential.

Next, we will demonstrate an example of using a Buffer object to operate structured data.

Operate on structured data

Assume that there is a student examination score database, and each record structure is as follows:

Student ID	Course code	Score
XXXXXX	XXXX	XX

The student ID is a six-digit number, and the course code is a four-digit number. The highest score is 100.

When using text to store the data, for example, using CSV format for storage may be like this:

100001,1001,99100002,1001,67100003,1001,88

Each record occupies 15 bytes of space, and its structure is as follows when binary storage is used:

Student ID	Course code	Score
3 bytes	2 bytes	1 byte

Each record only needs 6 bytes of space, which is only 40% of text storage space! The procedure for operating these records is as follows:

// Read a record // buf Buffer object // offset this record is at the beginning of the Buffer object // data {number, lesson, score} function writeRecord (buf, offset, data) {buf. writeUIntBE (data. number, offset, 3); buf. writeUInt16BE (data. lesson, offset + 3); buf. writeInt8 (data. score, offset + 5);} // write a record // buf Buffer object // offset this record is at the beginning of the Buffer object function readRecord (buf, offset) {return {number: buf. readUIntBE (offset, 3), lesson: buf. readUInt16BE (offset + 3), score: buf. readInt8 (offset + 5) };}// write record list // list record list, each of which contains {number, lesson, score} function writeList (list) {var buf = new Buffer (list. length * 6); var offset = 0; for (var I = 0; I <list. length; I ++) {writeRecord (buf, offset, list [I]); offset + = 6;} return buf ;} // read the record list // function readList (buf) of the buf Buffer object {var offset = 0; var list = []; while (offset <buf. length) {list. push (readRecord (buf, offset); offset + = 6;} return list ;}

We can write another program to see the effect:

Var list = [{number: 100001, lesson: 1001, score: 99}, {number: 100002, lesson: 1001, score: 88}, {number: 100003, lesson: 1001, score: 77}, {number: 100004, lesson: 1001, score: 66}, {number: 100005, lesson: 1001, score: 55},]; console. log (list); var buf = writeList (list); console. log (buf); // output
 
  
Var ret = readList (buf); console. log (ret);/* output [{number: 100001, lesson: 1001, score: 99}, {number: 100002, lesson: 1001, score: 88}, {number: 100003, lesson: 1001, score: 77}, {number: 100004, lesson: 1001, score: 66}, {number: 100005, lesson: 1001, score: 55}] */
 

Lei-proto module Introduction

In the above example, when the structure of each record changes, we need to modifyreadRecord()AndwriteRecord() To recalculate the offset of each field in the Buffer. When the recorded field is complex, it is prone to errors. For this reason, I have compiled the lei-proto module, which allows you to generate the correspondingreadRecord()And`writeRecord()Function.

First, run the following command to install this module:

$ npm install lei-proto --save

After using the lei-proto module, the preceding example can be changed to the following:

Var parsePorto = require ('lei-proto'); // generate the data encoding for the specified record structure/decoder var record = parsePorto (['number', 'uint ', 3], ['leson', 'uint', 2], ['score ', 'uint', 1]); function readList (buf) {var list = []; var offset = 0; while (offset <buf. length) {list. push (record. decode (buf. slice (offset, offset + 6); offset + = 6;} return list;} function writeList (list) {return Buffer. concat (list. map (record. encodeEx ));}

Run the same test program as above and you can see that the result is the same:

 
  [ { number: 100001, lesson: 1001, score: 99 }, { number: 100002, lesson: 1001, score: 88 }, { number: 100003, lesson: 1001, score: 77 }, { number: 100004, lesson: 1001, score: 66 }, { number: 100005, lesson: 1001, score: 55 } ]
 

Summary

The above is a detailed description of some ignored usage of Buffer in Node. js. For more information, see other related articles in the first PHP community!