Nodejs Stream Data Flow usage manual _node.js

1. Introduction

This article introduces the basic method of using node.js streams to develop programs.

<code class= "Hljs Mizar" > "We should have some ways of connecting programs like Garden Hose--screw in
another seg ment the it becomes necessary to massage the data in
another way. This is the
way of IO also. Doug McIlroy. October, 1964</code>

The first contact stream was a decades-old practice from the early days of UNIX to prove that stream thought can be very simple to develop a number of large systems. In Unix, stream is implemented through |, and in node, as a built-in stream module, many core modules and three-party modules are used. As with UNIX, the main operation of node stream is also. Pipe (), users can use the reverse pressure mechanism to control the reading and writing balance.

The stream can provide developers with a reusable, unified interface to control the read-write balance between the stream through an abstract stream interface.

2. Why use Stream

I/O in node is asynchronous, so reading and writing to disk and network requires a callback function to read the data, and the following is a simple code for a file download server:

<code class= "hljs javascript" >var http = require (' http ');
var fs = require (' FS ');
var server = Http.createserver (function (req, res) {
fs.readfile (__dirname + '/data.txt ', function (err, data) {
R Es.end (data)
;}); Server.listen (8000);</code>

The code can implement the required functionality, but the service needs to cache the entire file data to memory before sending the file data, and if the "data.txt" file is large and has a large concurrency, it wastes a lot of memory. Because the user needs to wait until the entire file is cached to memory to accept file data, this results in a very poor user experience. But fortunately (req, res) two parameters are stream, so we can use Fs.createreadstream () instead of Fs.readfile ():

<code class= "hljs javascript" >var http = require (' http ');
var fs = require (' FS ');
var server = Http.createserver (function (req, res) {
var stream = Fs.createreadstream (__dirname + '/data.txt '); 
   
    stream.pipe (res);
});
Server.listen (8000);</code>
   

The. Pipe () method listens for the ' data ' and ' end ' events of Fs.createreadstream () so that the "data.txt" file does not need to cache the entire file and can send a block of data to the client immediately after the client connection completes. Using. Pipe () Another benefit is that it solves the problem of read and write imbalances that result when the client's latency is very high. If you want to compress files and send them, you can use a three-party module to implement:

<code class= "hljs javascript" >var http = require (' http ');
var fs = require (' FS ');
var oppressor = require (' oppressor ');
var server = Http.createserver (function (req, res) {
var stream = Fs.createreadstream (__dirname + '/data.txt ');
Stream.pipe (oppressor (req)). pipe (res);
Server.listen (8000);</code>

This allows the file to compress the browsers that support gzip and deflate. The oppressor module will handle all the content-encoding.

Stream makes the development program simple.

3. Basic Concepts

There are five kinds of basic stream:readable, writable, transform, duplex, and "classic".

3-1, Pipe

All types of stream receipts are created with a. pipe () to create an input-output pair, receive a readable stream src and output its data to a writable stream of DST, as follows:

<code class= "Hljs perl" >src.pipe (DST) </code>

The. Pipe (DST) method returns a DST stream, which allows multiple. pipe () to be used successively, as follows:

<code class= "Hljs perl" >a.pipe (b). pipe (c). Pipe (d) </code>

The functionality is the same as the following code:

<code class= "Hljs perl" >a.pipe (b);
B.pipe (c);
C.pipe (d);</code>

3-2, readable streams

You can write the readable streams data to a writable, Transform, or duplex stream by invoking the. Pipe () method of readable streams.

<code class= "Hljs perl" >readablestream.pipe (DST) </code>

1> Create readable stream

Here we create a readable stream!

<code class= "Hljs perl" >var readable = require (' stream '). readable;
var rs = new readable;
Rs.push (' beep ');
Rs.push (' boop\n ');
Rs.push (null);
Rs.pipe (process.stdout);
$ node read0.js
beep boop
</code>

Rs.push (NULL) notification data recipient data has been sent.

Note that we did not call Rs.pipe (process.stdout) until all the data content was pressed into the readable stream, but all the data content we pressed into was fully output, because the readable stream caches all the pressed data before the receiver reads the data. But in many cases, a better approach is to press data into a readable stream instead of caching the entire data only when the data receives the request data. Now let's rewrite the. _read () function:

<code class= "Hljs javascript" >var readable = require (' stream '). readable;
var rs = readable ();
var c =;
Rs._read = function () {
Rs.push (String.fromCharCode (c + +));
if (C > ' z '. charCodeAt (0)) Rs.push (null);
Rs.pipe (process.stdout);</code>
<code class= "Hljs bash" >$ node read1.js
Abcdefghijklmnopqrstuvwxyz</code>

The code above is implemented by overriding the _read () method to push data into a readable stream only if the data recipient requests it. The _read () method can also receive a size parameter that indicates the data request data size, but the readable stream can ignore this parameter as needed.

Note that we can also use Util.inherits () to inherit the readable stream. To show that the _read () method is invoked only when the data recipient requests the data, we do a delay when the data is pressed into the readable stream, as follows:

<code class= "Hljs javascript" >var readable = require (' stream '). readable;
var rs = readable ();
var c = 97-1;
Rs._read = function () {
if (c >= ' z '. charCodeAt (0)) return Rs.push (null);
settimeout (function () {
Rs.push (String.fromCharCode (++c));
};
Rs.pipe (process.stdout);
Process.on (' Exit ', function () {
console.error (' \n_read () called ' + (c-97) + ' Times ');
};
Process.stdout.on (' Error ', process.exit);</code>

Run the program with the following command we found that the _read () method was invoked only 5 times:

<code class= "Hljs bash" >$ node Read2.js | Head-c5
ABCDE
_read () called 5 times</code>

The reason for using timers is that the system takes time to send a signal to notify the program to close the pipe. The Process.stdout.on (' Error ', FN) is used to process the system because the header command closes the pipe and sends the SIGPIPE signal because it causes the process.stdout to trigger the Epipe event. If you want to create a readable stream that presses any form of data, set the parameter Objectmode to True when you create the stream, for example: readable ({objectmode:true}).

2> Read readable stream data

In most cases, we simply use the pipe method to redirect data from a readable stream to another form of flow, but in some cases it may be more useful to read data directly from a readable stream. As follows:

<code class= "Hljs php" >process.stdin.on (' readable ', function () {
var buf = Process.stdin.read ();
Console.dir (BUF);
});
$ (echo abc; sleep 1; Echo def; sleep 1; echo Ghi) | Node Consume0.js 
<buffer 0a= "61=" 62= "63=" ">
<buffer 0a=" "64=" "65=" "66=" ">
<buffer 0a = "" "67=" "68=" "69=" ">
null</buffer></buffer></buffer></code>

When there is data in the readable stream that can be read, the stream triggers the ' readable ' event so that it can invoke the. Read () method to read the relevant data, and when no data is readable in the readable stream,. Read () returns null so that the. Read () call is terminated, Wait for the next ' readable ' event to be triggered. Here is an example of using. Read (n) to read 3 bytes at a time from standard input:

<code class= "Hljs JavaScript" >process.stdin.on (' readable ', function () {
var buf = Process.stdin.read (3);
Console.dir (BUF);
}); </code>

If the running program found that the output results are not complete!

<code class= "Hljs bash" >$ (echo abc; sleep 1; Echo def; sleep 1; echo Ghi) | Node Consume1.js 
<buffer 61= "62=" "63=" ">
<buffer 0a=" "64=" "65=" ">
<buffer 0a=" "66 =" "67 = "" ></buffer></buffer></buffer></code>

This should be left in the internal buffer of the stream for additional data data, and we need to notify the stream that we want to read more data. Read (0) can achieve this goal.

<code class= "Hljs JavaScript" >process.stdin.on (' readable ', function () {
var buf = Process.stdin.read (3);
Console.dir (BUF);
Process.stdin.read (0);
}); </code>

The results of this operation are as follows:

<code class= "Hljs xml" >$ (echo abc; sleep 1; Echo def; sleep 1; echo Ghi) | Node Consume2.js 
<buffer 0a= "64=" "65=" "> <buffer 0a="
"68=" "69=" "></buffer></buffer ></code>

We can use. Unshift () to remand the data back to the head of the data queue so that the data returned can be read. As shown in the following code, the contents of the standard input are output by line:

<code class= "Hljs javascript" >var offset = 0;
Process.stdin.on (' readable ', function () {
var buf = Process.stdin.read ();
if (!BUF) return;
for (; offset < buf.length; offset++) {
if (buf[offset) = = 0x0a) {
console.dir (0, offset). toString ( ));
BUF = Buf.slice (offset + 1);
offset = 0;
Process.stdin.unshift (BUF);
return;
}
Process.stdin.unshift (BUF);
});
$ tail-n +50000/usr/share/dict/american-english | Head-n10 | Node Lines.js 
' hearties '
heartiest '
heartily ' heartiness '
heartiness\ ' s
' Heartland '
heartland\ ' s '
heartlands ' heartless '
heartlessly ' </code>

Of course, there are many modules that can implement this function, such as: Split.

3-3, writable streams

Writable streams can only be used as the purpose parameter of the. pipe () function. The following code:

<code class= "Hljs perl" >src.pipe (Writablestream);</code>

1> Create writable Stream

Rewrite the. _write (chunk, enc, next) method to accept data from a readable stream.

<code class= "hljs php" >var writable = require (' stream '). writable;
var ws = writable ();
Ws._write = function (chunk, enc, next) {
console.dir (chunk);
Next ();
};
Process.stdin.pipe (WS);
$ (echo beep; sleep 1; echo Boop) | Node Write0.js 
<buffer 0a= "62=" "65=" "70=" ">
<buffer 0a=" "62=" "6f=" "70=" "></buffer> </buffer></code>

The first parameter chunk is the data that is written by the data-entry person. The second argument end is the encoding format for the data. The third parameter, next (ERR), notifies the data writer that the writer can write more time through the callback function. If the readable stream writes a string, the string is converted to buffer by default, and if the writable ({Decodestrings:false}) parameter is set when the stream is created, the conversion is not done. If the readable stream writes the data when the object, then you need to create the writable stream

<code class= "Hljs css" >writable ({objectmode:true}) </code>

2> writes data to writable stream

The. Write (data) method of the writable stream is invoked to complete the write.

<code class= "Hljs Vala" >process.stdout.write (' beep boop\n ');</code>

The. End () method is called to notify that the writable stream data has been written to completion.

<code class= "Hljs javascript" >var fs = require (' FS ');
var ws = Fs.createwritestream (' message.txt ');
Ws.write (' beep ');
settimeout (function () {
ws.end (' boop\n ');
}, 1000);
$ node Writing1.js 
$ cat message.txt
beep boop</code>

If you need to set the size of the buffer for the writable stream, you need to set the Opts.highwatermark when creating the stream so that if the data in the buffer is over Opts.highwatermark,.write method returns FALSE. When the buffer is writable, the writable stream triggers the ' drain ' event.

3-4, classic streams

Classic streams older interface, first appeared in the node 0.4 version, but to understand its operating principle is very good
Department. When a stream is registered with the return function of the "data" event, then the stream will work in the old version mode, that is, the old API will be used.

1>classic readable streams

The Classic readable streams event is an event trigger that triggers a "data" event if the Classic readable streams has data to read, and then triggers an "end" event when the data is read. PIPE () method to determine whether the stream has data readable by checking the value of the stream.readable. Here is an example of using the classic readable streams to print a-j letters:

<code class= "Hljs javascript" >var Stream = require (' stream ');
var stream = new stream;
Stream.readable = true;
var c =;
var IV = setinterval (function () {
if (++c >=) {
clearinterval (iv);
Stream.emit (' end ');
}
else Stream.emit (' Data ', String.fromCharCode (c));
Stream.pipe (process.stdout);
$ node Classic0.js
abcdefghij</code>

If you want to read data from the classic readable stream, register the callback function for the "data" and "end" two events, as follows:

<code class= "Hljs php" >process.stdin.on (' Data ', function (buf) {
console.log (BUF);
});
Process.stdin.on (' End ', function () {
console.log (' __end__ ');
});
$ (echo beep; sleep 1; echo Boop) | Node Classic1.js 
<buffer 0a= "62=" "65=" "70=" ">
<buffer 0a=" "62=" "6f=" "70=" ">
__end__ </buffer></buffer></code>

Note that if you read data in this way, you lose the benefit of using the new interface. For example, when you write to a very large stream of data, you need to pay attention to the balance of reading and writing data, otherwise it will result in a large amount of data caching in memory, resulting in a waste of large amounts of memory. It is generally strongly recommended that you use the streaming. Pipe () method so that you do not have to listen to the "data" and "End" events yourself, and you don't have to worry about the problem of read and write imbalances. Of course, you can also use through instead of listening to "data" and "end" events, such as the following code:

<code class= "Hljs php" >var through = require (' through ');
Process.stdin.pipe (Through (write, end));
function Write (buf) {
console.log (BUF);
}
Function End () {
console.log (' __end__ ');
}
$ (echo beep; sleep 1; echo Boop) | Node Through.js 
<buffer 0a= "62=" "65=" "70=" ">
<buffer 0a=" "62=" "6f=" "70=" ">
__end__< /buffer></buffer></code>

Alternatively, you can use Concat-stream to cache the contents of the entire stream:

<code class= "Hljs oxygene" >var concat = require (' Concat-stream ');
Process.stdin.pipe (concat (function (body) {
console.log (json.parse);
});
$ Echo ' {' beep ': ' Boop '} ' | Node Concat.js 
{beep: ' Boop '}</code>

Of course, if you have to listen to the "data" and "End" events yourself, you can use the. Pause () method to pause the classic readable streams continue to trigger the "data" event when the stream of the data is not writable. Wait until the stream of the write data is writable. Resume () method notifies the stream to continue to trigger the "data" event to continue reading
Data.

2>classic writable Streams

Classic writable streams is very simple. Only. Write (BUF),. End (BUF) and. Destroy () three methods. The BUF parameter of the. End (BUF) method is optional, which is equivalent to Stream.Write (BUF) If the parameter is selected; Stream.end () in such an operation, it is necessary to note that when the stream's buffer is full and the stream is not writable. The Write (BUF) method returns False, and if the stream is writable again, the stream triggers the drain event.

4, Transform

Transform is a stream that filters the output of the read data.

5, Duplex

The duplex stream is a bidirectional stream that is readable and writable, as in the following a is a duplex stream:

<code class= "Hljs livecodeserver" >a.pipe (b). Pipe (a) </code>

The above content is small series to introduce the Nodejs stream data flow manual, I hope to help you!