Nodejs Stream data Stream user manual, nodejsstream

Nodejs Stream data Stream user manual, nodejsstream

1. Introduction

This article describes the basic methods for developing programs using node. js streams.

<code class="hljs mizar">"We should have some ways of connecting programs like garden hose--screw inanother segment when it becomes necessary to massage data inanother way. This is the way of IO also."Doug McIlroy. October 11, 1964</code>

Stream is the first practice that has been used for decades since the beginning of unix. It has proved that Stream can easily develop some large systems. In unix, Stream is implemented through |. In node, as a built-in stream module, many core modules and third-party modules are used. Like unix, the main operation of node Stream is. pipe (). You can use the anti-press mechanism to control the balance between read and write.

Stream can provide developers with unified interfaces that can be reused and control the read/write balance between streams through abstract Stream interfaces.

2. Why Stream?

Node I/O is asynchronous. Therefore, the disk and network read and write operations need to be read through the callback function. Below is a simple code of the file download server:

<code class="hljs javascript">var http = require('http');var fs = require('fs');var server = http.createServer(function (req, res) {fs.readFile(__dirname + '/data.txt', function (err, data) {res.end(data);});});server.listen(8000);</code>

These codes can implement the required functions, but the service needs to cache the entire file data to the memory before sending the file data. If the "data.txt" file is large and the concurrency is large, it will waste a lot of memory. This is because the user needs to wait until the entire file is cached in the memory to receive the file data, which leads to a poor user experience. But fortunately, both parameters (req, res) are Stream, so we can use fs. createReadStream () instead of fs. readFile ():

<code class="hljs javascript">var http = require('http');var fs = require('fs');var server = http.createServer(function (req, res) {var stream = fs.createReadStream(__dirname + '/data.txt');stream.pipe(res);});server.listen(8000);</code>

. The pipe () method listens to fs. the 'data' and 'end' events of createReadStream (), so that the "data.txt" file does not need to cache the entire file. After the client connection is complete, a data block can be sent to the client immediately. Another advantage of using. pipe () is that it can solve the read/write imbalance problem caused by high client latency. If you want to compress the file and try again, you can use the third-party module:

<code class="hljs javascript">var http = require('http');var fs = require('fs');var oppressor = require('oppressor');var server = http.createServer(function (req, res) {var stream = fs.createReadStream(__dirname + '/data.txt');stream.pipe(oppressor(req)).pipe(res);});server.listen(8000);</code>

In this way, the files will be compressed on browsers that support gzip and deflate. The oppressor module processes all content-encoding.

Stream makes development programs easy.

3. Basic Concepts

There are five basic streams: readable, writable, transform, duplex, and "classic ".

3-1. pipe

All Stream types use. pipe () to create an input/output pair, receive a readable Stream src, and output its data to the writable Stream dst, as shown below:

<code class="hljs perl">src.pipe(dst)</code>

The. pipe (dst) method returns the dst stream, so that multiple. pipe () can be successively used, as shown below:

<code class="hljs perl">a.pipe( b ).pipe( c ).pipe( d )</code>

The function is the same as the following code:

<code class="hljs perl">a.pipe( b );b.pipe( c );c.pipe( d );</code>

3-2. readable streams

You can call the. pipe () method of Readable streams to write data of Readable streams into a Writable, Transform, or Duplex stream.

<code class="hljs perl">readableStream.pipe( dst )</code>

1> Create a readable stream

Here we create a readable stream!

<code class="hljs perl">var Readable = require('stream').Readable;var rs = new Readable;rs.push('beep ');rs.push('boop\n');rs.push(null);rs.pipe(process.stdout);$ node read0.jsbeep boop</code>

Rs. push (null) notifies the data recipient that the data has been sent.

We noticed that rs was not called before we pushed all the data into the readable stream. pipe (process. stdout); but all the data content we press into is completely output, because the readable stream caches all the pushed data before the receiver does not read the data. However, in many cases, the better way is to press the data into a readable stream instead of caching the entire data only when the data receives the requested data. Below we will rewrite the. _ read () function:

<code class="hljs javascript">var Readable = require('stream').Readable;var rs = Readable();var c = 97;rs._read = function () {rs.push(String.fromCharCode(c++));if (c > 'z'.charCodeAt(0)) rs.push(null);};rs.pipe(process.stdout);</code><code class="hljs bash">$ node read1.jsabcdefghijklmnopqrstuvwxyz</code>

The above Code uses the override _ read () method to enable medium-pressure data input to the readable stream only when the data recipient requests data. The _ read () method can also receive a size parameter that indicates the size of the data requested by the data request, but the read stream can ignore this parameter as needed.

Note that we can also use util. inherits () to inherit readable streams. To demonstrate that the _ read () method is called only when the data recipient requests data, we implement a latency when pushing data into the readable stream, as shown below:

<code class="hljs javascript">var Readable = require('stream').Readable;var rs = Readable();var c = 97 - 1;rs._read = function () {if (c >= 'z'.charCodeAt(0)) return rs.push(null);setTimeout(function () {rs.push(String.fromCharCode(++c));}, 100);};rs.pipe(process.stdout);process.on('exit', function () {console.error('\n_read() called ' + (c - 97) + ' times');});process.stdout.on('error', process.exit);</code>

Run the program using the following command and we found that the _ read () method was called only five times:

<code class="hljs bash">$ node read2.js | head -c5abcde_read() called 5 times</code>

The timer is used because the system needs time to send a signal to notify the program to close the pipeline. Process. stdout. on ('error', fn) is used to process the SIGPIPE signal sent by the system because of the header command to close the pipeline, because this will cause process. stdout to trigger the EPIPE event. To create a Readable stream that can be pushed into any form of data, you only need to set the objectMode parameter to true when creating the stream, for example: Readable ({objectMode: true }).

2> Read readable stream Data

In most cases, we only need to simply use the pipe method to redirect the data of the readable stream to another form of stream, but in some cases, it may be more useful to directly read data from the readable stream. As follows:

<code class="hljs php">process.stdin.on('readable', function () {var buf = process.stdin.read();console.dir(buf);});$ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume0.js <buffer 0a="" 61="" 62="" 63=""><buffer 0a="" 64="" 65="" 66=""><buffer 0a="" 67="" 68="" 69="">null</buffer></buffer></buffer></code>

When data in a readable stream can be read, the stream triggers the 'readable' event, which can be called. read () method to read the relevant data. When no data in the read-only stream can be read ,. read () returns null to end. read (), waiting for the next 'readable' event to trigger. The following is an example of Reading 3 bytes from the standard input each time using. read (n:

<code class="hljs javascript">process.stdin.on('readable', function () {var buf = process.stdin.read(3);console.dir(buf);});</code>

The following code runs and finds that the output result is incomplete!

<code class="hljs bash">$ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume1.js <buffer 61="" 62="" 63=""><buffer 0a="" 64="" 65=""><buffer 0a="" 66="" 67=""></buffer></buffer></buffer></code>

This should be because extra data is left in the internal buffer of the stream, and we need to notify the stream that we want to read more data. read (0) to achieve this goal.

<code class="hljs javascript">process.stdin.on('readable', function () {var buf = process.stdin.read(3);console.dir(buf);process.stdin.read(0);});</code>

The running result is as follows:

<code class="hljs xml">$ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume2.js <buffer 0a="" 64="" 65=""><buffer 0a="" 68="" 69=""></buffer></buffer></code>

We can use. unshift () to re-forward the data to the header of the data queue, so that we can continue to read the data of the escort. The following code outputs the standard input content in line:

<code class="hljs javascript">var offset = 0;process.stdin.on('readable', function () {var buf = process.stdin.read();if (!buf) return;for (; offset < buf.length; offset++) {if (buf[offset] === 0x0a) {console.dir(buf.slice(0, offset).toString());buf = buf.slice(offset + 1);offset = 0;process.stdin.unshift(buf);return;}}process.stdin.unshift(buf);});$ tail -n +50000 /usr/share/dict/american-english | head -n10 | node lines.js 'hearties''heartiest''heartily''heartiness''heartiness\'s''heartland''heartland\'s''heartlands''heartless''heartlessly'</code>

Of course, many modules can implement this function, such as split.

3-3. writable streams

Writable streams can only be used as the target parameter of the. pipe () function. The following code:

<code class="hljs perl">src.pipe( writableStream );</code>

1> Create a writable stream

Override the. _ write (chunk, enc, next) method to accept the data of a readable stream.

<code class="hljs php">var Writable = require('stream').Writable;var ws = Writable();ws._write = function (chunk, enc, next) {console.dir(chunk);next();};process.stdin.pipe(ws);$ (echo beep; sleep 1; echo boop) | node write0.js <buffer 0a="" 62="" 65="" 70=""><buffer 0a="" 62="" 6f="" 70=""></buffer></buffer></code>

The first chunk is the data written by the data input. The second parameter end is the data encoding format. The third parameter, next (err), uses the callback function to notify the data writer that more time can be written. If readable stream writes a string, the string is converted to Buffer by default. If the Writable ({decodeStrings: false}) parameter is set during stream creation, no conversion is performed. If the data written by readable stream is an object, you need to create a writable stream in this way.

<code class="hljs css">Writable({ objectMode: true })</code>

2> write data to writable stream

Call the. write (data) method of writable stream to write data.

<code class="hljs vala">process.stdout.write('beep boop\n');</code>

Call the. end () method to notify writable stream that the data has been written.

<code class="hljs javascript">var fs = require('fs');var ws = fs.createWriteStream('message.txt');ws.write('beep ');setTimeout(function () {ws.end('boop\n');}, 1000);$ node writing1.js $ cat message.txtbeep boop</code>

If you need to set the buffer size of writable stream, you need to set opts when creating the stream. highWaterMark, so that if the data in the buffer exceeds opts. highWaterMark ,. the write (data) method returns false. When the buffer zone is writable, writable stream triggers the 'drain' event.

3-4. classic streams

Classic streams is an old interface that was first available in node 0.4, but it is still very good to understand its operating principles.
. When a stream is registered with the return function of the "data" event, the stream will work in the old version mode and use the old API.

1> classic readable streams

The Classic readable streams event is an event trigger. If Classic readable streams has data to read, it triggers the "data" event and triggers the "end" event when the data is read .. The pipe () method checks the value of stream. readable to determine whether the stream is readable. Here is an example of printing A-J letters with Classic readable streams:

<code class="hljs javascript">var Stream = require('stream');var stream = new Stream;stream.readable = true;var c = 64;var iv = setInterval(function () {if (++c >= 75) {clearInterval(iv);stream.emit('end');}else stream.emit('data', String.fromCharCode(c));}, 100);stream.pipe(process.stdout);$ node classic0.jsABCDEFGHIJ</code>

To read data from classic readable stream, register the callback functions for the "data" and "end" events. The Code is as follows:

<code class="hljs php">process.stdin.on('data', function (buf) {console.log(buf);});process.stdin.on('end', function () {console.log('__END__');});$ (echo beep; sleep 1; echo boop) | node classic1.js <buffer 0a="" 62="" 65="" 70=""><buffer 0a="" 62="" 6f="" 70="">__END__</buffer></buffer></code>

Note that if you use this method to read data, the benefits of using the new interface will be lost. For example, when writing data to a stream with a very high latency, you need to pay attention to the balance between reading data and writing data. Otherwise, a large amount of data will be cached in the memory, leading to a waste of memory. We strongly recommend that you use the. pipe () method of the stream at this time, so that you do not have to listen to the "data" and "end" events on your own, and you do not have to worry about read/write imbalance. Of course, you can also use through instead of listening to "data" and "end" events, as shown in the following code:

<code class="hljs php">var through = require('through');process.stdin.pipe(through(write, end));function write (buf) {console.log(buf);}function end () {console.log('__END__');}$ (echo beep; sleep 1; echo boop) | node through.js <buffer 0a="" 62="" 65="" 70=""><buffer 0a="" 62="" 6f="" 70="">__END__</buffer></buffer></code>

Alternatively, you can use concat-stream to cache the content of the entire stream:

<code class="hljs oxygene">var concat = require('concat-stream');process.stdin.pipe(concat(function (body) {console.log(JSON.parse(body));}));$ echo '{"beep":"boop"}' | node concat.js { beep: 'boop' }</code>

Of course, if you have to listen for "data" and "end" events by yourself, you can use it when writing data streams is not writable. the pause () method pauses Classic readable streams and continues to trigger the "data" event. When the stream for data Writing is writable, use the. resume () method to notify the stream to continue triggering the "data" event to continue reading.
Data.

2> classic writable streams

Classic writable streams is very simple. There are only three methods:. write (buf),. end (buf), and. destroy .. The buf parameter of the end (buf) method is optional. If you select this parameter, it is equivalent to stream. write (buf); stream. when the stream buffer is full, the stream cannot be written. the write (buf) method returns false. If the stream is writable again, the stream triggers the drain event.

4. transform

Transform is a stream that filters the output of read data.

5. duplex

Duplex stream is a readable and writable two-way stream. The following is a duplex stream:

<code class="hljs livecodeserver">a.pipe(b).pipe(a)</code>

The above content is the Nodejs Stream data Stream user manual introduced by xiaobian. I hope it will help you!