NodejsStream data stream user manual _ node. js-js tutorial

This article mainly introduces the NodejsStream data stream user manual. If you are interested, learn it together. 1. Introduction

This article describes the basic methods for developing programs using node. js streams.

"We should have some ways of connecting programs like garden hose--screw inanother segment when it becomes necessary to massage data inanother way. This is the way of IO also."Doug McIlroy. October 11, 1964

Stream is the first practice that has been used for decades since the beginning of unix. It has proved that Stream can easily develop some large systems. In unix, Stream is implemented through |. In node, as a built-in stream module, many core modules and third-party modules are used. Like unix, the main operation of node Stream is. pipe (). You can use the anti-press mechanism to control the balance between read and write.

Stream can provide developers with unified interfaces that can be reused and control the read/write balance between streams through abstract Stream interfaces.

2. Why Stream?

Node I/O is asynchronous. Therefore, the disk and network read and write operations need to be read through the callback function. Below is a simple code of the file download server:

var http = require('http');var fs = require('fs');var server = http.createServer(function (req, res) {fs.readFile(__dirname + '/data.txt', function (err, data) {res.end(data);});});server.listen(8000);

These codes can implement the required functions, but the service needs to cache the entire file data to the memory before sending the file data. If the "data.txt" file is large and the concurrency is large, it will waste a lot of memory. This is because the user needs to wait until the entire file is cached in the memory to receive the file data, which leads to a poor user experience. But fortunately, both parameters (req, res) are Stream, so we can use fs. createReadStream () instead of fs. readFile ():

var http = require('http');var fs = require('fs');var server = http.createServer(function (req, res) {var stream = fs.createReadStream(__dirname + '/data.txt');stream.pipe(res);});server.listen(8000);

. The pipe () method listens to fs. the 'data' and 'end' events of createReadStream (), so that the "data.txt" file does not need to cache the entire file. After the client connection is complete, a data block can be sent to the client immediately. Another advantage of using. pipe () is that it can solve the read/write imbalance problem caused by high client latency. If you want to compress the file and try again, you can use the third-party module:

var http = require('http');var fs = require('fs');var oppressor = require('oppressor');var server = http.createServer(function (req, res) {var stream = fs.createReadStream(__dirname + '/data.txt');stream.pipe(oppressor(req)).pipe(res);});server.listen(8000);

In this way, the files will be compressed on browsers that support gzip and deflate. The oppressor module processes all content-encoding.

Stream makes development programs easy.

3. Basic Concepts

There are five basic streams: readable, writable, transform, duplex, and "classic ".

3-1. pipe

All Stream types use. pipe () to create an input/output pair, receive a readable Stream src, and output its data to the writable Stream dst, as shown below:

src.pipe(dst)

The. pipe (dst) method returns the dst stream, so that multiple. pipe () can be successively used, as shown below:

a.pipe( b ).pipe( c ).pipe( d )

The function is the same as the following code:

a.pipe( b );b.pipe( c );c.pipe( d );

3-2. readable streams

You can call the. pipe () method of Readable streams to write data of Readable streams into a Writable, Transform, or Duplex stream.

readableStream.pipe( dst )

1> Create a readable stream

Here we create a readable stream!

var Readable = require('stream').Readable;var rs = new Readable;rs.push('beep ');rs.push('boop\n');rs.push(null);rs.pipe(process.stdout);$ node read0.jsbeep boop

Rs. push (null) notifies the data recipient that the data has been sent.

We noticed that rs was not called before we pushed all the data into the readable stream. pipe (process. stdout); but all the data content we press into is completely output, because the readable stream caches all the pushed data before the receiver does not read the data. However, in many cases, the better way is to press the data into a readable stream instead of caching the entire data only when the data receives the requested data. Below we will rewrite the. _ read () function:

var Readable = require('stream').Readable;var rs = Readable();var c = 97;rs._read = function () {rs.push(String.fromCharCode(c++));if (c > 'z'.charCodeAt(0)) rs.push(null);};rs.pipe(process.stdout);$ node read1.jsabcdefghijklmnopqrstuvwxyz

The above Code uses the override _ read () method to enable medium-pressure data input to the readable stream only when the data recipient requests data. The _ read () method can also receive a size parameter that indicates the size of the data requested by the data request, but the read stream can ignore this parameter as needed.

Note that we can also use util. inherits () to inherit readable streams. To demonstrate that the _ read () method is called only when the data recipient requests data, we implement a latency when pushing data into the readable stream, as shown below:

var Readable = require('stream').Readable;var rs = Readable();var c = 97 - 1;rs._read = function () {if (c >= 'z'.charCodeAt(0)) return rs.push(null);setTimeout(function () {rs.push(String.fromCharCode(++c));}, 100);};rs.pipe(process.stdout);process.on('exit', function () {console.error('\n_read() called ' + (c - 97) + ' times');});process.stdout.on('error', process.exit);

Run the program using the following command and we found that the _ read () method was called only five times:

$ node read2.js | head -c5abcde_read() called 5 times

The timer is used because the system needs time to send a signal to notify the program to close the pipeline. Process. stdout. on ('error', fn) is used to process the SIGPIPE signal sent by the system because of the header command to close the pipeline, because this will cause process. stdout to trigger the EPIPE event. To create a Readable stream that can be pushed into any form of data, you only need to set the objectMode parameter to true when creating the stream, for example: Readable ({objectMode: true }).

2> Read readable stream Data

In most cases, we only need to simply use the pipe method to redirect the data of the readable stream to another form of stream, but in some cases, it may be more useful to directly read data from the readable stream. As follows:

process.stdin.on('readable', function () {var buf = process.stdin.read();console.dir(buf);});$ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume0.js 
  
   
    
     null
    
   
  

When data in a readable stream can be read, the stream triggers the 'readable' event, which can be called. read () method to read the relevant data. When no data in the read-only stream can be read ,. read () returns null to end. read (), waiting for the next 'readable' event to trigger. The following is an example of Reading 3 bytes from the standard input each time using. read (n:

process.stdin.on('readable', function () {var buf = process.stdin.read(3);console.dir(buf);});

The following code runs and finds that the output result is incomplete!

$ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume1.js 
  
   
    
   
  

This should be because extra data is left in the internal buffer of the stream, and we need to notify the stream that we want to read more data. read (0) to achieve this goal.

process.stdin.on('readable', function () {var buf = process.stdin.read(3);console.dir(buf);process.stdin.read(0);});

The running result is as follows:

$ (echo abc; sleep 1; echo def; sleep 1; echo ghi) | node consume2.js 
  
   
  

We can use. unshift () to re-forward the data to the header of the data queue, so that we can continue to read the data of the escort. The following code outputs the standard input content in line:

var offset = 0;process.stdin.on('readable', function () {var buf = process.stdin.read();if (!buf) return;for (; offset < buf.length; offset++) {if (buf[offset] === 0x0a) {console.dir(buf.slice(0, offset).toString());buf = buf.slice(offset + 1);offset = 0;process.stdin.unshift(buf);return;}}process.stdin.unshift(buf);});$ tail -n +50000 /usr/share/dict/american-english | head -n10 | node lines.js 'hearties''heartiest''heartily''heartiness''heartiness\'s''heartland''heartland\'s''heartlands''heartless''heartlessly'

Of course, many modules can implement this function, such as split.

3-3. writable streams

Writable streams can only be used as the target parameter of the. pipe () function. The following code:

src.pipe( writableStream );

1> Create a writable stream

Override the. _ write (chunk, enc, next) method to accept the data of a readable stream.

var Writable = require('stream').Writable;var ws = Writable();ws._write = function (chunk, enc, next) {console.dir(chunk);next();};process.stdin.pipe(ws);$ (echo beep; sleep 1; echo boop) | node write0.js 
  
   
  

The first chunk is the data written by the data input. The second parameter end is the data encoding format. The third parameter, next (err), uses the callback function to notify the data writer that more time can be written. If readable stream writes a string, the string is converted to Buffer by default. If the Writable ({decodeStrings: false}) parameter is set during stream creation, no conversion is performed. If the data written by readable stream is an object, you need to create a writable stream in this way.

Writable({ objectMode: true })

2> write data to writable stream

Call the. write (data) method of writable stream to write data.

process.stdout.write('beep boop\n');

Call the. end () method to notify writable stream that the data has been written.

var fs = require('fs');var ws = fs.createWriteStream('message.txt');ws.write('beep ');setTimeout(function () {ws.end('boop\n');}, 1000);$ node writing1.js $ cat message.txtbeep boop

If you need to set the buffer size of writable stream, you need to set opts when creating the stream. highWaterMark, so that if the data in the buffer exceeds opts. highWaterMark ,. the write (data) method returns false. When the buffer zone is writable, writable stream triggers the 'drain' event.

3-4. classic streams

Classic streams is an old interface that was first available in node 0.4, but it is still very good to understand its operating principles.
. When a stream is registered with the return function of the "data" event, the stream will work in the old version mode and use the old API.

1> classic readable streams

The Classic readable streams event is an event trigger. If Classic readable streams has data to read, it triggers the "data" event and triggers the "end" event when the data is read .. The pipe () method checks the value of stream. readable to determine whether the stream is readable. Here is an example of printing A-J letters with Classic readable streams:

var Stream = require('stream');var stream = new Stream;stream.readable = true;var c = 64;var iv = setInterval(function () {if (++c >= 75) {clearInterval(iv);stream.emit('end');}else stream.emit('data', String.fromCharCode(c));}, 100);stream.pipe(process.stdout);$ node classic0.jsABCDEFGHIJ

To read data from classic readable stream, register the callback functions for the "data" and "end" events. The Code is as follows:

process.stdin.on('data', function (buf) {console.log(buf);});process.stdin.on('end', function () {console.log('__END__');});$ (echo beep; sleep 1; echo boop) | node classic1.js 
  
   
    __END__
   
  

Note that if you use this method to read data, the benefits of using the new interface will be lost. For example, when writing data to a stream with a very high latency, you need to pay attention to the balance between reading data and writing data. Otherwise, a large amount of data will be cached in the memory, leading to a waste of memory. We strongly recommend that you use the. pipe () method of the stream at this time, so that you do not have to listen to the "data" and "end" events on your own, and you do not have to worry about read/write imbalance. Of course, you can also use through instead of listening to "data" and "end" events, as shown in the following code:

var through = require('through');process.stdin.pipe(through(write, end));function write (buf) {console.log(buf);}function end () {console.log('__END__');}$ (echo beep; sleep 1; echo boop) | node through.js 
  
   
    __END__
   
  

Alternatively, you can use concat-stream to cache the content of the entire stream:

var concat = require('concat-stream');process.stdin.pipe(concat(function (body) {console.log(JSON.parse(body));}));$ echo '{"beep":"boop"}' | node concat.js { beep: 'boop' }

Of course, if you have to listen for "data" and "end" events by yourself, you can use it when writing data streams is not writable. the pause () method pauses Classic readable streams and continues to trigger the "data" event. When the stream for data Writing is writable, use the. resume () method to notify the stream to continue triggering the "data" event to continue reading.
Data.

2> classic writable streams

Classic writable streams is very simple. There are only three methods:. write (buf),. end (buf), and. destroy .. The buf parameter of the end (buf) method is optional. If you select this parameter, it is equivalent to stream. write (buf); stream. when the stream buffer is full, the stream cannot be written. the write (buf) method returns false. If the stream is writable again, the stream triggers the drain event.

4. transform

Transform is a stream that filters the output of read data.

5. duplex

Duplex stream is a readable and writable two-way stream. The following is a duplex stream:

a.pipe(b).pipe(a)

The above content is the Nodejs Stream data Stream user manual introduced by xiaobian. I hope it will help you!