Node. js ------ Stream module

Node. js ------ Stream module
1. The analysis stream at the beginning is an abstract interface implemented by many objects in Node. For example, a request to an HTTP server is a stream, and stdout is also a stream. Stream is readable, writable, or both. Stream was first introduced from the early unix era. Decades of practice have proved that Stream can easily develop some large systems. In unix, Stream is implemented through "|. As a built-in stream module in node, many core modules and third-party modules are used. Like unix, the main operation of node stream is. pipe (). You can use the anti-press mechanism to control the balance between read and write. Stream can provide developers with unified interfaces that can be reused and control the read/write balance between streams through abstract Stream interfaces. A TCP connection is both a readable stream and a writable stream, while an Http connection is different. An http request object is a readable stream, and an http response object is a writable stream. By default, the stream is transmitted in the form of buffer, unless you set other encoding formats for it. The following is an example: 1 var http = require ('http '); 2 var server = http. createServer (function (req, res) {3 res. writeHeader (200, {'content-type': 'text/plain '}); 4 res. end ("Hello, Big Bear! "); 5}); 6 server. listen (8888); 7 console. log ("http server running on port 8888... "). garbled characters may occur after running, because the specified character set is not set, for example," UTF-8 ". Just modify it: 1 var http = require ('HTTP '); 2 var server = http. createServer (function (req, res) {3 res. writeHeader (200, {4 'content-type': 'text/plain; charset = UTF-8 '// Add charset = utf-85}); 6 res. end ("Hello, Big Bear! "); 7}); 8 server. listen (8888); 9 console. log ("http server running on port 8888... "); why is I/O in Streamnode asynchronous? Therefore, data reading and writing to the disk and network needs to be read through the callback function. The following is an example of code for downloading a file: 1 var http = require ('http'); 2 var fs = require ('fs'); 3 var server = http. createServer (function (req, res) {4 fs. readFile (_ dirname + '/data.txt', function (err, data) {5 res. end (data); 6}); 7}); 8 server. listen (8888); the Code can implement the required functions, but the service is sending file data Previously, the entire file data needs to be cached to the memory. If the "data.txt" file is large and the concurrency is large, it will waste a lot of memory. This is because the user needs to wait until the entire file is cached in the memory to receive the file data, which leads to a poor user experience. But fortunately, both parameters (req, res) are Stream, so we can use fs. createReadStream () instead of fs. readFile (). 1 var http = require ('http'); 2 var fs = require ('fs'); 3 var server = http. createServer (function (req, res) {4 var stream = fs. createReadStream (_ dirname + '/data.txt'); 5 stream. pipe (res); 6}); 7 server. listen (8888 );. the pipe () method listens to fs. the 'data' and 'end' events of createReadStream (), so that the "data.txt" file does not need to cache the entire file. After the client connection is complete, a data block can be sent to the client immediately. Another advantage of using. pipe () is that it can solve the read/write imbalance problem caused by high client latency. There are five basic streams: readable, writable, transform, duplex, and "classic ". (For specific usage, see the api.) 2. When the instance is introduced, when the data to be processed cannot be loaded at one time in the memory, or when reading data while processing is more efficient, we need to use data streams. NodeJS provides operations on data streams through various streams. Take the big file copy program as an example. We can create a read-only data stream for the data source, for example: 1 var rs = fs. createReadStream (pathname); 2 rs. on ('data', function (chunk) {4 doSomething (chunk); // you can play the details yourself 5}); 6 rs. on ('end', function () {8 cleanUp (); 9}); in the code, data Events are continuously triggered by the source, regardless of whether the doSomething function can process them. The code can be modified as follows to solve this problem. 1 var rs = fs. createReadStream (src); 2 rs. on ('data', function (chunk) {3 rs. pause (); 4 doSomething (chunk, function () {5 rs. resume (); 6}); 7}); 8 rs. on ('end', function () {9 cleanUp (); 10}); adds a callback to the doSomething function, so we can pause Data Reading before processing data, and continue reading data after processing the data. In addition, we can also create a write-only data stream for the data target, as shown below: 1 var rs = fs. createReadStream (src); 2 var ws = fs. createWriteStream (dst); 3 rs. on ('data', function (chunk) {4 ws. write (chunk); 5}); 6 rs. on ('end', function () {7 ws. end (); 8}); after doSomething is replaced by writing data to write-only data streams, the above Code looks like a file copy program. However, the above Code has the problem mentioned above. If the write speed cannot keep up with the reading speed, the internal cache of the write data stream will burst. We can. the Return Value of the write method is used to determine whether the input data is written to the target or temporarily stored in the cache. Based on the drain event, it is used to determine when only the write data stream has written the cached data to the target, you can pass in the next data to be written. The Code is as follows: 1 var rs = fs. createReadStream (src); 2 var ws = fs. createWriteStream (dst); 3 rs. on ('data', function (chunk) {4 if (ws. write (chunk) ==== false) {5 rs. pause (); 6} 7}); 8 rs. on ('end', function () {9 ws. end (); 10}); 11 ws. on ('drain', function () {12 rs. resume (); 13}); finally, data is transferred from read-only data streams to write-only data streams, and explosion-proof warehouse control is included. Because there are many such application scenarios, such as the large file copy program above, NodeJS directly provides the. pipe method to do this, and its internal implementation method is similar to the above Code. The following is a more complete file copy process: 1 var fs = require ('fs'), 2 path = require ('path'), 3 out = process. stdout; 4 5 var filePath = '/bb/bigbear.mkv'; 6 7 var readStream = fs. createReadStream (filePath); 8 var writeStream = fs.createWriteStream('file.mkv '); 9 10 var stat = fs. statSync (filePath); 11 12 var totalSize = stat. size; 13 var passedLength = 0; 14 var lastSize = 0; 15 var startTime = Date. now (); 16 17 readStream. On ('data', function (chunk) {18 19 passedLength + = chunk. length; 20 21 if (writeStream. write (chunk) ==== false) {22 readStream. pause (); 23} 24}); 25 26 readStream. on ('end', function () {27 writeStream. end (); 28}); 29 30 writeStream. on ('drain', function () {31 readStream. resume (); 32}); 33 34 setTimeout (function show () {35 var percent = Math. ceil (passedLength/totalSize) * 100); 36 var size = Math. ceil (pa SsedLength/1000000); 37 var diff = size-lastSize; 38 lastSize = size; 39 out. clearLine (); 40 out. cursorTo (0); 41 out. write ('finished '+ size + 'mb,' + percent + '%, speed:' + diff * 2 + 'mb/s '); 42 if (passedLength <totalSize) {43 setTimeout (show, 500); 44} else {45 var endTime = Date. now (); 46 console. log (); 47 console. log ('total time: '+ (endTime-startTime)/1000 +' seconds. '); 48} 49}, 500); you can save the above Code as "copy. js "test: we have added a recursive setTimeout (or directly use setInterval) as a bystander, and observe the progress every Ms, and write the completed size, percentage, and replication speed together to the console. When the replication is completed, the total time consumed is calculated. 3. Summarize (1) and understand the concept of Stream. (2) familiar with the relevant Stream api (3), pay attention to the details of the control, such as: copying large files, using the "chunk data" form for sharding. (4) pipe usage (5), again emphasizing the concept that a TCP connection is both a readable stream and a writable stream, while an Http connection is different, an http request object is a readable stream, while an http response object is a writable stream.