Big Bear nodejs------Stream Module

First, the opening analysis

A stream is an abstract interface that is implemented by many objects in Node. For example, a request to an HTTP server is a stream, and stdout is also a stream. A stream is readable, writable, or both.

The earliest contact with stream began with early Unix, and decades of practice proved that stream thought could easily develop some huge systems.

In Unix, stream is "|"  implemented. In node, as the built-in stream module, many core modules and three-party modules are used.

like UNIX, node stream The main operation is also that the user .pipe() can use the anti-pressure mechanism to control the reading and writing balance.

Stream provides developers with the ability to reuse a unified interface that controls the read-write balance between streams through an abstract stream interface.

A TCP connection is both a read and a writable stream, while an HTTP connection is different, an HTTP request object is a readable stream, and an HTTP response object is a writable stream.

The transfer process of a stream is transmitted by default in the form of buffer, unless you give him another encoding, here's an example:

1 var http = require (' http '); 2  var server = Http.createserver (function(req,res) {3     Res.writeheader (200, {') Content-type ': ' Text/plain '}); 4     Res.end ("Hello, Big Bear! ") ; 5  }) ; 6  Server.listen (8888); 7  Console.log ("HTTP server running on port 8888 ...");

After running, there will be garbled characters, the reason is that the specified character set, such as:"Utf-8" .

Just a few changes:

1  varHTTP = require (' http ') ;2  varServer = Http.createserver (function(req,res) {3Res.writeheader (200,{4' Content-type ': ' Text/plain;charset=utf-8 '//Add Charset=utf-85     }) ;6Res.end ("Hello, Big Bear! ") ;7  }) ;8Server.listen (8888) ;9Console.log ("HTTP server running on port 8888 ...");

Operation Result:

Why use Stream the I/O in node is asynchronous, so reading and writing to the disk and the network requires a callback function to read the data, and here is a file download example

On the code:

 1  var  http = require (' http ' ) ; 2  var  fs = require (' FS ' function   ( Err, data) { 5   res.end (data);  6  });  7  });  8  Server.listen (8888); 

The code can implement the required functionality, but the service needs to cache the entire file data to memory before sending the file data, if the "data.txt" file is
Large and concurrent, it wastes a lot of memory. Because the user needs to wait until the entire file is cached to memory to accept the file data, this causes
The user experience is pretty bad. But fortunately (req,res) , two parameters are stream, so we can use fs.createReadStream() insteadfs.readFile()。如下：

1 var http = require (' http '); 2 var fs = require (' FS '); 3 var server = Http.createserver (function  (req, res) {4     var stream = Fs.createreadstream (__dirname + '/data.txt '); 5     Stream.pipe (res); 6 }) ; 7 Server.listen (8888);

.pipe()Method fs.createReadStream() ‘data‘ ‘end‘ , so that the "data.txt" file does not need to cache the entire
Files, you can send a block of data to the client immediately after the client connection is complete. .pipe() The use of another benefit is that it can be resolved when the customer
The problem of read-write imbalance caused by very large end-delay.

There are five kinds of basic Stream:readable,writable,transform,duplex,and "classic". (Please check the API for your own use)

Second, the introduction of examples

We need to use the data stream when the data that needs to be processed is not loaded in memory at one time, or when processing is more efficient while reading. The Nodejs provides the Stream operation of the data stream through various.

As an example of a large file copy program, we can create a read-only data stream for the data source, such as the following:

1 var rs = fs.createreadstream (pathname); 2 function (chunk) {4     // specific details of their own arbitrary play 5 }); 6 function () {8    cleanUp (); 9 });

Events in the code are constantly data triggered, regardless of doSomething whether the function is handled. The code can continue to do the following modifications to solve this problem.

1 varrs =Fs.createreadstream (SRC);2Rs.on (' Data ',function(chunk) {3 rs.pause ();4DoSomething (Chunk,function () {5 Rs.resume ();6     }) ;7 }) ;8Rs.on (' End ',function () {9 cleanUp ();Ten})  ;

doSomethinga callback is added to the function, so we can pause the data read before processing the data and continue to read the data after processing the data.

In addition, we can create a write-only data stream for the data target, as follows:

 1  var  rs =  Fs.createreadstream (SRC);  2  var  ws =  Fs.createwritestream (DST);  3  rs.on (' data ', function   (Chunk) { 4   Ws.write (chunk);  5  });  6  rs.on (' End ', function   () { 7   Ws.end ();  8 }); 

doSomethingInstead of writing data into a write-only stream, the above code looks like a file-copy program. However, the above code has the above mentioned problem, if the write speed can not keep up with the reading speed, the write-only data flow inside the cache will explode. Depending on .write The return value of the method, we can determine whether the incoming data is written to the target, or temporarily put it in the cache, and according to the drain event to determine when the write-only data stream has already written the data in the cache to the target, you can pass in the next data to be written. So the code looks like this:

1 varrs =Fs.createreadstream (SRC);2 varWS =Fs.createwritestream (DST);3Rs.on (' Data ',function(chunk) {4     if(Ws.write (chunk) = = =false) {5 rs.pause ();6     }7 }) ;8Rs.on (' End ',function () {9 ws.end ();Ten }); OneWs.on (' drain ',function () { A Rs.resume (); -}) ;

Finally, the data flow from read-only data to write-only data flow is carried out, and includes the explosion-proof warehouse control. Because of this many scenarios, such as the large file copy program on top, Nodejs directly provides a .pipe way to do this, and its internal implementation is similar to the above code.

The following is a more complete process for copying files:

1 varFS = require (' FS '),2Path = require (' path '),3out =process.stdout;4 5 varFilePath = '/bb/bigbear.mkv ';6 7 varReadstream =Fs.createreadstream (filePath);8 varWritestream = Fs.createwritestream (' file.mkv '));9 Ten varStat =Fs.statsync (filePath); One  A varTotalSize =stat.size; - varPassedlength = 0; - varLastsize = 0; the varStartTime =Date.now (); -  -Readstream.on (' Data ',function(chunk) { -  +Passedlength + =chunk.length; -  +   if(Writestream.write (chunk) = = =false) { A readstream.pause (); at   } - }); -  -Readstream.on (' End ',function() { - writestream.end (); - }); in  -Writestream.on (' drain ',function() { to Readstream.resume (); + }); -  theSetTimeout (functionShow () { *   varPercent = Math.ceil ((passedlength/totalsize) * 100); $   varSize = Math.ceil (passedlength/1000000);Panax Notoginseng   vardiff = size-lastsize; -Lastsize =size; the out.clearline (); +Out.cursorto (0); AOut.write (' completed ' + size + ' MB, ' + percent + '%, Speed: ' + diff * 2 + ' MB/s '); the   if(Passedlength <totalsize) { +SetTimeout (show, 500); -}Else { $     varEndTime =Date.now (); $ Console.log (); -Console.log (' Total time: ' + (Endtime-starttime)/1000 + ' seconds. ‘); -   } the}, 500);

You can save the above code as " copy.js" experiment We added a recursive setTimeout (or use setinterval directly) to be a bystander,

Each 500ms observes the completion progress and writes the completed size, percentage, and replication speed to the console, and when the copy is complete, the total time spent is calculated.

Three, summarize

(1), understand the stream concept.

(2), skilled in using the relevant stream API

(3), pay attention to the details of the control, such as: Large file copy, using the "chunk data" in the form of fragmented processing.

(4) The use of pipe

(5), again emphasizing a concept: A TCP connection is both a readable stream and a writable stream, while an HTTP connection is different, an HTTP request object is a readable stream, and an HTTP response object is a writable stream.

hahaha, the end of this article, not to be continued, hope and we have enough communication, common progress ... To shout ... (*^__^*)                     

Big Bear nodejs------Stream Module