Nodejs Learning Notes Stream module _node.js

First, the opening analysis

A stream is an abstract interface that is implemented by many objects in the Node. For example, a request to an HTTP server is a stream, and stdout is also a stream. The stream is readable, writable, or both.

The first contact stream began in early Unix, and decades of practice have proved that stream thought can be very simple to develop a number of large systems.

In Unix, the stream is implemented through "|". In node, as the built-in stream module, many core modules and three-party modules are used.

As with UNIX, the main operation of node stream is also. Pipe (), users can use the reverse pressure mechanism to control the reading and writing balance.

The stream can provide developers with a reusable, unified interface to control the read-write balance between the stream through an abstract stream interface.

A TCP connection is both readable and writable, while an HTTP connection is different, an HTTP request object is a readable stream, and an HTTP response object is a writable stream.

The streaming process is transmitted by default in the form of buffer, unless you give him another form of encoding, here is an example:

Copy Code code as follows:

var http = require (' http ');
var server = Http.createserver (function (req,res) {
Res.writeheader ({' Content-type ': ' Text/plain '});
Res.end ("Hello, Big Bear!") ") ;
}) ;
Server.listen (8888);
Console.log ("HTTP server running on port 8888 ...");

After running, there will be garbled appearance, because the specified character set is not set, for example, "Utf-8".

Just change it:

Copy Code code as follows:

var http = require (' http ');
var server = Http.createserver (function (req,res) {
Res.writeheader (200,{
' Content-type ': ' Text/plain;charset=utf-8 '//Add Charset=utf-8
}) ;
Res.end ("Hello, Big Bear!") ") ;
}) ;
Server.listen (8888);
Console.log ("HTTP server running on port 8888 ...");

Run Result:

Why use Stream
I/O in node is asynchronous, so reading and writing to disk and network requires a callback function to read the data, the following is a file download example
Code on:

Copy Code code as follows:

var http = require (' http ');
var fs = require (' FS ');
var server = Http.createserver (function (req, res) {
Fs.readfile (__dirname + '/data.txt ', function (err, data) {
Res.end (data);
}) ;
}) ;
Server.listen (8888);

The code can implement the required functionality, but the service needs to cache the entire file data to memory before sending the file data, if the "data.txt" file is
Large and with a large amount of concurrency, it will waste a lot of memory. Because the user needs to wait until the entire file is cached to memory to accept the file data, which causes
The user experience is rather bad. But fortunately (req,res) two parameters are stream, so we can use Fs.createreadstream () instead of Fs.readfile (). As follows:

Copy Code code as follows:

var http = require (' http ');
var fs = require (' FS ');
var server = Http.createserver (function (req, res) {
var stream = Fs.createreadstream (__dirname + '/data.txt ');
Stream.pipe (RES);
}) ;
Server.listen (8888);

The. Pipe () method listens for the ' data ' and ' end ' events of Fs.createreadstream () so that "data.txt" files do not need to cache the whole
file that can send a block of data to the client immediately after the client connection completes. Using. Pipe () Another benefit is that it can be resolved when the customer
The problem of read and write imbalance caused by the very high end delay.

There are five basic Stream:readable,writable,transform,duplex,and "classic". (Please consult your own API for specific use)

Second, the example introduces

We need to use the data stream when it is not possible to load the data that needs to be processed at once in memory, or when the processing is more efficient while reading one side. NODEJS provides operations on the flow of data through a variety of stream streams.

In the case of a large file copy program, we can create a read-only stream for the data source, as shown in the following example:

Copy Code code as follows:

var rs = fs.createreadstream (pathname);
Rs.on (' Data ', function (chunk) {
DoSomething (chunk); Specific details of their own arbitrary play
});
Rs.on (' End ', function () {
CleanUp ();
}) ;

The data event in the code is constantly triggered, regardless of whether the dosomething function is handled or not. The code can continue to do the following modifications to solve the problem.

Copy Code code as follows:

var rs = fs.createreadstream (src);
Rs.on (' Data ', function (chunk) {
Rs.pause ();
DoSomething (chunk, function () {
Rs.resume ();
}) ;
}) ;
Rs.on (' End ', function () {
CleanUp ();
})  ;

A callback is added to the dosomething function, so we can pause the data reading before processing the data and continue to read the data after processing the data.

In addition, we can create a write-only data stream for the data target, as follows:

Copy Code code as follows:

var rs = fs.createreadstream (src);
var ws = Fs.createwritestream (DST);
Rs.on (' Data ', function (chunk) {
Ws.write (chunk);
}) ;
Rs.on (' End ', function () {
Ws.end ();
}) ;

DoSomething instead of writing data to a write-only stream, the above code looks like a file copy program. However, the above code is the problem mentioned above, if the write speed can not keep up with the reading speed, write-only data flow inside the cache will burst. We can use the return value of the. Write method to determine whether the incoming data was written to the target, or was temporarily cached, and based on the drain event to determine when the write-only data stream has written the data in the cache to the target, and can pass in the next pending data. So the code is as follows:

Copy Code code as follows:

var rs = fs.createreadstream (src);
var ws = Fs.createwritestream (DST);
Rs.on (' Data ', function (chunk) {
if (ws.write (chunk) = = False) {
Rs.pause ();
}
}) ;
Rs.on (' End ', function () {
Ws.end ();
});
Ws.on (' Drain ', function () {
Rs.resume ();
}) ;

Finally, the data flow from the read-only data stream to the write-only data stream is implemented, and the explosion-proof silo control is included. Because this is a lot of usage scenarios, such as the large file copy program above, Nodejs directly provides a. Pipe method to do this, and its internal implementation is similar to the code above.

The following is a more complete procedure for copying files:

Copy Code code as follows:

var fs = require (' FS '),


Path = require (' path '),


out = Process.stdout;


var filePath = '/bb/bigbear.mkv ';


var readstream = Fs.createreadstream (FilePath);


var writestream = fs.createwritestream (' file.mkv ');


var stat = Fs.statsync (FilePath);


var totalsize = stat.size;


var passedlength = 0;


var lastsize = 0;


var starttime = Date.now ();


Readstream.on (' Data ', function (chunk) {


Passedlength + = Chunk.length;


if (writestream.write (chunk) = = False) {


Readstream.pause ();


}


});


Readstream.on (' End ', function () {


Writestream.end ();


});


Writestream.on (' Drain ', function () {


Readstream.resume ();


});


SetTimeout (function Show () {


var percent = Math.ceil ((passedlength/totalsize) * 100);


var size = Math.ceil (passedlength/1000000);


var diff = size-lastsize;


lastsize = size;


Out.clearline ();


Out.cursorto (0);


Out.write (' completed ' + size + ' MB, ' + percent + '%, Speed: ' + diff * 2 + ' MB/s ');


if (Passedlength < totalsize) {


SetTimeout (show, 500);


} else {


var endtime = Date.now ();


Console.log ();


Console.log (' Shared: ' + (Endtime-starttime)/1000 + ' sec. ');


}


}, 500);

You can save the above code as "copy.js." We added a recursive settimeout (or directly using setinterval) to be a bystander,

The completed progress is observed at 500ms, and the completed size, percentage, and copy speed are written to the console, and the total time spent is calculated when the copy is complete.

Third, to sum up

(1) Understanding the stream concept.

(2) Proficiency in the use of related stream API

(3), attention to the details of the control, such as: Large file copy, using the "chunk data" in the form of fragmentation processing.

(4), the use of pipe

(5) Again, emphasize the concept that a TCP connection is both readable and writable, while HTTP connections are different, an HTTP request object is a readable stream, and an HTTP response object is a writable stream.