Node. js: Stream module _ node. js

The fs module of nodejs does not provide a copy method, but we can easily implement one. this method is to read all the file content into the memory and then write the file, this is not a big problem for small text files, such as grunt-file-copy. I. opening analysis

Stream is an abstract interface implemented by many objects in Node. For example, a request to an HTTP server is a stream, and stdout is also a stream. Stream is readable, writable, or both.

Stream was first introduced from the early unix era. decades of practice have proved that Stream can easily develop some large systems.

In unix, Stream is implemented through "|. As a built-in stream module in node, many core modules and third-party modules are used.

Like unix, the main operation of node stream is. pipe (). you can use the anti-press mechanism to control the balance between read and write.

Stream can provide developers with unified interfaces that can be reused and control the read/write balance between streams through abstract Stream interfaces.

A TCP connection is both a readable stream and a writable stream, while an Http connection is different. an http request object is a readable stream, and an http response object is a writable stream.

By default, the stream is transmitted in the form of buffer, unless you set other encoding formats for it. The following is an example:

The code is as follows:

Var http = require ('http ');
Var server = http. createServer (function (req, res ){
Res. writeHeader (200, {'content-type': 'Text/plain '});
Res. end ("Hello, Big Bear! ");
});
Server. listen (8888 );
Console. log ("http server running on port 8888 ...");

Garbled characters may occur after running, because the specified character set is not set, such as UTF-8 ".

Just modify it:

The code is as follows:

Var http = require ('http ');
Var server = http. createServer (function (req, res ){
Res. writeHeader (200 ,{
'Content-type': 'Text/plain; charset = utf-8 '// add charset = utf-8
});
Res. end ("Hello, Big Bear! ");
});
Server. listen (8888 );
Console. log ("http server running on port 8888 ...");

Running result:

Why Stream?
Node I/O is asynchronous. Therefore, data reading and writing to the disk and network needs to be read through the callback function. The following is an example of object downloading.
Code:

The code is as follows:

Var http = require ('http ');
Var fs = require ('Fs ');
Var server = http. createServer (function (req, res ){
Fs. readFile (_ dirname + '/data.txt', function (err, data ){
Res. end (data );
});
});
Server. listen (8888 );

The code can implement the required functions, but the service needs to cache the entire file data to the memory before sending the file data. if the "data.txt" file is very
If the concurrency is large, a lot of memory will be wasted. Because the user needs to wait until the entire file is cached in the memory to accept the file data, this causes
The user experience is quite poor. But fortunately, both parameters (req, res) are Stream, so we can use fs. createReadStream () instead of fs. readFile (). As follows:

The code is as follows:

Var http = require ('http ');
Var fs = require ('Fs ');
Var server = http. createServer (function (req, res ){
Var stream = fs. createReadStream (_ dirname + '/data.txt ');
Stream. pipe (res );
});
Server. listen (8888 );

The. pipe () method listens to the 'data' and 'end' events of fs. createReadStream (), so that the "data.txt" file does not need to be cached.
Files. after the client connection is complete, a data block can be sent to the client immediately. Another benefit of using. pipe () is that it can be addressed when the customer
The read/write imbalance caused by high end latency.

There are five basic streams: readable, writable, transform, duplex, and "classic ". (For specific usage, refer to the api)

II. instance introduction

When the data to be processed cannot be loaded at one time in the memory, or the data stream is used for reading and processing more efficiently. NodeJS provides operations on data streams through various streams.

Taking the big file copy program as an example, we can create a read-only data stream for the data source, for example:

The code is as follows:

Var rs = fs. createReadStream (pathname );
Rs. on ('data', function (chunk ){
DoSomething (chunk); // you can exploit the details at will.
});
Rs. on ('end', function (){
CleanUp ();
});

Data events in the code are constantly triggered by the source, regardless of whether the doSomething function can process them. The code can be modified as follows to solve this problem.

The code is as follows:

Var rs = fs. createReadStream (src );
Rs. on ('data', function (chunk ){
Rs. pause ();
DoSomething (chunk, function (){
Rs. resume ();
});
});
Rs. on ('end', function (){
CleanUp ();
});

A callback is added to the doSomething function, so we can pause data reading before processing the data and continue reading the data after processing the data.

In addition, we can also create a write-only data stream for the data target, as shown below:

The code is as follows:

Var rs = fs. createReadStream (src );
Var ws = fs. createWriteStream (dst );
Rs. on ('data', function (chunk ){
Ws. write (chunk );
});
Rs. on ('end', function (){
Ws. end ();
});

After doSomething writes data to write-only data streams, the above code looks like a file copy program. However, the above code has the problem mentioned above. if the write speed cannot keep up with the reading speed, the internal cache of the write data stream will burst. We can. the return value of the write method is used to determine whether the input data is written to the target or temporarily stored in the cache. based on the drain event, it is used to determine when only the write data stream has written the cached data to the target, you can pass in the next data to be written. The code is as follows:

The code is as follows:

Var rs = fs. createReadStream (src );
Var ws = fs. createWriteStream (dst );
Rs. on ('data', function (chunk ){
If (ws. write (chunk) === false ){
Rs. pause ();
}
});
Rs. on ('end', function (){
Ws. end ();
});
Ws. on ('drain', function (){
Rs. resume ();
});

Finally, data is transferred from read-only data streams to write-only data streams, and explosion-proof warehouse control is included. Because there are many such application scenarios, such as the large file copy program above, NodeJS directly provides the. pipe method to do this, and its internal implementation method is similar to the above code.

The following is a more complete file copy process:

The code is as follows:

Var fs = require ('Fs '),
Path = require ('PATH '),
Out = process. stdout;
Var filePath = '/bb/bigbear.mkv ';
Var readStream = fs. createReadStream (filePath );
Var writeStream = fs.createWriteStream('file.mkv ');
Var stat = fs. statSync (filePath );
Var totalSize = stat. size;
Var passedLength = 0;
Var lastSize = 0;
Var startTime = Date. now ();
ReadStream. on ('data', function (chunk ){
PassedLength + = chunk. length;
If (writeStream. write (chunk) === false ){
ReadStream. pause ();
}
});
ReadStream. on ('end', function (){
WriteStream. end ();
});
WriteStream. on ('drain', function (){
ReadStream. resume ();
});
SetTimeout (function show (){
Var percent = Math. ceil (passedLength/totalSize) * 100 );
Var size = Math. ceil (passedLength/1000000 );
Var diff = size-lastSize;
LastSize = size;
Out. clearLine ();
Out. cursorTo (0 );
Out. write ('finished '+ size + 'MB,' + percent + '%, speed:' + diff * 2 + 'MB/s ');
If (passedLength <totalSize ){
SetTimeout (show, 500 );
} Else {
Var endTime = Date. now ();
Console. log ();
Console. log ('total Time: '+ (endTime-startTime)/1000 +' seconds. ');
}
},500 );

You can save the above code as "copy. js? 7.1.2 "in the experiment, we added a recursive setTimeout (or directly use setInterval) as a bystander,

Observe the completion progress every MS, and write the completed size, percentage, and replication speed to the console. when the replication is complete, the total time consumed is calculated.

III. Summary

(1) understand the concept of Stream.

(2) skilled in using related Stream APIs

(3) pay attention to the details of the control, such as copying large files, using the "chunk data" form for sharding.

(4) use of pipe

(5) re-emphasize the concept that a TCP connection is both a readable stream and a writable stream, while an Http connection is different. an http request object is a readable stream, the http response object is a writable stream.