node. JS Stream-Basic article
Bingbin · 2016-07-08 11:51
Background
When building a more complex system, it is often disassembled into several parts that are functionally independent. The interfaces of these parts follow certain specifications and are connected in some way to accomplish more complex tasks together. For example, the shell |
connects parts through a pipeline, and its input and output specifications are text streams.
In node. js, the built-in stream module also implements a similar function, with sections .pipe()
connected.
In view of the fact that there are fewer articles on stream in the domestic system, and that more and more open source tools are using stream, this series of articles will introduce the relevant content from the following aspects:
- Basic types of streams, and basic usage of stream modules
- How streaming and back pressure work
- How to develop a streaming program, including an analysis of Gulp and browserify, and a practical example.
This article is the first article in a series of articles.
Four types of streams
Stream provides the following four types of streams:
var Stream = require (' stream ')var readable = stream.readablevar writable = Stream.writablevar Duplex = Stream.duplexvar Transform = Stream.transform
Use Stream
streaming that enables data, such as:
var fs = require (' FS ')// ' fs.createreadstream ' creates a ' readable ' object to read ' Bigfile ' Content, and output to standard output // if using ' fs.readfile ' may fail Fs.createreadstream (Bigfile) due to a large file size . PIPE ( Process.stdout)
Readable
Create a readable stream.
Example: Streaming consumes data from an iterator.
' Use strict 'Const readable= require (' stream '). Readableclass Toreadable extends Readable {constructor (iterator) {super () This. iterator =iterator}//subclasses need to implement this method //This is the logic of production data_read () {Const RES= This. Iterator.next ()if(res.done) {//data source is exhausted, call ' push (NULL) ' Notification flow return This. Push (NULL)} setTimeout (()= { //adding data to a stream through the ' push ' method This. push (Res.value + ' \ n ')) }, 0)}}module.exports= Toreadable
When actually used, new ToReadable(iterator)
a readable stream is returned, downstream of which the data in the iterator can be streamed.
Const ITERATOR =function(limit) {return{Next:function () { if(limit--) { return{done:false, Value:limit +math.random ()}} return{done:true } } }}(1e10) Const readable=Newtoreadable (iterator)//listen for the ' data ' event and get one data at a timeReadable.on (' data ', data =process.stdout.write (data))//all data is read outReadable.on (' End ', () = Process.stdout.write (' Done '))
Execute the above code, there will be 10 billion random numbers are continuously written into the standard output stream.
When you create a readable stream, you need to inherit and Readable
implement the _read
method.
_read
The method is to read the logic of the specific data from the underlying system, i.e. the logic of the production data.
- In a
_read
method, the push(data)
data is placed in a readable stream by a call for downstream consumption.
- In a
_read
method, you can call synchronously push(data)
or asynchronously.
- When all the data is produced, it must be called
push(null)
to end the readable stream.
- Once the stream has ended, it is no longer possible to invoke the
push(data)
add data.
You can data
consume readable streams by listening for events.
- After the first time the event is monitored, it is continuously
data
readable
invoked _read()
to data
output the data by triggering the event.
- The first
data
event is triggered in the next tick, so it is safe to place the logic before the data output in the event listener (in the same tick).
- Events are triggered when the data is all consumed
end
.
In the example above, it process.stdout
represents the standard output stream, which is actually a writable stream. The use of a writable stream is described in the next section.
Writable
Creates a writable stream.
The first way to create a class of readable streams is through inheritance, which is also useful for creating a class of writable streams, but it is only a _write(data, enc, next)
method, not a method, that needs to be implemented _read()
.
There are some simple cases where you don't need to create a class of streams, but just a stream object that you can do in the following ways:
Const WRITABLE = require (' stream '). Writableconst writable=writable ()//implementing the ' _write ' approach//This is the logic that writes the data to the underlyingWritable._write =function(data, enc, next) {//writes data from the stream to the underlyingProcess.stdout.write (Data.tostring (). toUpperCase ())//when the write is complete, call the ' next () ' method to notify that the next data is passed inProcess.nexttick (Next)}//all data is written to the underlyingWritable.on (' Finish ', () = Process.stdout.write (' Done ')))//writes a data to the streamWritable.write (' a ' + ' \ n ')) Writable.write (' B ' + ' \ n ') Writable.write (' C ' + ' \ n ')//the ' end ' method needs to be called when no data is written to the streamWritable.end ()
- Upstream
writable.write(data)
writes data to a writable stream by calling. write()
method is called _write()
to data
write to the underlying.
- In
_write
, when the data is successfully written to the underlying, you must call the next(err)
tell stream to start processing the next data.
next
The call can be either synchronous or asynchronous.
- Upstream must be called
writable.end(data)
to end a writable stream, which data
is optional. After that, the new data can no longer be called write
.
end
after a method call, an event is triggered when all underlying writes are completed finish
.
Duplex
Creates a readable writable stream.
Duplex
is actually a Readable
class of streams that inherit the same Writable
.
Therefore, an Duplex
object can be used as a readable stream (Implementation method required _read
) or as a writable stream (requires implementation _write
method).
varDuplex = require (' stream '). DuplexvarDuplex =Duplex ()//Readable end-of-layer read logicDuplex._read =function () { This. _readnum = This. _readnum | | 0if( This. _readnum > 1) { This. Push (NULL) } Else { This. Push ("+" ( This. _readnum++)) }}//Writable end-of-layer write logicDuplex._write =function(buf, enc, next) {//A, BProcess.stdout.write (' _write ' + buf.tostring () + ' \ n ') Next ()}//0, 1Duplex.on (' data ', data = Console.log (' OnData '), Data.tostring ())) Duplex.write (A) Duplex.write (' B ') duplex.end ()
The method is implemented in the code above _read
, so you can listen for data
events to consume Duplex
the resulting data.
At the same time, the method is realized _write
, which can be used as downstream consumption data.
Because it is both readable and writable, it is said to have two ends: the writable and the readable end.
The interface of the writable end is Writable
consistent, used as downstream, and the interface of the readable end is Readable
consistent, and is used as upstream.
Transform
In the above example, the data in the readable stream (0, 1) is separated from the data in the writable stream (' A ', ' B '), but the data written in the writable Transform
side is transformed and automatically added to the readable end.
Tranform
Inherited from Duplex
, and has implemented _read
and _write
methods, while requiring the user to implement a _transform
method.
' Use strict 'Const Transform= require (' stream '). Transformclass Rotate extends Transform {constructor (n) {super ()//rotate the letter ' n ' position This. Offset = (n | | 13)% 26 } //convert data written to writable end to readable_transform (buf, enc, next) {varres = Buf.tostring (). Split ("). Map (c + = { varCode = c.charcodeat (0) if(c >= ' a ' && c <= ' z ') {Code+= This. Offsetif(Code > ' z '. charCodeAt (0) ) {Code-= 26 } } Else if(c >= ' A ' && C <= ' Z ') {Code+= This. Offsetif(Code > ' Z '. charCodeAt (0) ) {Code-= 26 } } returnString.fromCharCode (Code)}). Join (‘‘) //call the Push method to add the transformed data to the readable end This. Push (RES)//call the next method to prepare to process the nextnext ()}}varTransform =NewRotate (3) Transform.on (' Data ', data =process.stdout.write (data)) Transform.write (' Hello, ') Transform.write (' world! ') transform.end ()//Khoor, zruog!
Objectmode
In the previous sections of the example, you often see the call data.toString()
. toString()
is this call necessary?
This section describes how to control the type of data in a flow and naturally has an answer.
In the shell, |
connect the upstream and downstream with a pipe (). The upstream output is the text stream (the standard output stream), and the downstream input is also the text stream (the standard input stream). In the flow described in this article, the default is the same.
For a readable stream, push(data)
data
only the String
or Buffer
type, and data
the data that is consumed by the event output are Buffer
types. In the case of a writable stream, write(data)
data
only the String
or type is used Buffer
, and _write(data)
the call simultaneous comes in as a data
Buffer
type.
That is, the data in the stream is type by default Buffer
. Once the resulting data is put into the stream, it is turned into consumption, and the written data is also converted to the type when it is passed to the Buffer
underlying write logic Buffer
.
But each constructor receives a configuration object and has an option to objectMode
true
"man sows, reap" as soon as it is set.
Readable
When not set objectMode
:
Const readable = require (' stream '= readable () Readable.push (' a ') Readable.push (' B ') Readable.push (null) readable.on (' data ', data = console.log (data))
Output:
<buffer 61><buffer 62>
Readable
objectMode
after setting:
Const readable = require (' stream 'true }) Readable.push (' a ') Readable.push (' B ') ) Readable.push ({}) Readable.push (null) readable.on (' data ', data = console.log (data))
Output:
ab{}
As objectMode
you can see, push(data)
the data is output as it was set. At this point, you can produce any type of data.
Notice
Stream Series Total Three article:
- The first part: The basic article, introduced the stream interface fundamental use.
- The second part: Advanced article, focus on how stream bottom support streaming data processing, and its back pressure mechanism.
- The third part: the actual combat article. Describes how to use stream for program design. Summarize two design patterns from browserify and gulp, and build an example of an application that automatically generates changelog for a git repository based on stream.
Reference documents
- Github,substack/browserify-handbook
- Github,zoubin/streamify-your-node-program
node. js stream-Basic article