NodeJS the core of the difficulty (network, File) stream three: readable?

What is a readable stream

A readable stream is a stream of production data that is used for program Consumption. Our common data production methods are reading disk files, reading network request content, and so on, look at the previous introduction of what is a streaming example:

const rs = fs.createReadStream(filePath);

RS is a readable stream that produces data in a way that reads disk files, and our common console Process.stdin is also a readable stream:

process.stdin.pipe(process.stdout);

A simple sentence can be used to print the console input, Process.stdin the way to produce data is to read the User's input in the Console.

Look back at our definition of a readable stream: a readable stream is a stream of production data used for program Consumption.

Custom readable streams

In addition to what the system provides to us, fs.CreateReadStream we often use the SRC method provided by gulp or Vinyl-fs

gulp.src([‘*.js‘, ‘dist/**/*.scss‘])

If we want to produce data in a certain way and give it to the program, how do we start?

Two simple steps

1. Inheriting the readable class of the Sream module
3. Overriding the **_read** method, calling This.push to put the produced data into the queue to be read

The readable class has done most of the work of the readable stream, and we only need to inherit it, and then write the way the production data is written in the _read method to achieve a custom readable stream.

If we want to implement a stream that produces a random number every 100 milliseconds (nothing useful)

Const readable=Require' Stream ').Readable;Class RandomnumberstreamExtends Readable{Constructor (max){Super ()}_read (){Const CTX=This;SetTimeout (()={Const Randomnumber=parseint (math. Random () * 10000) //can only push string or Buffer, for easy display hit a carriage return ctx. Push ( ' ${ Randomnumber}\n}, 100) }}module. Exports = randomnumberstream;          

Class inherits part of the code is very simple, mainly look at the implementation of the _read method, there are a few noteworthy places

1. The readable class defaults to the implementation of the _read method, but nothing is done, and what we do is overwrite overrides
2. The _read method has a parameter of size that specifies how much data should be read back to the read method, but only a reference data, many implementations ignore this parameter, and we ignore it here, and we'll refer to it in detail later
3. By pushing the data through This.push to the buffer, the buffer concept is referred to as a temporary understanding that it can be consumed by squeezing into a pipe.
4. The content of push can only be a string or Buffer, not a number
5. The push method has a second parameter, encoding, that is specified when the first argument is a string encoding

Take a look at the effect

const RandomNumberStream = require(‘./RandomNumberStream‘);const rns = new RandomNumberStream();rns.pipe(process.stdout);

So we can see that the numbers are showing up on the console, we have a readable stream that generates random numbers, and there are a few minor problems to be solved.

How to stop

We push a number every 100 milliseconds to the buffer, so it's like reading a local file, how do I stop to identify the data after it's read?

Push a null on the buffer to be able. Let's modify the code to allow the consumer to define how many random numbers are needed:

Const readable=Require' Stream ').Readable;Class RandomnumberstreamExtends Readable{Constructor (max){Super ()This.Max= Max;}_read (){Const CTX=This;SetTimeout (()={if (CTx.Max{Const Randomnumber=parseint (Math.Random ()*10000);can only push strings or Buffer, in order to facilitate display hit a carriage returnCTx.Push`${randomnumber}\n " Span class= "va" >ctx. max -= 1} else {ctx.push (null) } }, 100) }}module. Exports = randomnumberstream;           

We used a max's identity, which allowed the consumer to specify the number of characters needed, to be specified when instantiated

const RandomNumberStream = require(‘./RandomNumberStream‘);const rns = new RandomNumberStream(5);rns.pipe(process.stdout);

This way you can see that the console only prints 5 characters

Why setTimeout, not setinterval?

The attentive classmate may notice that we produce a random number every 100 milliseconds is not the call of the setinterval, but the use of the setTimeout, why is it just a delay and not repeat production, the result is correct?

This requires understanding the two ways in which the flow works

1. Flow mode: data is read out by the underlying system and delivered to the application as quickly as possible
2. Pause Mode: The Read () method must be displayed to read several blocks of data

The stream is in suspend mode by default, that is, it requires the program to explicitly call the read () method, but we can get the data without a call in our example, because our flow is switched to flow mode through the pipe () method, so our _read () method is automatically called repeatedly, Until the data is read, we only need to read the data once every time the _read () method is Completed.

Flow mode and Pause mode switching

There are several ways that a stream can switch from the default pause mode to flow mode:

1. To start data snooping by adding an event listener
2. Call the Resume () method to start the data flow
3. Call the pipe () method to transfer data to another writable stream

There are two different ways to switch from flow mode to pause mode:

1. Call the pause () method to pause a stream when there is no pipe () in the stream
2. Pipe (), You need to remove the listener for all data events, and then call the Unpipe () method

Data event

Using the pipe () method, the data enters the writable stream from the readable stream, but to us it seems to be a black box, how exactly does the data flow? We see two important nouns when switching between flow mode and pause Mode.

1. Data events corresponding to the flow pattern
2. Pause mode corresponds to the Read () method

These two mechanisms are the reason we can drive data flow, first look at the flow mode data event, Once we listen to the data of the readable stream, the event, the flow into the flow mode, we can rewrite the above call flow code

const randomnumberstream Span class= "op" >= require (const RNs = new randomnumberstream (5) ; RNs. on ( ' data ' = > {console. Log (chunk) ;              

So we can see that the console prints a result similar to the following

<Buffer3935370a><buffer 31 30 Span class= "hljs-attr" >35 37 0a><< Span class= "hljs-name" >buffer 38 35 31 Span class= "hljs-attr" >30 0a>< Buffer 33 30 35 35 0a><buffer 34 36 34 32 0a>             

The data event is triggered when the readable stream produces the data that is available for consumption, which is then passed as much as possible after the database event listener is Bound. The listener of the data event can receive the buffer data passed by a readable stream in the first parameter, which is the chunk that we print, and if you want to display as a number, you can call the ToString () method of Buffer.

An end event is also triggered when the data is processed and should not be called synchronously for the stream, so if we want to do something after we're done we need to listen to this event, and we'll add a sentence at the end of the code:

rns.on(‘end‘, () => { console.log(‘done‘);});

This will show ' done ' after the data has been received.

of course, errors occur during data processing to trigger an error event, and we can also listen and do exception handling:

rns.on(‘error‘, (err) => { console.log(err);});

Read (size)

The stream requires the program to explicitly call the Read () method in pause mode to get the Data. The read () method pulls and returns some data from the internal buffer and returns NULL when there is no more data available.

When reading data using the Read () method, it returns data for the specified byte if the size parameter is passed, or null if the specified size byte is not available. If the size parameter is not specified, all data in the internal buffer is Returned.

Now there is a contradiction, in the flow mode down production of data, and then triggered by data event notification to the program, this is Convenient. In the pause mode requires the program to read, then there is a possibility that the reading time is not produced, if we only use the polling method is somewhat inefficient.

NodeJS provides us with a readable event that is triggered when a readable stream is ready for the data, that is, to listen to the event, to receive the notification and the data we have to read again:

Const RNS= new randomnumberstream ( 5) rns. on ( ' readable '  = {let chunkwhile ((chunk = rns. Read ()) !== null) { console. Log (chunk) }})           

So that we can also read the data, it is worth noting that not every call to the Read () method can return data, mentioned earlier if the available data does not reach size then return null, so we added a judgment in the Program.

Will the data be missing?

I often worry about a problem when I start using the flow mode, where the readable stream is produced at the time of creation, so it will not produce some data before we bind the readable event, triggering the readable event, we are not bound, This is not an extreme situation that will cause the loss of the initial Data.

But the fact is not, according to NodeJS event loop we create the stream and invoke the event listener in an event queue, the child production data because of the asynchronous operation, is already in the next event queue, we listen to events slower than the data production block, the data is not Lost.

See here, everybody actually to data event, readable event trigger time, Read () method How much data each time reads, when return null also have certain question, because so far we touch to still is a black box, after we introduced the writable stream will be in back Pressure mechanism part of these internal details combined with the source code detailed explanation, and listen to tell bar.

NodeJS the core of the difficulty (network, File) stream three: readable?