Subtly copying a stream

Scene

In the actual business, there may be repeated consumption of a readable stream, such as in the pre-filter parsing the request body, get the body for the relevant permissions and identity authentication; The authentication is passed to the business context through the post-frame or post-filter again parsing the request body. Therefore, the need to repeat consumption of the same flow is not very wonderful, it is similar to the JS context through the deep clone of an object to manipulate the object copy, to prevent the source data is contaminated.

  Const KOA = require (' Koa '); const APP = new Koa (); Let parse = function (CTX) {return new Promise (res) =>{        let chunks = [],len = 0, BODY = null;        Ctx.req.on (' Data ', (chunk) =>{Chunks.push (chunk) len + chunk.length});            Ctx.req.on (' End ', () =>{BODY = (Buffer.concat (Chunks,len)). ToString ();        Res (body);    });    })}//Certified App.use (Async (Ctx,next) = {Let BODY = Json.parse (decodeURIComponent (await parse (CTX));    if (body.name! = ' admin ') {return ctx.body = ' permission denied! ' } await next ();})    Parse body body, pass to business layer App.use (Async (Ctx,next) = {Let BODY = await parse (CTX);    Ctx.postbody = body; await next ();})  App.use (Async CTX = {ctx.body = ' Hello world\n '; Ctx.body + = ' Post body: ${ctx.postbody} ';}); App.listen (+);  

The code snippet above does not work properly and the request cannot be responded to. This is because the request body is consumed in the authentication logic of the pre-filter, and the request body cannot be consumed again in the second-level filter, so the request is blocked. In the actual business, the authentication logic is often related to each company specification, is a "two-party library"; the second-quarter filter in the example is usually present as a three-party library, so in order to not affect the third-party package consumption request, the Ctx.req must be saved in the Certified two-party package. The data for this readable stream still exists, which involves the main thrust of this article.

Realize

The replication stream is not as simple and straightforward as copying an object, the use of the stream is one-time, and once a readable stream is consumed (written to a writeable object), the readable stream is non-renewable and can no longer be used. But with some simple tricks you can restore a readable stream again, although the recovered stream is the same as the previous stream, but it is not the same object, so the properties and prototypes of the two objects are different, which often affects the subsequent use, but the method is always there, and see below.

Realization of a: readable stream of "shadow of the Technique"

The "shadow copy" of a readable stream is similar to Naruto's, but is limited to the nature of the stream of the cloned object, which guarantees that the cloned stream has the same data. But the cloned stream cannot have other properties of the original object, but we can inherit the property and method through the way the prototype chain inherits.

Let readable = require (' stream '). Readable;let fs = require (' FS '); Let path = require (' path '); class Newreadable extends readable{constructor (Originreadab        Le) {super ();        this.originreadable = originreadable;    This.start ();        } start () {This.originReadable.on (' data ', (Chunck) =>{This.push (Chunck);        });        This.originReadable.on (' End ', () =>{this.push (null);                });        This.originReadable.on (' Error ', (e) =>{This.push (e);    }); }//As a readable implementation class, you must implement the _read function, otherwise throw the Error _read () {}}app.use (async (Ctx,next) = {Let clonereq = new    Newreadable (Ctx.req);    Let cloneReq2 = new newreadable (ctx.req); At this point, Ctx.req has been consumed (no content), all the data are completely cloned two streams//consumption clonereq, get authentication data let BODY = Json.parse (decodeURIComponent (await parse    ({req:clonereq}));    The cloned CLONEREQ2 re-set the prototype chain, inheriting ctx.req original property clonereq2.__proto__ = Ctx.req;   Thereafter re-ctx.req copy, leaving the subsequent filter consumption ctx.req = CLONEREQ2; if (body.name! = ' admin ') {return ctx.body = ' permission denied! ' } await next ();})

Reviews: This technique can replicate multiple readable streams at the same time, and it needs to be re-assigned to the original stream, inheriting the original attribute, so that the subsequent repeated consumption is not affected.

Implementation of two: lazy people to achieve

The stream module has a special class, the Transform. With regard to the characteristics of transfrom, as I have described in detail in the Transform of node , he has the dual characteristics of readable and writable streams, so the use of transfrom can quickly and easily achieve cloning.

First, the pipe function is used to direct the readable stream to two transform streams (two because of the need to consume a stream in the pre-filter, followed by a second consumption of the filter).

let cloneReq = new Transform({    highWaterMark: 10*1024*1024,    transform: (chunk,encode,next)=>{        next(null,chunk);    }});let cloneReq2 = new Transform({    highWaterMark: 10*1024*1024,    transform: (chunk,encode,next)=>{        next(null,chunk);    }});ctx.req.pipe(cloneReq)ctx.req.pipe(cloneReq2)

In the above code, it seems that the Ctx.req stream is consumed (pipe) two times, in fact, the pipe function can be regarded as readable and writeable implementation of a backpressure "syntax sugar" implementation, specifically through The stream-readable and writeable in node are understood, so the results are " Ctx.req was consumed once, but the data was copied in the read buffer of the two Transfrom objects of Clonereq and CLONEREQ2, which implemented clone "

In fact, pipe for readable and writeable do the current limit, first for the readable data event to listen, and execute the writeable write function, When the write buffer of writeable is greater than a critical value (Highwatermark), the Write function returns False (this means that writeable cannot match readable speed, writeable write buffer is full), at this point, The pipe modifies the readable mode, executes the pause method, enters paused mode, and stops reading the read buffer. While writeable begins flushing the write buffer, the drain event is triggered asynchronously after the flush completes, in which the readable is set to the flowing state, and the flow function continues to refresh the read buffer, thus completing the pipe current limit. It is important to note that readable and writeable each maintain a buffer that differs from the implementation: the readable buffer is an array that holds buffer, string, and object types, while writeable is a list with a forward list. Store the data that you want to write in turn.

Finally, at the same time as the data is copied, the additional properties can be copied to one of the objects:

// 将克隆出的cloneReq2重新设置原型链，继承ctx.req原有属性cloneReq2.__proto__ = ctx.req;// 此后重新给ctx.req复制，留给后续过滤器消费ctx.req = cloneReq2;

At this point, the implementation of clone through transform is complete. The complete code is as follows (the most pre-filter):

// 认证app.use(async (ctx,next) => {    // let cloneReq = new NewReadable(ctx.req);    // let cloneReq2 = new NewReadable(ctx.req);    let cloneReq = new Transform({        highWaterMark: 10*1024*1024,        transform: (chunk,encode,next)=>{            next(null,chunk);        }    });    let cloneReq2 = new Transform({        highWaterMark: 10*1024*1024,        transform: (chunk,encode,next)=>{            next(null,chunk);        }    });    ctx.req.pipe(cloneReq)    ctx.req.pipe(cloneReq2)    // 此时，ctx.req已被消费完(没有内容)，所有的数据都完全在克隆出的两个流上    // 消费cloneReq，获取认证数据    let body = JSON.parse(decodeURIComponent(await parse({req: cloneReq})));    // 将克隆出的cloneReq2重新设置原型链，继承ctx.req原有属性    cloneReq2.__proto__ = ctx.req;    // 此后重新给ctx.req复制，留给后续过滤器消费    ctx.req = cloneReq2;    if(body.name != 'admin'){        return ctx.body = 'permission denied!'    }    await next();})

Description

Is it reasonable for

1. Ctx.req to execute two pipe to corresponding clonereq and CLONEREQ2 and then immediately consume Clonereq objects? If the source data is large enough, pipe is not finished in the consumption of clonereq, will there be any problem?

   In fact, most of the pipe functions are asynchronous operations, that is, for the source and destination flow to do some of the flow control measures. The destination stream uses a Clonereq object that, during instantiation, the transform function passes the accepted data directly to the transform object's readable stream cache by invoking the next function, triggering ' Readable and data events '. In this way, we consume the Clonereq object below also through the "Listen to data event" implementation, so even if the ctx.req data is still not consumed, the following can still consume Clonereq objects normally. The data stream can still be viewed from ctx.req--clonereq---consumption.

2. The disadvantage of using the transform stream for clone-readable streams:

   In the previous example, the instantiation of the Transfrom stream passed in a parameter Highwatermark , The role of this parameter in Transfrom is explained in drill down into node transform above, that is, when the Transfrom stream reads buffer size < Highwatermark, The Transfrom stream stores the received data in a read buffer, waits for consumption, and executes the transfrom function;

   Therefore, when the source content to clone is greater than Highwatermark, cloning is not possible in this way, because the source content is >highwatermark, The Transfrom method is not executed without subsequent consumption of the Transfrom stream (when the Transfrom stream is consumed, the read buffer of the Transfrom stream becomes smaller, and when its size

   So it is important to set a reasonable highwatermark size, and the default Highwatermark is 16kB.