How to write a KOA middleware to enable the continuation of a breakpoint

The continuation of the breakpoint in this article is just an understanding of the continuation of the breakpoint. There are a lot of imperfections in it, just a record of my implementation of a continuation of the breakpoint. People should also find that I use some H5 API, the old browser will not support, and I do not take cross-domain into account, there are some of the possible wait ~ Bar. (How do you feel so many questions??? Laugh ~)

This article refers to the warehouse: Point me

These days in earnest to learn KOA framework, understand its principles and KOA middleware implementation methods. When I was studying how KOA handled the uploaded form data, I had a flash, could it be used for the continuation of a breakpoint?

Breakpoint continuation is not the server end of the self-high, he also needs the front-end of the mate, and I am only ready to BA la a rough prototype, so this feature I prepared:

• Backend: Handwritten KOA middleware handles breakpoint data
• Front end: Native JS

The process of the continuation of the breakpoint is not complicated, but there are many small knowledge points need get, otherwise it is difficult to understand the continuation of the breakpoint work process. There are many ways to implement a breakpoint, but I've only studied the way Ajax works, so the little things to prepare are as follows:

KOA section: Headers's content-type

Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryE1FeIoZcbW92IXSd

The HTML form component provides a total of three ways of encoding: application/x-www-form-urlencoded (default), multipart/form-data text/plain . The first two methods are more common, the last of which is not used, and is not recommended. The first two differences are that the default method cannot be uploaded <input type="file"/> . So if we need to upload files, then we must use them multipart/form-data .

form- raw data uploaded

In KOA, the data that the server obtains is the raw data unprocessed binary. We need to format this data to extract valid content. Let's analyze how to deal with this raw data .

When we upload, we will find a phenomenon, is content-type also with a small tail multipart/form-data; boundary=----WebKitFormBoundarygNnYG0jyz7vh9bjm , this long string of strings is used to do? Take a glance at the complete raw data :

------WebKitFormBoundarygNnYG0jyz7vh9bjmContent-Disposition: form-data; name="size"668------WebKitFormBoundarygNnYG0jyz7vh9bjmContent-Disposition: form-data; name="file"; filename="checked.png"Content-Type: image/png------WebKitFormBoundarygNnYG0jyz7vh9bjm--

We found that there were no separate fields between ------WebKitFormBoundarygNnYG0jyz7vh9bjm them. So here's the boundary one used to split the field.

Aboutboundary

• Its value can be customized, but the browser will help us define
• Cannot exceed 70 characters
• In the raw data , you need to add in front -- , that --boundary is, if it is the end of the delimiter, then add one at the end -- , that is--boundary--

For more information, please refer to the Multipart content-type

request data and end listening events in HTTP

Send the data to the server, he has to have a way to accept it, right? So this time, we need to configure the data listening data acceptance, and end the listening data acceptance is complete.

Each time the data event is triggered, the obtained data is a buffer type of data, and then the obtained data is added to the buf array, and so on, and then the end of the Buffer.concat buffer data concatenated to become a complete buffer. That's it, the server takes the client's data to completion.

This paragraph is very simple, it ctx.req is encapsulated in the KOA request .

let buf = [];let allData;ctx.req.on("data",(data)=>{    buf.push(data)});ctx.req.on("end",(data)=>{    allData=Buffer.concat(buf)})

Handling of Buffer

The key part came, and this part of the pit got me so miserable.

What we server obtains raw data is not a string, but a string Buffer . What is buffer? is the binary data. Although we can move to a Buffer string and then process it, it's a headache to have a coding problem because toString the encoding format is the default utf-8 . If it is not utf-8 , then we get the result is very problematic. So if you want to process Buffer data, you still need to use the Buffer data. For example ------WebKitFormBoundarygNnYG0jyz7vh9bjm , I would like to know the position of this section in buffer. Then I can turn this section into buffer and then go through the query one by one.

A history of blood and blood between me and raw data (p haha):

Raw Data	I'm
I'm a binary stream	I'll take care of you.
	I'm going to turn you into my favorite string, human readable language, and then divide you
If I had been human readable, then you could do so, in case I was a picture or other format, emmm	What's the problem?
Then you're not going to see me like that.	？？？
In short, if I was a picture, you turn me into text, write a file, I am a bunch of garbled	What??? (φ dish φ)
So you can only use my kind to deal with me.	Similar?
That is, the binary stream	Which means I'm going to turn the delimiter into a binary stream and then split you?
That's it.	Big Brother, I lost.
Although I am a binary stream, but you can use a familiar way to query me	Hey? Do you have a shortcut?
buf.indexOf(value)can help you query location	Oh
Buf.slice ([start[, end]]) can help you split my	Oh
I can only help you here.	Come on, don't send.

Implementation code:

function splitBuffer(buffer,sep) {    let arr = [];    let pos = 0;//当前位置    let sepPosIndex = -1;//分隔符的位置    let sepPoslen = Buffer.from(sep).length;//分隔符的长度，以便确定下一个开始的位置    do{        sepPosIndex=buffer.indexOf(sep,pos)        if(sepPosIndex==-1){            //当sepPosIndex是-1的时候，代表已经到末尾了，那么直接直接一口读完最后的buffer            arr.push(buffer.slice(pos));        }else{            arr.push(buffer.slice(pos,sepPosIndex));        }      pos = sepPosIndex+sepPoslen    }while(-1!==sepPosIndex)    return arr}

Front-end section: the slice method of Fileapi in H5

sliceBefore is a method for the array, now the file can also be used slice to split the pull, but it is important to note that this method is a new API, that is, many old browsers are not available.

The usage is simple:

//初始位置，长度//这里的File对象是一个Blob，一个类似于二进制的流，所以这里是以字节为单位的。File.slice(startByte, length);

the native Ajax implementation XMLHttpRequest method of JS

Create a newXMLHttpRequest

xhr = new XMLHttpRequest();

Open a post for the requested link

xhr.open("post", "/submit", true);

Configuration onreadystatechange to capture the status of the request link.

xhr.onreadystatechange = function(){    //xhr.readyState    //处理完成的逻辑};

readyState	meaning
0	Initialization
1	Load in
2	Load complete
3	Partially available
4	Load complete

The preparation is done, and the final send, request the link.

xhr.send(表单数据);

The following section will write how to generate the form data in send

encapsulating form Data FormData

FormDataThe use is very friendly, is to follow the health value one by one paired on it.

var formData = new FormData();formData.append("test", "I am FormData");formData.append("file", 你选择的文件);

Although simple, it is possible to simulate the data format of the post to the server.

Detailed usage, dot me

Main logic of the continuation of the breakpoint

Writing so much about the subsequent development of the breakpoint to continue the relevant knowledge points, we can start to write. The logic of the continuation of a breakpoint is not complicated, presumably:

Client Clients	server-side server
I want to upload a file	Ok,no problem, but you can only use the post to pass me
My files are very large, form can I submit them directly?	How big, if very big, once our connection disconnects, we all have naught! Be careful!
Well,well, I'll take my file into a small piece and slice give it to you slowly.	Come on, baby~, I don't mind you coming a few more times.
The first partsend	Accept in ...
Waiting in ...	Accept completed, processing the received BLOB, processing completed has been written, you can pass the second part of the ~
Part IIsend	Accept in ...
Waiting in ...	Accept completed, processing the accepted BLOB, processing completed has been written, you can pass the third part of the ~
...	...
...	Finally, I'll take care of your papers.
...	Ok~ Transmission Success

The processing mode of the client side of the breakpoint continuous transmission

From the above logic, this front-end process can be divided into:

• Determine the file size, the same length slice of the root play
• Callback upload according to the number of slices

Slicing files

The continuation of the breakpoint is the client actively send, server-side passive acceptance of a process, so here is the client to do a file segmentation, the size of the file according to range the segmentation, range the size can be customized. Here I want to prevent each upload slice to calculate the position, so put all the position in advance into currentSlice the array. Then take the position sequentially. Note: This segmentation is all calculated in bytes.

createSlices(){    let s=0,e=-1,range=1024;    for(let i = 0;i<Math.ceil(this.file.size/range);i++){        s=i*range,e=e+range        e=e>this.file.size-1?this.file.size-1:e;        this.currentSlice.push([s,e])    }}

Now that we know how many slices of fragmented fragments, we can get progress by dividing the uploaded fragments by the total fragments, and then make a progress. It feels like a complex look here, calm ~ I just put the interface style is added ~

updateProcess(){    let process=Math.round(this.currentIndex/this.currentSlice.length*100)    this.fileProcess.innerHTML=`<span class="process"><span style='width:${process}%'></span><b>${process}%</b></span><span>${this.fileSize}</span>`},

Also note that the file is in bytes, this is very unfriendly to the user, in order to tell the user how large the file, we need to convert. Here I am the dynamic conversion, not a fixed unit, because if a file only a few KB, and then I use the unit of G to calculate, then is the eyeful of 0. Here can be based on the size of the file large, specific analysis of the situation. I have only given a KB and MB calculation here. The ElseIf can be added to the condition by itself.

calculateSize(){    let fileSize=this.fileSize/1024;    if(fileSize<512){        this.fileSize=Math.round(fileSize)+"KB"    } else {        this.fileSize=Math.round(fileSize/1024)+"MB"    }},

Slice files upload one by one

Now that you have to upload it, you have to summon it XMLHttpRequest . Uploading files to Ajax. The upload file must be enctype="multipart/form-data" , so please also FormData help us to create form form data.

First create a form data bar ~, in fact, we only need to upload a file blob files on it, but the server is not so witty, able to add a unique identity to the file, so we in the file to add the file information, such as filename, file size, and the location of the file segmentation. This part is free to play, see what you need to add what paragraph, such as time, User ID, bar la ~

createFormData(){    let formData = new FormData();    let start=this.currentSlice[this.currentIndex][0]    let end=this.currentSlice[this.currentIndex][1]    let fileData=this.file.slice(start,end)    formData.append("start", start);    formData.append("end", end);    formData.append("size", this.file.size);    formData.append("fileOriName", this.file.name);    formData.append("file", fileData);    return formData;}

Finally the warm-up finished, it should be uploaded. This is a standard XMLHttpRequest upload template, there is a very friendly and kind. This side does not touch the cross-domain and so on what the problem, so very friendly. You only need to recall this upload method after the upload is successful. Upload them individually. Until the last slice. Here in order to see the upload process, so I added a 500ms delay, this is only for visual effects, after all, I just tried a few megabytes of files, upload too fast.

createUpload(){    let _=this    let formData=this.createFormData()    let xhr = new XMLHttpRequest();    xhr.open("post", "/submit", true);    xhr.onreadystatechange = function(){        if (xhr.readyState == 4&&parseInt(xhr.status)==200){            _.currentIndex++;            if(_.currentIndex<=_.currentSlice.length-1){                setTimeout(()=>{                    _.createUpload()                },500)            }else{                //完成后的处理            }            _.updateProcess()        }    };    xhr.send(formData);}

How a breakpoint continues to be processed by the server side

From the above logic, this back-end process can be divided into:

• Accept the data stream of the file, add buffer
• Acceptance completed, extract content
• Rename file name
• Write Local
• Get the file again from the first step until all slices have been accepted.

Receive data stream

This estimate is the simplest part of the entire process, and node listens, assembles, and is done!

let buf=[]ctx.req.on("data",(data)=>{    buf.push(data)});ctx.req.on("end",(data)=>{    if(buf.length>0){        string=Buffer.concat(buf)    }})

Extract Content

Do you remember that we are passing binary, and this binary in addition to the text field, there is also the binary of the file. At this point, we need to extract the fields first and then separate the files from the normal text.

First assemble the delimiter, here is a rule, is content-type in the boundary front need to add -- .

boundary=ctx.headers["content-type"].split("=")[1]boundary = '--'+boundary

As mentioned above, binary segmentation can only be binary, so I can change the delimiter into binary, and then split the received content.

function splitBuffer(buffer,sep) {    let arr = [];    let pos = 0;//当前位置    let sepPosIndex = -1;//分隔符的位置    let sepPoslen = Buffer.from(sep).length;//分隔符的长度，以便确定下一个开始的位置    do{        sepPosIndex=buffer.indexOf(sep,pos)           if(sepPosIndex==-1){            //当sepPosIndex是-1的时候，代表已经到末尾了，那么直接直接一口读完最后的buffer            arr.push(buffer.slice(pos));        }else{            arr.push(buffer.slice(pos,sepPosIndex));        }      pos = sepPosIndex+sepPoslen    }while(-1!==sepPosIndex)    return arr}

After the partition is finished, we will begin to deal with it! Extract the fields. Here we will extract the content into a string, first of all, this is to determine the field type, and secondly if it is not a file, then you can extract our field text, if it is a file type, then can not be willful toString , we need to put the binary file content is perfectly preserved.

------WebKitFormBoundaryl8ZHdPtwG2eePQ2FContent-Disposition: form-data; name="file"; filename="blob"Content-Type: application/octet-streamk换行*2乱码换行*1------WebKitFormBoundaryl8ZHdPtwG2eePQ2F--

The content of the upload is probably long. This way, the code for the empty line is \r\n that the conversion to binary is a 2 position, so the interception of two blank lines can get the field information and content. Because there is also a blank line at the end, so in the interception of binary file content, in addition to the length of the head of the length of +2 line, the end of the 1 for the president to add, so it is line.slice(head.length + 4, -2) this way.

function copeData(buffer,boundary){    let lines = splitBuffer(buffer,boundary);    lines=lines.slice(1,-1);//去除首尾    let obj={};    lines.forEach(line=>{        let [head,tail] = splitBuffer(line,"\r\n\r\n");        head = head.toString();        if(head.includes('filename')){ // 这是文件            obj["file"]= line.slice(head.length + 4, -2)        }else{          // 文本          let name = head.match(/name="(\w*)"/)[1];          let value= tail.toString().slice(0,-2);          obj[name]=value        }    });}

Renaming files

We upload files generally do not exist in the original name preservation, in case everyone likes to pass the name of the file? What a headache! This time I need to rename, I generally like to use MD5 to calculate the new file name. Here we can splice some of the fields we uploaded
such as time, the main is to give a special logo to ensure that the current uploaded files to distinguish between other files. After all, the same content is calculated with the same MD5, the same file name MD5 calculated does not play a role in the distinction.

Of course the suffix of the file cannot be forgotten! Otherwise the file can not be opened. So remember to extract the file suffix.

let fileOriName=crypto.createHash("md5").update(obj.fileOriName).digest("hex")let fileSuffix=obj.fileOriName.substring(obj.fileOriName.lastIndexOf(".")+1)

Save File

Here I am based on whether it is the first slice, to see if it is new overwrite or append the file content again. Attention, because if the file does not exist directly appendFileSync will be an error. But repetition writeFileSync overwrites the content. So we need to distinguish, we can judge the existence of the file to differentiate ~.

if(parseInt(obj.start)===0){    fs.writeFileSync(__dirname+`/uploads/${fileOriName}.${fileSuffix}`,obj.file);}else{    fs.appendFileSync(__dirname+`/uploads/${fileOriName}.${fileSuffix}`,obj.file);}

Repeat repeat Repeat

Repeat ~ until the client's slices are all delivered ~

Appendix:

Do not understand KOA can look at my other articles:

The basis of this article, refer to the koa,5 step handwritten a rough web framework

For the realization of router, this KOA simple router hand-knocking guide please accept

A simple template engine implementation approach for KOA template implementation