about using Connect-busboy to implement file upload optimization instructions

This blog post completely on the optimization of the previous article first saw node. js in the Express4.0 framework using Connect-busboy to implement file upload because the last blog to use Connect-busboy to upload files, found a significant bug

Bug Description

File display upload 100%, and then preview the time, occasionally found that the picture can only display part of this situation in the PNG format picture especially serious yesterday re-review code, found a bug, of course, and connect-busboy a little relationship is not, but involved in the flow process.

Here put the last blog posted in the upload code is put up to analyze

function upload(req, res, next) {   req.busboy.on(‘file‘, function (fieldname, file, filename, encoding, mimetype) {       var tmp_path=path.join(os.tmpDir(), path.basename(filename));       file.pipe(fs.createWriteStream(tmp_path));       file.on(‘end‘,function(){           var uuid = tool.generateUUID();           commfile.savePathFile(uuid, mimetype, tmp_path, true, function(err, fileBase64) {               if (err) {                   res.json({                       success:false,                       url:‘‘                   });               }else{                   res.json({                       success: true,                       url: ‘/file/‘ + uuid                   });               }           });       });   });   req.pipe(req.busboy);   }

For the code, let's take a careful look.req.busboy.on(‘file‘ ,function(......).....)

1. The Req.busboy.on method is used to listen for the event of a form-submitted parameter, where on (' file ') is the Listen form commit binary file stream event. Above the listener event callback function passed over several parameters, we focus on the file parameter file is a binary file stream handle, it is a standard stream 2. Then I take the server temp file directory, and file name, combined into a file path
A path like this:/users/zhangzhi/data/temp/aaa.png var tmp_path=path.join(os.tmpDir(), path.basename(filename)); 3. Then I write the file stream to the path defined abovefile.pipe(fs.createWriteStream(tmp_path));

• There's a process redundancy problem here.

It's got a pipe on it. Hope to read my other article about the pipe principle node. JS asynchronous non-blocking in IO

1. Then I read the temporary file into a byte array, converted to a base64 string, stored in the LEVELDB databasefile.on(‘end‘ ....)

• This is where the bug appears.

Let's go back and analyze the process redundancy problem and bug process redundancy

If my picture is stored directly under a folder in the server, then the above-mentioned process is no problem, (of course, do not need a temporary file directory, directly to the server designated to save the picture directory) but my current blog images are all in the LevelDB database, So there is no need to first create a picture in the temp directory, then read the image byte array, and then into the Base64 string to write the database in the middle of the detour is to save the temporary file and then read

We've got the file upload stream handle, so it can be plugged directly into the binary array, temporarily saved, after all, not the huge file above, the memory tolerance of 0, the following code

var dataArr = [];file.on("data", function (chunk){      dataArr.push(chunk);});

Above we listen to the file data event, through a stream of byte stream to the server, we no matter 3,721 first into the Dataarr binary array then what we do, we can directly convert the above Dataarr data into a base64 string into the database , the specific code implementation below, we should analyze the above bug

Bug appears

We mentioned above the 4th step has an invisible bug, when uploading files occasionally appear, especially PNG pictures, 100% exposure, uncle can endure, aunt also can't endure ah ... Why is there a bug in the 4th step? Or you can't get out of the whole process, simply review it.

The uploaded file stream is written to the temp file directory, assuming that this process is called A we read the temporary file into the Base64 string, this process is called when we should read the temporary file, should be at the end of a process, and the end of a process refers to file upload stream All finished and the file stream is all written to the temporary file but the File.on (' End ', function () {...} that we are listening on is just listening to the event that the file stream ends, and whether the file stream is completely written to the temp file is not known at all. But then we made the wrong decision to start reading the temporary file into the Base64 string so: when the file stream was only passed through HTTP, but did not finish to the temporary file, which is we immediately read the picture, must be incomplete picture

If you have to do this here, change the code if you change it:

function upload (req, res, next) {req.busboy.on (' file ', function (fieldname, file, filename, encoding, mimetype) {        var tmp_path=path.join (Os.tmpdir (), path.basename (filename));        var passedlength = 0;        var writestream = Fs.createwritestream (Tmp_path);                 File.on (' Data ', function (trunk) {passedlength + = Chunk.length;                 if (writestream.write (chunk) = = = False) {file.pause ();        }        });        File.on (' End ', function () {////when there is no data read, close write data stream writestream.end ();        }); Writestream.on (' Drain ', function () {//Speed after writing///Here we're going to do it. Converting a picture file to a Base64 string writes a database operation var uuid = TOOL.G            Enerateuuid ();                    Commfile.savepathfile (UUID, MimeType, Tmp_path, True, function (err, fileBase64) {if (err) {                Res.json ({success:false, url: '});           }else{         Res.json ({success:true, url: '/file/' + uuid}                );              }           });        });    Req.pipe (Req.busboy); }

The above code is not tested and is directly knocked up, just to illustrate the correct way of handling the original process

The process problem is clear, and the bug is found. The following fixes

First on the code

  req.busboy.on (' file ', function (fieldname, file, filename, encoding,        MimeType) {var dataarr = [], len = 0,strbase64;            File.on ("Data", function (chunk) {Dataarr.push (chunk);        Len + = Chunk.length;        });            File.on ("End", function () {strBase64 = Buffer.concat (Dataarr, Len). toString (' base64 ');            var uuid=tool.generateuuid ();                        Commfile.savefile (Uuid,mimetype,strbase64,function (Err) {if (err) {Res.json ({                Success:false, url: '});                    }else{Res.json ({success:true, url: '/file/' + UUID                });    }            })        });    }); Req.pipe (Req.busboy);  

Simply describe the above code file.on ("Data" event to write the byte stream in the array in real time File.on ("End" when the upload file stream ends, the bytes array is converted to base64 string strBase64 = Buffer.concat (Dataar R, Len). toString (' base64 '); Finally, we write the Base64 string to the database so upload the image will not appear to write to the database incomplete picture of the bug.

From: about using Connect-busboy to implement file upload optimization instructions

You might be interested.

• Pandoc Markdown Generating PDF documents
• Forever the great modules that are indispensable in the node. JS Web Application
• node. js aes/ecb/pkcs5padding encryption and decryption common to other languages
• Ejs master page makes your node. JS app development faster and more efficient-a commoner
• node. JS implements a simple login blocker

about using Connect-busboy to implement file upload optimization instructions