[Node. js] uses the File API to upload files asynchronously, and node. jsapi

[Node. js] uses the File API to upload files asynchronously, and node. jsapi

Address: http://www.moye.me/2014/11/05/html5-filereader/

 

Recently, I was working on a network disk project, which involved uploading large files. The question is: How can I display the File Upload progress in real time?

Problem breakdown

It seems that I have done similar functional modules (based on the. NET platform) a few years ago. The solution is as follows:

• Form-based submission
• The Server assigns an identifier (GUID) based on the uploaded file and performs stream reading.
• The Browser initiates Ajax to pull the file upload status

The problem with this solution is the file size (up to 2 GB ). The so-called real-time display of File Upload progress, I personally think the ideal solution is:

• The Browser needs to tell the Server File Size
• The Browser must be able to read files in multiple parts.
• The Server calculates the progress based on the received block and file size, and notifies the Browser.
• The Browser continues reading multipart upload when the progress is not completed.

HTML5 File API

In the above solution, the biggest difficulty is that the Browser Side reads files in blocks. Fortunately, the HTML5 File API provides the following interface: FileReader

With the FileReader object, web applications can asynchronously read files (or raw data buffering) stored on users' computers, you can use a File object or Blob Object to specify the File or data to be read. The File object can be the FileList object returned by the user after selecting a File on the <input> element ......

The interesting thing is that the Blob interface has only one method: slice () -- it is not hard to imagine that it is used for data partitioning. The method signature is like:

Blob slice(  optional long long start,  optional long long end,  optional DOMString contentType};

The W3C Draft shows that the File interface actually inherits from the Blob interface, which means File. slice (start, end) can return the block data of the file, combined with FileReader. readAsBinaryString method. We can read any part of data in the local file on the Browser side.

About FileReader

First of all, FileReader is not supported by every browser, and the compatibility test (unfortunately, the giant IE is dragging back ...... :

Operating System	Firefox	Chrome	Internet Explorer	Opera	Safari
Windows	Supported	Supported	Not Supported	Supported	Not Supported
MAC OS X	Supported	Supported	N/	Supported	Supported

Second, to use the readAsBinaryString method, you need to subscribe to the onloadend event of FileReader. That is, when the block data read operation is complete, the event subscription method will obtain the binary block data that has been read:

currentFileReader.onload = function (evnt) {    console.log('Data content length: ', evnt.target.result.length);};

B/S communication

When block data is obtained, the remaining problem is how to issue the data. There are some options: AJAX, rich client programming, and WebSocket. Because the Network Disk Project is developed based on Node, I chose Socket. IO as the communication framework between the two ends of B/S.

Starting from end B

Page preparation and reference:

<Div> <progress id = "progressBar" value = "0" max = "100"> </progress> </div> <input type = "button" id = "choose -button "value =" SELECT file "> <input type =" file "id =" choose-file "class =" hidden "/> </div> <script src =" https://code.jquery.com/jquery-1.10.2.min.js & gt; </script & gt; <script src = "/socket. io/socket. io. js "> </script>

Browser compatibility test first:

if (!window.File && !window.FileReader) {    alert('Your browser does not support the File API. Please use modern browser');    return;} else {     var socket = io.connect();     var currentFile = null;     var currentFileReader = null;}

After you select a file, process the event:

$('#choose-file').on('change', function () {    currentFile = document.getElementById('choose-file').files[0];    if (currentFile) {        currentFileReader = new FileReader();        currentFileReader.onload = function (evnt) {            socket.emit('upload', {                'Name': currentFile.name,                'Segment': evnt.target.result            });        };        socket.emit('start', {            'Name': currentFile.name,            'Size': currentFile.size        });    }});

From the code above, we can see that socket. emit ('start') is the beginning of the interaction process, which tells the Server File Information; FileReader. onload is the emit data on the Server by block. There is still a piece of code that triggers FileReader:

socket.on('moreData', function (data) {     updateProgressBar(data.percent);    var position = data.position * 524288;    var newFile = null;    if (currentFile.slice)        newFile = currentFile.slice(position, position + Math.min(524288, currentFile.size - position));    else if (currentFile.webkitSlice)        newFile = currentFile.webkitSlice(position, position + Math.min(524288, currentFile.size - position));    else if (currentFile.mozSlice)        newFile = currentFile.mozSlice(position, position + Math.min(524288, currentFile.size - position));    if (newFile)        currentFileReader.readAsBinaryString(newFile); // trigger upload event});

The moreData message on the Browser side is triggered by the Server side. After receiving the start message, the Server side will send the moreData message to the Browser side. Note that different browsers have different implementations for the Blob. slice interface (blob. Batch slice () in versions earlier than Firefox 12 and blob. webkitSlice () in Safari ()

Upload finishing work:

socket.on('done', function (data) {    delete currentFileReader;    delete currentFile;    updateProgressBar(100);});

Server implementation

First, a global data structure is required to save the descriptor of each uploaded file (deleted from the scope after being uploaded ):

  var Files = {};

Then initialize Socket. IO and prepare the file descriptor:

var io = require('socket.io').listen(server);io.sockets.on('connection', function (socket) {    //prepare for uploading    socket.on('start', function (data) {         var name = data.Name;        var size = data.Size;        var filePath = '/tmp';        var position = 0;        Files[name] = { // define storage structure            fileSize: size,            data: '',            downloaded: 0,            handler: null,            filePath: filePath,        };        Files[name].getPercent = function () {            return parseInt((this.downloaded / this.fileSize) * 100);        };        Files[name].getPosition = function () {            return this.downloaded / 524288;        };        fs.open(Files[name].filePath, 'a', 0755, function (err, fd) {            if (err)                console.log('[start] file open error: ' + err.toString());            else {                Files[name].handler = fd; // the file descriptor                socket.emit('moreData', { 'position': position, 'percent': 0 });            }        });            });});

When the Server receives the upload message, it does not write it immediately. Instead, it is buffered and written in 10 MB in batches:

socket.on('upload', function (data) {    var name = data.Name;    var segment = data.Segment;    Files[name].downloaded += segment.length;    Files[name].data += segment;    if (Files[name].downloaded === Files[name].fileSize) {        fs.write(Files[name].handler, Files[name].data, null, 'Binary',            function (err, written) {            //uploading completed            delete Files[name];            socket.emit('done', { file: file });        });    } else if (Files[name].data.length > 10485760) { //buffer >= 10MB        fs.write(Files[name].handler, Files[name].data, null, 'Binary',            function (err, Writen) {            Files[name].data = ''; //reset the buffer            socket.emit('moreData', {                'position': Files[name].getPosition(),                'percent': Files[name].getPercent()            });        });    }    else {        socket.emit('moreData', {            'position': Files[name].getPosition(),            'percent': Files[name].getPercent()        });    }});

Summary

The biggest problem with the File-based API upload solution is compatibility, IE, you know... However, the times are always making progress, and we cannot be dragged by decaying and lagging people. Maybe the power of developers and users can really bring these decaying and lagging things out of our sight.

 

For more articles, see my blog new address: http://www.moye.me/

Form has input = "file" to upload files. There is a js method mysubmit () {which wants to write asynchronous storage, but does not know the value of the serialized file}

Ajax cannot complete the file upload function. If necessary, use the hidden iframe implementation, or you can directly search for components such as ajax uploader.

Ajax does not support this function. The core logic of all plug-ins is to dynamically create an iframe, dynamically construct a form in the iframe, and then submit the form, rather than ajax.

Nodejs creates an http server to receive uploaded files

Time is limited. View by yourself
Var connect = require ('connect ') var http = require ('http') var app = connect () var multipart = require ('connect-multipart '); // parse urlencoded request bodies into req. bodyvar bodyParser = require ('body-parser ') app. use (bodyParser. urlencoded () app. use (multipart () // respond to all requestsapp. post ('/upload', function (req, resp) {console. log (req. body, req. files); // don't forget to delete all req. files when done}); // create node. js http server and listen on porthttp. createServer (app ). listen (3000) Reference: github.com/senchalabs/connect