How to use Node. js to implement content sharding in HTTP206 tutorial _ node. js

This article mainly introduces how to use Node. js to implement content sharding in HTTP206. Node. js is a JavaScript framework for servers. For more information, see Introduction

In this article, I will describe the basic concept of HTTP status 206, and use Node. js implements it step by step. we will also use an example based on its usage in the most common scenarios to test the code: an HTML5 page that can start playing video files at any point in time.
A Brief Introduction to Partial Content

The HTTP 206 Partial Content status code and its related message header provide a mechanism for browsers and other user proxies to receive part of the Content from the server rather than all the Content. this mechanism is widely used in the transmission of video files supported by most browsers and players such as Windows Media Player and VLC Player.

The basic process can be described in the following steps:

• Browser request content.
• The server tells the browser that the content can be divided into partial requests using the Accept-Ranges message header.
• The browser resends the request and uses the Range header to tell the server the required content Range.

The server will respond to browser requests in two situations:

• If the Range is reasonable, the server returns part of the requested Content and carries the 206 Partial Content status code. The Range of the current Content is stated in the Content-Range message header.
• If the Range is unavailable (for example, it is larger than the total number of bytes of the content), the server returns the 416 request Range unreasonable Requested Range Not Satisfiable status code. the available Range is also declared in the Content-Range message header.

Let's take a look at each key message header in these steps.

Accept-Ranges: bytes)

This is a byte header sent by the server to display the content that can be distributed to the browser by division. this value declares that each range of requests that can be accepted is in most cases bytes.

Range: bytes (bytes) = (start)-(end)

This is the message header that the browser informs the server of part of the content range. note that both the start and end positions are included and start from 0. this message header can also be left empty. its meaning is as follows:

• If the end position is removed, the server returns the last available byte from the declared start position to the end position of the entire content.
• If the start position is removed, the end position parameter can be described as the number of bytes that can be returned by the server from the last available bytes.

Content-Range: bytes = (start)-(end)/(total)

This message header will appear along with HTTP status code 206. the start and end values show the range of the current content. like the Range message header, both values are included and start from scratch. total this value declares the total number of available bytes.

Content-Range: */(total)

The header information is the same as the preceding one, but it is in another format and is sent only when the HTTP status code 416 is returned. The total number represents the total number of bytes available for the body.

Here is an example with 2048 bytes. Note that the difference between the start point and the focus is omitted.

The first 1024 bytes of the request.

Sent by the browser:

GET /dota2/techies.mp4 HTTP/1.1Host: localhost:8000Range: bytes=0-1023

Server return:

HTTP/1.1 206 Partial ContentDate: Mon, 15 Sep 2014 22:19:34 GMTContent-Type: video/mp4Content-Range: bytes 0-1023/2048Content-Length: 1024 (Content...)

Request with no endpoint

Sent by the browser:

GET /dota2/techies.mp4 HTTP/1.1Host: localhost:8000Range: bytes=1024-

Server return:

HTTP/1.1 206 Partial ContentDate: Mon, 15 Sep 2014 22:19:34 GMTContent-Type: video/mp4Content-Range: bytes 1024-2047/2048Content-Length: 1024 (Content...)

Note: the server does not need to return all the remaining bytes in a single response, especially when the body is too long or has other performance considerations. The following two examples are acceptable in this case:

Content-Range: bytes 1024-1535/2048Content-Length: 512

The server returns only half of the remaining body. The range of the next request starts from 1,536th bytes.

Content-Range: bytes 1024-1279/2048Content-Length: 256

The server returns only the 256 bytes of the remaining body. The range of the next request starts from 1,280th bytes.

The last 512 bytes of the request

Sent by the browser:

GET /dota2/techies.mp4 HTTP/1.1Host: localhost:8000Range: bytes=-512

Server return:

HTTP/1.1 206 Partial ContentDate: Mon, 15 Sep 2014 22:19:34 GMTContent-Type: video/mp4Content-Range: bytes 1536-2047/2048Content-Length: 512 (Content...)

The range of unavailable requests:

Sent by the browser:

GET /dota2/techies.mp4 HTTP/1.1Host: localhost:8000Range: bytes=1024-4096

Server return:

HTTP/1.1 416 Requested Range Not SatisfiableDate: Mon, 15 Sep 2014 22:19:34 GMTContent-Range: bytes */2048

After understanding the workflow and header information, we can use Node. js to implement this mechanism.

Start implementation with Node. js

Step 1: Create a simple HTTP server

We will start with a basic HTTP server as in the example below. This is enough to process most browser requests. First, we initialize the objects we need and use initFolder to represent the file location. To generate the Content-Type header, we list the file extensions and their corresponding MIME names to form a dictionary. In the callback function httpListener (), only GET is allowed. If other methods are available, the server returns 405 Method Not Allowed. if the file does Not exist in initFolder, the server returns 404 Not Found.

// Initialize the required object var http = require ("http"); var fs = require ("fs"); var path = require ("path "); var url = require ("url"); // The initial directory. you can change it to the desired directory var initFolder = "C: \ Users \ User \ Videos" at any time "; // list the required file extension and MIME name. var mimeNames = {". css? 7.1.2 ":" text/css ",". html ":" text/html ",". js? 7.1.2 ":" application/javascript ",". mp3 ":" audio/mpeg ",". mp4 ":" video/mp4 ",". ogg ":" application/ogg ",". ogv ":" video/ogg ",". oga ":" audio/ogg ",". txt ":" text/plain ",". wav ":" audio/x-wav ",". webm ":" video/webm ";}; http. createServer (httpListener ). listen (8000); function httpListener (request, response) {// we will only accept GET requests; otherwise, 405 'method Not allowed' if (request. method! = "GET") {sendResponse (response, 405, {"Allow": "GET"}, null); return null;} var filename = initFolder + url. parse (request. url, true, true ). pathname. split ('/'). join (path. sep); var responseHeaders ={}; var stat = fs. statSync (filename); // check whether the file exists. if the file does Not exist, the system returns the 404 Not Found if (! Fs. existsSync (filename) {sendResponse (response, 404, null, null); return null;} responseHeaders ["Content-Type"] = getMimeNameFromExt (path. extname (filename); responseHeaders ["Content-Length"] = stat. size; // file size sendResponse (response, 200, responseHeaders, fs. createReadStream (filename);} function sendResponse (response, responseStatus, responseHeaders, readable) {response. writeHead (responseStatus, responseHeaders); if (readable = null) response. end (); else readable. on ("open", function () {readable. pipe (response) ;}); return null;} function getMimeNameFromExt (ext) {var result = mimeNames [ext. toLowerCase ()]; // it is best to give a default value if (result = null) result = "application/octet-stream"; return result;}

Step 2-use a regular expression to capture the Range Message Header

With the foundation of this HTTP server, we can now use the following code to process the Range message header. We use a regular expression to split the message header to obtain the start and end strings. Then convert the parseInt () method to the integer number. if the returned value is NaN (not a number), the string is not in the message header. the totalLength parameter shows the total number of bytes of the current file. we will use it to calculate the start and end positions.

function readRangeHeader(range, totalLength) {    /*     * Example of the method 'split' with regular expression.     *      * Input: bytes=100-200     * Output: [null, 100, 200, null]     *      * Input: bytes=-200     * Output: [null, null, 200, null]     */   if (range == null || range.length == 0)    return null;   var array = range.split(/bytes=([0-9]*)-([0-9]*)/);  var start = parseInt(array[1]);  var end = parseInt(array[2]);  var result = {    Start: isNaN(start) ? 0 : start,    End: isNaN(end) ? (totalLength - 1) : end  };     if (!isNaN(start) && isNaN(end)) {    result.Start = start;    result.End = totalLength - 1;  }   if (isNaN(start) && !isNaN(end)) {    result.Start = totalLength - end;    result.End = totalLength - 1;  }   return result;}

Step 3: Check whether the data range is reasonable

Return to the httpListener () function. after the HTTP method is passed, check whether the request data range is available. if the browser does not send a Range message header, the request will be treated as a general request directly. the server returns the entire file, and the HTTP status will be 200 OK. in addition, we will also check whether the start and end locations are larger or equal than the file length. in this case, the requested data range cannot be met. the returned status will be 416 Requested Range Not Satisfiable, and Content-Range will also be sent.

var responseHeaders = {};  var stat = fs.statSync(filename);  var rangeRequest = readRangeHeader(request.headers['range'], stat.size);    // If 'Range' header exists, we will parse it with Regular Expression.  if (rangeRequest == null) {    responseHeaders['Content-Type'] = getMimeNameFromExt(path.extname(filename));    responseHeaders['Content-Length'] = stat.size; // File size.    responseHeaders['Accept-Ranges'] = 'bytes';         // If not, will return file directly.    sendResponse(response, 200, responseHeaders, fs.createReadStream(filename));    return null;  }   var start = rangeRequest.Start;  var end = rangeRequest.End;   // If the range can't be fulfilled.   if (start >= stat.size || end >= stat.size) {    // Indicate the acceptable range.    responseHeaders['Content-Range'] = 'bytes */' + stat.size; // File size.     // Return the 416 'Requested Range Not Satisfiable'.    sendResponse(response, 416, responseHeaders, null);    return null;  }

Step 4: meet the request

Finally, it is confusing. For status 216 Partial Content, we have another format of Content-Range message header, including start, end position, and total number of bytes of the current file. we also have the Content-Length message header, whose value is equal to the difference between the start position and the end position. In the last code, we called createReadStream () and assigned the start and end positions to the object of the second parameter option, this means that the returned stream will only contain read-only data from start to end.

// Indicate the current range.   responseHeaders['Content-Range'] = 'bytes ' + start + '-' + end + '/' + stat.size;  responseHeaders['Content-Length'] = start == end ? 0 : (end - start + 1);  responseHeaders['Content-Type'] = getMimeNameFromExt(path.extname(filename));  responseHeaders['Accept-Ranges'] = 'bytes';  responseHeaders['Cache-Control'] = 'no-cache';   // Return the 206 'Partial Content'.  sendResponse(response, 206,     responseHeaders, fs.createReadStream(filename, { start: start, end: end }));

The following is the complete httpListener () callback function.

function httpListener(request, response) {  // We will only accept 'GET' method. Otherwise will return 405 'Method Not Allowed'.  if (request.method != 'GET') {    sendResponse(response, 405, { 'Allow': 'GET' }, null);    return null;  }   var filename =    initFolder + url.parse(request.url, true, true).pathname.split('/').join(path.sep);   // Check if file exists. If not, will return the 404 'Not Found'.   if (!fs.existsSync(filename)) {    sendResponse(response, 404, null, null);    return null;  }   var responseHeaders = {};  var stat = fs.statSync(filename);  var rangeRequest = readRangeHeader(request.headers['range'], stat.size);   // If 'Range' header exists, we will parse it with Regular Expression.  if (rangeRequest == null) {    responseHeaders['Content-Type'] = getMimeNameFromExt(path.extname(filename));    responseHeaders['Content-Length'] = stat.size; // File size.    responseHeaders['Accept-Ranges'] = 'bytes';     // If not, will return file directly.    sendResponse(response, 200, responseHeaders, fs.createReadStream(filename));    return null;  }   var start = rangeRequest.Start;  var end = rangeRequest.End;   // If the range can't be fulfilled.   if (start >= stat.size || end >= stat.size) {    // Indicate the acceptable range.    responseHeaders['Content-Range'] = 'bytes */' + stat.size; // File size.     // Return the 416 'Requested Range Not Satisfiable'.    sendResponse(response, 416, responseHeaders, null);    return null;  }   // Indicate the current range.   responseHeaders['Content-Range'] = 'bytes ' + start + '-' + end + '/' + stat.size;  responseHeaders['Content-Length'] = start == end ? 0 : (end - start + 1);  responseHeaders['Content-Type'] = getMimeNameFromExt(path.extname(filename));  responseHeaders['Accept-Ranges'] = 'bytes';  responseHeaders['Cache-Control'] = 'no-cache';   // Return the 206 'Partial Content'.  sendResponse(response, 206,     responseHeaders, fs.createReadStream(filename, { start: start, end: end }));}

Test implementation

How can we test our code? As mentioned in the introduction, the most common scenarios of some texts are stream and video playback. Therefore, we create a mainPlayer with the ID and containLabel . The onLoad () function will be triggered when mainPlayer pre-reads the metadata of the current video. this is used to check whether there are numeric parameters in the URL. If yes, mainPlayer will jump to the specified time point.