Principle of resumable upload of HTTP files,

Principle of resumable upload of HTTP files,

A colleague ran to me a few days ago and said that we wanted to resume the resumable upload of the AD material video, that is, this time a video was cached in half and we don't need to start over again next time, you can start downloading at the original location. to provide a better user experience.

At the same time, we need to inform the terminal of the last modification time/file signature (md5) through the business interface that supports the spam address. This is used to determine whether the file I want to change, also tell the Size of the terminal file.

I want to solve this problem by changing the existing interface to provide more data. The following is a simple implementation of the principle:

Key points:

For resumable data transfer, there are two key points:

1. The terminal knows whether the current file and the last loaded file have changed. If there is any change, download the file again from the position of offset 0.

2. The terminal records the offset successfully downloaded last time and tells the server that the server can spit data from a specific offset.

 

File Change awareness:

Pre-service interface solution:

For key aspect 1, you can use a pre-service interface to determine the business scenarios of most products. Here is a brief introduction:

For non-download tool products, such as video apps (Qiyi, Youku), the video will request relevant business information before playback, mainly returning the name of the film, the main actors, and other column information, at the same time, a playback address is returned, which is the most important information for playback.

The stream play address is where we can make an article. If the film "The first episode of Prince Edward" is updated (a dirty picture is required by the radio and television ), the backend system allows this service interface to spit out different playback addresses/A Different url parameter (? Ver = 1.1)/location parameter (# ver1.1 ). Such pure-natural URL changes allow terminals to think that they are not the same film, but need to be reloaded.

 

HTPP standard ETAG solution:

How can I solve the problem of downloading tool classes without business interfaces?

There is no front interface for the download tool class. You can use the http etag to identify whether the file has been modified.

ETAG principle: If the resource content on the URL changes, a new and different ETag will be allocated. Using ETag in this way is similarFingerprintAnd they can be quickly compared to determine whether the resources of the two versions are the same. The comparison of ETag is only meaningful to the same URL. The ETag values of resources on different URLs may be the same or different, and they cannot be inferred from the comparison of ETag.

ETAG is an optional field of HTTP and does not conform to its implementation. In fact, many fields in the industry are generated using MD5 signatures (linux shell md5sum)

Typical usage:

Server: Nginx> 1.3.3 module with ETAG. You can also add an ETAG field to SetHeaders in the Business Code.

Client:

First request:

String etag = httpURLConnection. getHeaderField ("ETag ");

ETag: "b428eab9654aa7c87091e"

Second request (resumable upload ):

HttpURLConnection. setRequestProperty ("If-None-Match", "b428eab9654aa7c87091e ");

If-None-Match: "b428eab9654aa7c87091e"

If the ETag value matches, this means that the resource has not changed, and the server will send a very short response, including the HTTP "304 not modified" status. The status 304 tells the client that the cached version is the latest and should be used.

However, if the ETag value does not match, this means that the resource may change, then a complete response will be returned, including the resource content, it seems that ETag is not used. In this case, the client can replace the previous Cache version with the newly returned resource and the new ETag.

 

Resume support:

For a C/C ++ programmer, a system-level implementation solution will be obtained immediately:

1. The client uploads the current offset

2. The server seek starts to read data from the http connection at the specified offset of the file.

However, in an era where open solutions and standards are constantly improved, we do not need to implement one by ourselves (this is also the reason why C/C ++ R & D engineers like me are declining ), let's see how the HTTP protocol solves this problem:

HTTP header Range field:

Range: used for client-to-server requests. You can use this field to specify the size and unit of a file to be downloaded. Typical formats include:

Range: bytes = 0-499 download content in the Range of bytes 0-499
Range: bytes = 500-999 download content in the Range of 500-999 bytes
Range: bytes =-500 download the last 500 bytes of content
Range: bytes = 500-download content starting from 500th bytes to the end of the file

Here is a simple and crude example.

Curl -- header "Range: bytes = 0-20000" xxx.com/memcache?- o part1
Curl -- header "Range: bytes = 20001-223651" xxx.com/memcache?- o part2
Cat part1 part2> a.pdf

 

Derivative reading:

Use ETags to reduce Web application bandwidth and Load