Analysis of HTTP Response chunked format

Sometimes, when the server generates an HTTP response, the information size cannot be determined. In this case, Content-Length cannot be used to write the length in advance, but the message length needs to be generated in real time. In this case, the server generally uses chunked encoding.

During chunked encoding transmission, the header of the reply message contains transfer-coding and is defined as chunked, indicating that the content will be transmitted using chunked encoding.

 

Chunked encoding is composed of several chunks.The length is 0.. Each Chunk is divided into two parts: the header and the body. The header content specifies the total number of characters in the next body section (Hexadecimal number) And the number unit (generally not written). The body part is the actual content of the specified length.Carriage return line feed (CRLF). In the last chunk with a length of 0, the content is called footer, which is some additional header information (which can be ignored directly ).

 

Let's simulate the data structure:
[Chunk size] [Press enter] [chunk data body] [Press enter] [chunk size] [Press enter] [chunk data body] [Press enter] [0] [Press enter] [footer content (if any) )] [Press enter]

Note that chunk-size is represented by hexadecimal ASCII Code. For example, 86ae (the actual hexadecimal format should be: 38366165) is calculated as length: 34478, it indicates that there are 34478 consecutive bytes of data after the carriage return.
After tracking the returned data of www.yahoo.com, it is found that there will be more spaces in chunk-size. It may be a fixed length of 7 bytes. If the length is less than 7 bytes, it is supplemented by space. The ASCII code of the space is 0x20.

 

Decoding process:
The purpose of chunked encoding decoding is to combine chunk-data in chunked blocks into one block as the reporting style, and record the length of the chunked block.
The decoding process included in rfc2616 is as follows: (pseudo code)
Length: = 0 // set the length counter to 0
Read chunk-size, chunk-extension (if any) and CRLF // read chunk-size, chunk-Extension
// And CRLF
While (chunk-size> 0) {// indicates not last-chunk
Read chunk-data and CRLF // read chunk-size chunk-data, skip CRLF
Append Chunk-data to entity-body // append this chunk-data block to entity-body
Read chunk-size and CRLF // read the chunk-size and CRLF of the new chunk
}
The format of read entity-header // entity-header is name: valuecrlf. If it is null, only CRLF
While (entity-header not empty) // that is, it is not a blank line with only CRLF
{
Append entity-header to existing header fields
Read entity-Header
}
Content-Length: = length // The length of the new message style calculated after the entire decoding process is completed
// Write the message as the value of the Content-Length Field
Remove "chunked" from transfer-encoding // remove the chunked mark from the domain value in transfer-encoding at the same time

 

Sampleencoded response

HTTP/1.1 200 OKContent-Type: text/plainTransfer-Encoding: chunked25This is the data in the first chunk1Aand this is the second one0

Same as abve, raw bytes in hex

0000-000F   48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d   HTTP/1.1 200 OK.0010-001F   0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74   .Content-Type: t0020-002F   65 78 74 2f 70 6c 61 69 6e 0d 0a 54 72 61 6e 73   ext/plain..Trans0030-003F   66 65 72 2d 45 6e 63 6f 64 69 6e 67 3a 20 63 68   fer-Encoding: ch0040-004F   75 6e 6b 65 64 0d 0a 0d 0a 32 35 0d 0a 54 68 69   unked....25..Thi0050-005F   73 20 69 73 20 74 68 65 20 64 61 74 61 20 69 6e   s is the data in0060-006F   20 74 68 65 20 66 69 72 73 74 20 63 68 75 6e 6b    the first chunk0070-007F   0d 0a 0d 0a 31 41 0d 0a 61 6e 64 20 74 68 69 73   ....1A..and this0080-008F   20 69 73 20 74 68 65 20 73 65 63 6f 6e 64 20 6f    is the second o0090-009F   6e 65 0d 0a 30 0d 0a 0d 0a                        ne..0....

Same as abve, in Java code

public static final byte[] CHUNKED_RESPONSE;static { StringBuilder sb = new StringBuilder();sb.append("HTTP/1.1 200 OK/r/n");sb.append("Content-Type: text/plain/r/n");sb.append("Transfer-Encoding: chunked/r/n/r/n");sb.append("25/r/n");sb.append("This is the data in the first chunk/r/n"); // 37 bytes of payload// (conveniently consisting of ASCII characters only)sb.append("/r/n1A/r/n");sb.append("and this is the second one"); // 26 bytes of payload// (conveniently consisting of ASCII characters only)sb.append("/r/n0/r/n/r/n");CHUNKED_RESPONSE = sb.toString().getBytes(java.nio.charset.Charset.forName("US-ASCII"));}

Decoded data

This is the data in the first chunkand this is the second one

Basically, the checked encoding method is used.