Use proxy_intercept_errors and recursive_error_pages to proxy multiple times 3

302 is a frequently used status code in the HTTP protocol, and is a type of redirection method. Its semantics is often interpreted as "MovedTemporarily ". Here, we will mention that most of the 302 used in reality is misuse (mixed with 303,307). In HTTP1.1, the meaning of it is "Found". 302 is sometimes obvious and sometimes hidden. Simplest

302 is a frequently used status code in the HTTP protocol, and is a type of multiple redirection methods. Its semantics is often interpreted as "Moved Temporarily ". As mentioned above, most of the 302 used in reality are misuse (mixed with 303,307). In HTTP/1.1, its syntax is "Found ". 302 is sometimes obvious and sometimes hidden. Simplest

302 is a frequently used status code in the HTTP protocol, and is a type of multiple redirection methods. Its semantics is often interpreted as "Moved Temporarily ". As mentioned above, most of the 302 used in reality are misuse (mixed with 303,307). In HTTP/1.1, its syntax is "Found ".

302 is sometimes obvious and sometimes hidden. The simplest case is that when we enter A web site A in the browser, the address bar of the browser will automatically jump to B, and then open A Web page, this situation may be 302.

Relatively concealed situations often occur in players embedded in webpages. For example, when you open a youku video playback page, capture packets and observe, and you will often find the 302 shadow. However, because these URLs are not directly opened in the browser, they cannot be changed in the address bar of the browser. Of course, if you specifically copy these specific URLs to the address bar of the browser, it can still be observed.

Youku was mentioned in the previous section. In fact, most online video websites use 302 of the total traffic. The reason is very simple: CDN is used when the traffic of video websites is usually large. The difference is only between self-built CDN and commercial CDN. Because 302 of the redirection semantics (repeated again, 302 of the semantics are widely misused, we may use 302 or 303 when we use 307, but it will not be entangled in this issue later). It can be well combined with scheduling in CDN.

Let's take an example. Open a Netease video playback page, capture the package, and find the url in the 302 status. For example:

http://flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

Copy it to the address bar of the browser and you will find that the address bar is quickly changed to another url. This Url is not fixed and may be:

http://14.18.140.83/f6c00af500000000-1408987545-236096587/data6/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

You can use the curl tool to better understand the entire process:

curl -I "http://flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -LHTTP/1.1 302 Moved Temporarily Server: nginx Date: Mon, 25 Aug 2014 14:49:43 GMT Content-Type: text/html Content-Length: 154 Connection: keep-alive NG: CCN-SW-1-5L2 X-Mod-Name: GSLB/3.1.0 Location: http://119.134.254.9/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 HTTP/1.1 302 Moved Temporarily Server: nginx Date: Mon, 25 Aug 2014 14:49:41 GMT Content-Type: text/html Content-Length: 154 Connection: keep-alive X-Mod-Name: Mvod-Server/4.3.3 Location: http://119.134.254.7/cc89fdac00000000-1408983581-2095617481/data4/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 NG: CHN-SW-1-3Y1 HTTP/1.1 200 OK Server: nginx Date: Mon, 25 Aug 2014 14:49:41 GMT Content-Type: video/mp4 Content-Length: 3706468 Last-Modified: Mon, 25 Aug 2014 00:23:50 GMT Connection: keep-alive Cache-Control: no-cache ETag: "53fa8216-388e64" NG: CHN-SW-1-3g6 X-Mod-Name: Mvod-Server/4.3.3 Accept-Ranges: bytes 

We can see that this process went through 302 twice.

Put this example aside for the moment, and then talk about another important term: proxy. We usually play to say that some leaders are of the 302 type, and some leaders are of the proxy type. 302 a type of leader will quickly transfer a task to another person, while a proxy-type leader will participate in the task and even finish all the tasks.

Back to the example above, if there are multiple 302 s in the middle of a url, what should we do if we need to design a proxy with Nginx to hide all the 302 s in the middle?

1. Original Proxy

We know that Nginx itself is an excellent proxy server. Therefore, we will first set up an Nginx forward proxy with the server IP address 192.168.109.128 (one of my test VMS ).

The initial configuration is simplified as follows:

server {        listen 80;        location / {                rewrite_by_lua '                        ngx.exec("/proxy-to" .. ngx.var.request_uri)                ';        }        location ~ /proxy-to/([^/]+)(.*) {                proxy_pass http://$1$2$is_args$query_string;        }}

The function is

http://192.168.109.128/xxxxxx

When you access this proxy, the proxy will go to the Real Server represented by xxxxxx.

The test results are as follows:

curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -LHTTP/1.1 302 Moved Temporarily Server: nginx/1.4.6 Date: Mon, 25 Aug 2014 14:50:54 GMT Content-Type: text/html Content-Length: 154 Connection: keep-alive NG: CCN-SW-1-5L2 X-Mod-Name: GSLB/3.1.0 Location: http://183.61.140.24/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 HTTP/1.1 302 Moved Temporarily Server: nginx Date: Mon, 25 Aug 2014 14:50:55 GMT Content-Type: text/html Content-Length: 154 Connection: keep-alive X-Mod-Name: Mvod-Server/4.3.3 Location: http://183.61.140.20/540966e500000000-1408983655-236096587/data1/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 NG: CHN-ZJ-4-3M4 HTTP/1.1 200 OK Server: nginx Date: Mon, 25 Aug 2014 14:50:55 GMT Content-Type: video/mp4 Content-Length: 3706468 Last-Modified: Mon, 25 Aug 2014 00:31:03 GMT Connection: keep-alive Cache-Control: no-cache ETag: "53fa83c7-388e64" NG: CHN-ZJ-4-3M4 X-Mod-Name: Mvod-Server/4.3.3 Accept-Ranges: bytes

It can be seen that although the proxy is used, the process is no different from the original access. The access process is:

http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

Nginx will forward the request

http://flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4

The latter will return a 302 immediately. Therefore, as a proxy, Nginx will pass the 302 back to the client, and the client will initiate a new request to repeat the previous 302. here is a problem. Once the Nginx proxy returns 302 at the backend, the client is detached from the Nginx proxy, and Nginx cannot act as a complete proxy.

2. 1st modifications

Modify the configuration file:

server {        listen 80;        location / {                rewrite_by_lua '                        ngx.exec("/proxy-to" .. ngx.var.request_uri)                ';        }        location ~ /proxy-to/([^/]+)(.*) {                proxy_pass http://$1$2$is_args$query_string;                error_page 302 = @error_page_302;        }        location @error_page_302 {                rewrite_by_lua '                        local _, _, upstream_http_location = string.find(ngx.var.upstream_http_location, "^http:/(.*)$")                        ngx.header["zzzz"] = "/proxy-to" .. upstream_http_location                        ngx.exec("/proxy-to" .. upstream_http_location);                ';        }}

The difference is that an error_page is used to continue the proxy with the 302 destination location when the proxy's backend returns 302, rather than directly returning it to the client. And this logic contains the meaning of recursion, tracking 302 all the way until the address of 200 is finally returned. The test results are as follows:

curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -LHTTP/1.1 302 Moved Temporarily Server: nginx/1.4.6 Date: Mon, 25 Aug 2014 15:01:17 GMT Content-Type: text/html Content-Length: 154 Connection: keep-alive NG: CCN-SW-1-5L2 X-Mod-Name: GSLB/3.1.0 Location: http://183.61.140.24/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 HTTP/1.1 302 Moved Temporarily Server: nginx Date: Mon, 25 Aug 2014 15:01:17 GMT Content-Type: text/html Content-Length: 154 Connection: keep-alive X-Mod-Name: Mvod-Server/4.3.3 Location: http://183.61.140.20/a90a952900000000-1408984277-236096587/data1/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 NG: CHN-ZJ-4-3M4 HTTP/1.1 200 OK Server: nginx Date: Mon, 25 Aug 2014 15:01:17 GMT Content-Type: video/mp4 Content-Length: 3706468 Last-Modified: Mon, 25 Aug 2014 00:31:03 GMT Connection: keep-alive Cache-Control: no-cache ETag: "53fa83c7-388e64" NG: CHN-ZJ-4-3M4 X-Mod-Name: Mvod-Server/4.3.3 Accept-Ranges: bytes

The modification is still unsuccessful!
Why? After analysis, we have added a head print statement in the location @ error_page_302. However, this header was not typed in the test, the process does not go to the location @ error_page_302.

The reason is that

error_page 302 = @error_page_302;

Error_page is the return code of this processing by default. As a proxy, the status code is 200 as long as the response of the upstream server is successfully forwarded. that is, what we really need to check is the status code returned by the proxy's backend server, rather than the status code returned by the proxy itself. Check Nginx wiki. The proxy_intercept_errors command is exactly for this:

Syntax: proxy_intercept_errors on | off;Default:    proxy_intercept_errors off;Context:    http, server, locationDetermines whether proxied responses with codes greater than or equal to 300 should be passed to a client or be redirected to nginx for processing with the error_page directive.

3. Second modification

server {        listen 80;        proxy_intercept_errors on;        location / {                rewrite_by_lua '                        ngx.exec("/proxy-to" .. ngx.var.request_uri)                ';        }        location ~ /proxy-to/([^/]+)(.*) {                proxy_pass http://$1$2$is_args$query_string;                error_page 302 = @error_page_302;        }        location @error_page_302 {                rewrite_by_lua '                        local _, _, upstream_http_location = string.find(ngx.var.upstream_http_location, "^http:/(.*)$")                        ngx.header["zzzz"] = "/proxy-to" .. upstream_http_location                        ngx.exec("/proxy-to" .. upstream_http_location);                ';        }}

Compared with the previous modification, the difference is that a proxy_intercept_errors command is added. The test results are as follows:

curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -L HTTP/1.1 302 Moved TemporarilyServer: nginx/1.4.6Date: Mon, 25 Aug 2014 15:05:54 GMTContent-Type: text/htmlContent-Length: 160Connection: keep-alivezzzz: /proxy-to/183.61.140.24/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 

This is even more amazing. I directly returned a 302 status and won't continue to jump.
The problem is that although the first 302 request successfully enters @ error_page_302, the subsequent error_page command does not work. That is to say, error_page only checks the status code returned by the first backend, but does not continue to check the subsequent backend status code.

Check the information. At this time, another command, recursive_error_pages, will be used.

4. 3rd modifications

server {        listen 80;        proxy_intercept_errors on;        recursive_error_pages on;        location / {                rewrite_by_lua '                        ngx.exec("/proxy-to" .. ngx.var.request_uri)                ';        }        location ~ /proxy-to/([^/]+)(.*) {                proxy_pass http://$1$2$is_args$query_string;                error_page 302 = @error_page_302;        }        location @error_page_302 {                rewrite_by_lua '                        local _, _, upstream_http_location = string.find(ngx.var.upstream_http_location, "^http:/(.*)$")                        ngx.header["zzzz"] = "/proxy-to" .. upstream_http_location                        ngx.exec("/proxy-to" .. upstream_http_location);                ';        }}

Compared with the previous command, only the recursive_error_pages on command is added. The test results are as follows:

curl -I "http://192.168.109.128/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4" -L HTTP/1.1 200 OK Server: nginx/1.4.6 Date: Mon, 25 Aug 2014 15:09:04 GMT Content-Type: video/mp4 Content-Length: 3706468 Connection: keep-alive zzzz: /proxy-to/14.18.140.83/f48bad0100000000-1408984745-236096587/data6/flv.bn.netease.com/tvmrepo/2014/8/5/P/EA3I1J05P/SD/EA3I1J05P-mobile.mp4 Last-Modified: Mon, 25 Aug 2014 00:21:07 GMT Cache-Control: no-cache ETag: "53fa8173-388e64" NG: CHN-MM-4-3FE X-Mod-Name: Mvod-Server/4.3.3 Accept-Ranges: bytes

It can be seen that Nginx finally returns 200 successfully. In this case, Nginx truly functions as a Proxy, hiding multiple 302 links in a request and returning only one final result from the client.

5. Summary

To sum up, using commands proxy_pass, error_page, proxy_intercept_errors, and recursive_error_pages, you can hide the jump details of a request from the client and directly return the final result with a 200 status code.

It is strange that there is no recursive_error_pages instruction in the official Nginx wiki.

This article from: http://pureage.info/, original address: http://pureage.info/2014/08/25/hiding-302-using-proxy-pass.html, thanks to the original author to share.