Introduction: X-Forwarded-For in Http network protocol

Introduction: X-Forwarded-For in Http network protocol

I always think that for those who are engaged in Web Front-end development, HTTP and other common network knowledge are essential. On the one hand, a lot of front-end work, such as Web performance optimization, most of the rules are directly matched with the features of HTTP, HTTPS, SPDY, TCP and other protocols. If we do not start from the Protocol itself, we simply follow the principle, it is likely to be counterproductive. On the other hand, with the development and growth of Node, more and more front-end students begin to write server programs, or even frameworks (ThinkJS is developed by front-end engineers and has a Node framework that is used by many front-end engineers ), mastering the necessary network knowledge is critical to the security, deployment, and O & M of server programs.

Background

X-Forwarded-For is an extension header. HTTP/1.1 (RFC 2616) protocol does not define it. It was first introduced by the cache proxy software Squid to indicate the real IP address of the HTTP request, it has become a de facto standard and is widely used by major HTTP proxy, Server Load balancer, and is written into RFC 7239 (Forwarded HTTP Extension) standard.

The format of the X-Forwarded-For request header is very simple, as shown in the following code:

X-Forwarded-For: client, proxy1, proxy2

As you can see, the content of XFF consists of multiple parts separated by commas (,) and spaces. The first part is the device IP address farthest from the server, and then the IP address of each level of proxy device.

If an HTTP request passes through three proxies Proxy1, Proxy2, and Proxy3 before arriving at the server, the IP addresses are IP1, IP2, and IP3 respectively, and the real IP address of the user is IP0, then according to the XFF standard, the server will eventually receive the following information:

X-Forwarded-For: IP0, IP1, IP2

Proxy3 directly connects to the server. It will append IP2 to XFF, indicating that it is forwarding the request for Proxy2. The list does not contain IP3. IP3 can be obtained through the Remote Address field of the server. We know that an HTTP connection is based on a TCP connection. The HTTP protocol does not have an IP Address. The Remote Address comes from a TCP connection, indicating the IP Address of the device that establishes a TCP connection with the server. In this example, It is ip3.

The Remote Address cannot be forged because three handshakes are required to establish a TCP connection. If the source IP Address is forged, a TCP connection cannot be established, and no subsequent HTTP requests are generated. Different languages use different methods to obtain Remote addresses. For example, php is $ _ SERVER ["REMOTE_ADDR"], and Node is req. connection. remoteAddress, but the principles are the same.

Problem

With the above background knowledge, I started to talk about the problem. I used Node to write a simple Web Server for testing. The HTTP protocol has nothing to do with the language. Here, Node is used for convenience demonstration, and the same conclusion can be obtained in any other language. In addition, Nginx is also used in this article. If you are interested, the same applies to Apache or other Web servers.

The following code listens to port 9009 and outputs some information after receiving the HTTP request:

var http = require('http');http.createServer(function (req, res) {res.writeHead(200, {'Content-Type': 'text/plain'});res.write('remoteAddress: ' + req.connection.remoteAddress + '\n');res.write('x-forwarded-for: ' + req.headers['x-forwarded-for'] + '\n');res.write('x-real-ip: ' + req.headers['x-real-ip'] + '\n');res.end();}).listen(9009, '0.0.0.0');

In addition to the Remote Address and X-Forwarded-For described earlier, this Code also has an X-Real-Ip, which is a custom header. X-Real-Ip is usually used by the HTTP proxy to indicate the IP address of the device that creates a TCP connection with it. This device may be another proxy or a Real request end. It should be noted that X-Real-Ip does not belong to any standards currently. The proxy and Web application can use any custom header to transmit this information.

Now you can use the domain name + port number to directly access the Node service, and then configure an Nginx reverse proxy:

location / {proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header Host $http_host;proxy_set_header X-NginX-Proxy true;proxy_pass http://127.0.0.1:9009/;proxy_redirect off;}

My Nginx listens to port 80, so you can access the services forwarded by Nginx without the port.

Test direct access to the Node service:

curl http://t1.imququ.com:9009/remoteAddress: 114.248.238.236x-forwarded-for: undefinedx-real-ip: undefined

Since my computer is directly connected to the Node service, Remote Address is my IP Address. At the same time, I did not specify additional custom headers, so the last two fields are undefined.

Then, access the services forwarded by Nginx:

curl http://t1.imququ.com/remoteAddress: 127.0.0.1x-forwarded-for: 114.248.238.236x-real-ip: 114.248.238.236

This time, my computer accesses the Node service through Nginx, and the obtained Remote Address is actually the local IP Address of Nginx. The above two lines in Nginx configuration work, adding two additional custom headers for the request:

proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

In fact, deploying Web applications in the production environment generally adopts the second method above, which has many advantages. What is not the focus of this article. This introduces a hidden risk: Many Web applications obtain the IP address from the HTTP request header to obtain the real IP address of the user.

The HTTP request header can be constructed at will. We use the-H Parameter of curl to construct X-Forwarded-Fox and X-Real-Ip, and then test it.

Direct access to the Node service:

curl http://t1.imququ.com:9009/ -H 'X-Forwarded-For: 1.1.1.1' -H 'X-Real-Ip: 2.2.2.2'remoteAddress: 114.248.238.236x-forwarded-for: 1.1.1.1x-real-ip: 2.2.2.2

For Web applications, X-Forwarded-Fox and X-Real-Ip are two common request headers. Naturally, they are output without any processing. This indicates that for the direct connection deployment method, the IP information carried in the request header cannot be sent except the Remote Address obtained from the TCP connection.

Access Services forwarded by Nginx:

curl http://t1.imququ.com/ -H 'X-Forwarded-For: 1.1.1.1' -H 'X-Real-Ip: 2.2.2.2'remoteAddress: 127.0.0.1x-forwarded-for: 1.1.1.1, 114.248.238.236x-real-ip: 114.248.238.236

This time, Nginx will append my IP address after X-Forwarded-For and overwrite the X-Real-IP request header with my Ip address. This indicates that with Nginx processing, the final section X-Forwarded-For and X-Real-Ip cannot be constructed and can be used to obtain the user IP address.

User IP addresses are often used in scenarios related to Web security, such as checking user logon regions and IP-based Access Frequency Control. In this scenario, it is more important to ensure that IP addresses cannot be constructed. After the previous tests and analysis, the Remote Address obtained from the TCP connection must be used for Web applications directly deployed for users. For Web applications deployed with reverse proxy such as Nginx, after correctly configuring the Set Header behavior, you can use the last section of X-Real-Ip or X-Forwarded-Ip transmitted by Nginx (in fact, they must be equivalent ).

Then, how does the Web application determine whether the request is directly sent or forwarded by a controllable proxy? Adding additional request headers during proxy forwarding is a method, but it is not safe because the request header is too easy to construct. If this is the case, the custom header must be long enough to be rare and kept confidential.

It is also a way to determine whether the Remote Address is a local IP Address, but it is not perfect because the Remote Address is 127.0.0.1 for access on the Nginx server, whether directly connected or through the Nginx proxy. This problem can be ignored normally. What's more troublesome is that the reverse proxy server and the actual Web application are not necessarily deployed on the same server. Therefore, it is more reasonable to collect the list of all Proxy Server IP addresses. After obtaining the Remote Address, the Web application compares them one by one to determine the access method.

Generally, in order to simplify the logic, the production environment will block the form of direct access to Web applications through a port, and only access through Nginx is allowed. Is that okay? Not necessarily.

First, if the user accesses Nginx through proxy, the last section of X-Forwarded-For and X-Real-Ip get the proxy IP address, this can only be used in security-related scenarios, however, in some scenarios where the weather is displayed based on the IP address, you need to obtain the real IP address of the user as much as possible. In this case, the first IP address in X-Forwarded-For can be used. At this time, you need to pay attention to a problem, or take the previous example for testing:

curl http://t1.imququ.com/ -H 'X-Forwarded-For: unknown, <>"1.1.1.1'remoteAddress: 127.0.0.1x-forwarded-for: unknown, <>"1.1.1.1, 114.248.238.236x-real-ip: 114.248.238.236

The last section of X-Forwarded-For is appended to Nginx, but the previous sections are all from the request headers received by Nginx. The input content is completely untrusted. You must be extremely careful when using the service. It must comply with the IP Format before use. Otherwise, it is easy to cause SQL injection, XSS, and other security vulnerabilities.

Conclusion

When performing security-related operations on Web applications that directly provide external services, they can only obtain IP addresses through Remote Address and cannot trust any request headers;

Web applications that use Nginx and other Web servers for reverse proxy are correctly configured, use the last section X-Forwarded-For or X-Real-Ip to obtain the IP Address (because the Remote Address obtains the Intranet IP Address of the Nginx server ); meanwhile, Web applications should be prohibited from providing services externally;

In security-independent scenarios, For example, if the IP address is used to display the weather in the region, you can obtain the IP address from the front of X-Forwarded-For, but you need to verify the validity of the IP address format;

PS: It is not reasonable to configure Nginx like this in some articles on the Internet:

proxy_set_header X-Real-IP $remote_addr;proxy_set_header X-Forwarded-For $remote_addr;

After this configuration, the security is indeed improved, but all the proxy information before the request arrives at Nginx is erased, so it cannot provide better services for users who actually use the proxy. We should also understand the intermediate principles and analyze specific scenarios.