Crawl HTTPS traffic with Fiddler on the server

From: http://yoursunny.com/t/2011/FiddlerHTTPS/crawling HTTPS traffic on the server with fiddler

Sunshine Boy published in 2011-03-19

In the process of developing Internet applications, network interfaces are often established or utilized. In order to debug the use of network interfaces, it is often necessary to look at network traffic or packets that flow into and out of the network interface. The "Grab tool" is a class of tools used to record data over a network interface.

We know that the network protocol is layered, and the OSI model divides the network protocol into 7 levels. Different gripper tools can work on different levels of network protocols. The commonly used Wireshark is a gripper tool that works on the network layer, not only with a large number of protocol analyzers, but also by writing a Wireshark plugin to identify the custom protocol. Although the Wireshark is powerful, it does not solve all of the problem of grasping packets, the reason is:

• Wireshark work in the network layer, and if the computer is configured with IPSec Transport layer encryption, traffic at the network layer is encrypted and nothing is seen.
• Today a large number of network interfaces use HTTPS encryption, Wireshark cannot crawl to the plaintext content of HTTPS traffic.

So, how do you crawl the plaintext content of the HTTPS request? You can do it with fiddler.

Fiddler is an HTTP capture software written by Eric Lawrence. Fiddler works on the application tier as an HTTP proxy server for other programs. It can crawl and analyze HTTP traffic directly, or it can crawl and analyze HTTPS traffic as an "intermediary."

Fiddler How to Crawl HTTPS traffic

TLS is an end-to-end Transport Layer encryption protocol, which is an integral part of the HTTPS protocol. When the HTTPS site is accessed, HTTP requests and responses are encrypted between the browser and server via the TLS protocol, and the confidentiality and integrity of the data is ensured through digital certificate technology, and any "middleman", including a proxy server, can only forward data without eavesdropping or tampering with the data.

To crawl the plaintext content of HTTPS traffic, fiddler must decrypt HTTPS traffic. However, the browser will check the digital certificate and discover that the session has been bugged. To fool the browser, fiddler encrypts HTTPS traffic by using another digital certificate. After fiddler is configured to decrypt HTTPS traffic, a CA certificate named Do_not_trust_fiddlerroot is automatically generated and used to issue a TLS certificate for each domain name. If the Do_not_trust_fiddlerroot certificate is included in the list of trusted CAs for the browser or other software, the browser or other software will assume that the HTTPS session is trustworthy and will not eject a "certificate error" warning.

When the HTTPS traffic decryption function is turned on, fiddler will prompt the user to include the Do_not_trust_fiddlerroot certificate in the list of trusted CAs for Internet Explorer. This is sufficient for debugging the client, and Firefox users can easily import the Do_not_trust_fiddlerroot certificate manually. However, it is not enough to crawl HTTPS requests made by ASP on the server-you must import the Do_not_trust_fiddlerroot certificate into the trusted CA list of machine account.

Operation Recording

The video shows the following actions:

1. Enable fiddler HTTPS traffic decryption function
2. Import the Do_not_trust_fiddlerroot certificate into the trusted CA list for machine account
3. Set the proxy server for PHP scripts to 127.0.0.1:8888 and crawl HTTPS requests

Packet capture configuration for major network development frameworks

The Fiddler works as a proxy server (port number 8888). You can use Fiddler as long as the development framework supports setting up an HTTP proxy server.

#PHP Curl$ch=Curl_init(' Https://ajax.aspnetcdn.com/ajax/jQuery/jquery-1.5.1.min.js ');curl_setopt($ch,curlopt_returntransfer,1);  ( $ch ,curlopt_proxy , ' 127.0.0.1:8888 ' );  Set up a proxy server curl_setopt ( $ch curlopt_ssl_verifypeer,0);  This is required if PHP compiles without OpenSSL  $resp =curl_exec $ch curl_close $ch      

<!--ASP. Web. config--><configuration> <system.net>  <defaultproxy enabled = "false" > < Proxy proxyaddress=/> </defaultproxy>  </system.net></configuration                 

If you are using a Linux server, install Fiddler on a Windows computer and tick allow remote computers to connect at Tools-fiddler options-connections and manually import the fiddlerroot.cer into the trust CA list of the Linux server, and finally set the proxy server to port 8888 of the IP that the fiddler resides on.

Crawl HTTPS traffic with Fiddler on the server