Talking about HTTPS and fiddler crawling HTTPS protocol

talking about HTTPS and fiddler crawling HTTPS protocol

Recently want to try to do some interface acquisition and processing based on fiddler recording function, one of the problems is that simple connection fiddler only crawl HTTP protocol, critical login requests and other HTTPS protocol are not captured, So fiddler to be able to crawl both HTTPS and HTTP protocol, setting is only a small step, the key is to understand the principle of HTTPS protocol, fiddler crawl HTTPS protocol principle, and then to better understand how to set. This article mainly consists of three parts, the first part of the way of comparing the common image of the principle of HTTPS, the second part is on the basis of the first section of the introduction of Fiddler crawl HTTPS protocol principle, the third part is how to fiddler and mobile phone set to capture the HTTPS protocol.

First, talking about HTTPS

We all know that HTTP is not a secure transport, HTTPS protocol based on HTTPS using SSL protocol is relatively safe. At present, more and more enterprises choose to use the HTTPS protocol to communicate with users, such as Baidu, Google and so on. HTTPS requires a handshake between the client (browser) and the server (Web site) before transmitting the data, which establishes the password information for both parties to encrypt the transmitted data during the handshake. There are a lot of information on the Internet, some of them are too obscure to understand, especially need some knowledge of cryptography. I did a simple collation, shaving complex bottom-level implementation, single from the understanding of the SSL protocol from the perspective of the macroscopic understanding of HTTPS. In a word, HTTPS is a non-symmetric encryption algorithm (such as the RSA algorithm) to negotiate key generation and exchange, and then in the subsequent communication process using a negotiated key for symmetric encrypted communication . The principle and process diagram of the HTTPS protocol transmission is as follows:

HTTPS protocol Transmission principle
In a total of 8 steps, let's take a look at each step to see what's going on:

1. In the first step, the client initiates a clear request: sends a set of cryptographic rules that it supports and a random number (Random_c) to the server.

3. The second step, the server initial response: the server according to its own encryption rules, from the client's request to choose a set of encryption algorithm and hash algorithm, generate random numbers, and their identity information in the form of a certificate (CA) sent back to the browser. The CA certificate contains information such as the server address, the encrypted public key, and the issuing authority of the certificate. At this point the server gives the client the option to use the encryption rule, CA certificate, and a random number (random_s).
5. The third step is to do four things after the client receives the initial response from the server:
   (1) Certificate verification: Verify the legality of the certificate (the authority that issued the certificate is legitimate, the address of the website contained in the certificate is the same as the address being accessed).
   (2) Generate password: The browser generates a random number of passwords (Pre_master) and is encrypted with the public key (Enc_pre_master) in the CA certificate for transmission to the server.
   (3) Calculate the negotiation key:
   At this point the client has obtained all the information needed to calculate the negotiation key: two plaintext random number random_c and random_s with their own calculation generated pre-master, calculated to obtain the negotiated key Enc_key.
   Enc_key=fuc (Random_c, random_s, Pre-master)
   (4) Generate handshake information: Use the agreed hash to calculate the handshake message, and use the negotiated key Enc_key and the agreed algorithm to encrypt the message.
7. In the fourth step, the client sends the data generated by the third step to the server:
   There are three data to be sent here:
   (1) server random number password Enc_pre_master encrypted with public key
   (2) The client to the server notification, "we will have to use the agreed algorithm and negotiation key to communicate the OH."
   (3) The client encrypts the generated handshake information.
9. Fifth step, the server receives the data sent by the client to do the following four things: (1) Private key decryption: use their own private key to decrypt the received enc_pre_master from the password pre_master.
   (2) Calculate the negotiation key: At this point the server has obtained all the information required to calculate the negotiation key: two plaintext random number random_c and random_s and pre-master, calculated to get the negotiated key Enc_key.
   Enc_key=fuc (Random_c, random_s, Pre-master)
   (3) Decryption handshake message: Use the negotiation key Enc_key to decrypt the handshake message from the client and verify that the hash is consistent with the client.
   (4) Generate handshake messages use the negotiated key Enc_key and the agreed-upon algorithm to encrypt a handshake message and send it to the client.
11. In the sixth step, the server sends the data generated by the fifth step to the client:
    There are two data to be sent here:
    (1) The server to the client's notification, "Listen to you, we will use the agreed algorithm and negotiation key for communication Oh."
    (2) The handshake information generated by the server encryption.
13. The seventh step, the client gets handshake information decryption, handshake end.
    The client decrypts and computes the hash of the handshake message, which is the end of the handshake process if it is consistent with the hash of the server.
15. Eighth step, normal encrypted communication
    After the handshake succeeds, all communication data will be encrypted and decrypted by the previously negotiated key Enc_key and the agreed-upon algorithm.

Here the client and the server send an encrypted handshake message to each other and verify that the purpose is to ensure that both sides obtain a consistent password, and can encrypt and decrypt the data normally, for the subsequent transmission of real data to do a test. In addition, HTTPS generally uses the encryption and hashing algorithm as follows: Asymmetric encryption algorithm: RSA,DSA/DSS symmetric encryption algorithm: Aes,rc4,3deshash algorithm: md5,sha1,sha256 where the asymmetric encryption algorithm is used to encrypt the generated password during the handshake, The symmetric encryption algorithm is used to encrypt the data that is actually transmitted, while the hash algorithm is used to verify the integrity of the data. Because the password generated by the browser is the key to the entire data encryption, the asymmetric encryption algorithm is used to encrypt it at the time of transmission. Asymmetric encryption algorithm generates public and private keys, the public key can only be used to encrypt data, so can be transferred at will, and the server's private key is used to decrypt the data, so the server will be very careful to keep their private keys to prevent leakage.

Second, fiddler crawl HTTPS protocol principle

As we all know, Fiddler is a good proxy tool that can crawl protocol requests for debugging. The principle and configuration of Fiddler crawl HTTP protocol is relatively simple, the fiddler and the client configuration, can make fiddler easily get HTTP request. However, due to the particularity of the HTTPS protocol, to further configure the Fiddler, we first need to understand the Fiddler crawl HTTPS protocol principle to better understand how to configure Fiddler. Fiddler itself is a protocol proxy tool, in the previous section of the HTTPS schematic, the client-server communication process is all obtained by Fiddler, that is, as shown:

Fiddler crawl HTTPS protocol schematic diagram

We see that the Fiddler crawl HTTPS protocol is mainly done in the following steps:

1. In the first step, fiddler intercepts the HTTPS request sent by the client to the server, fiddler masquerading as the client sending a request to the server for handshake.

2. The second step, the server sends back the corresponding, fiddler obtains the CA certificate to the server, decrypts with the root certificate public key, verifies the server data signature, obtains the public key of the server CA certificate. Then fiddler forges its own CA certificate and passes the impersonation server certificate to the client browser.
3. The third step, and the normal process of the client's operation, the client based on the returned data for certificate verification, generate password Pre_master, with fiddler forged certificate public key encryption, and generate HTTPS communication with the symmetric key Enc_key.
4. In the fourth step, the client passes important information to the server and is intercepted by fiddler. Fiddler will intercept the ciphertext with its own forged certificate of the private key to get and calculate the HTTPS communication with the symmetric key Enc_key. Fiddler the symmetric key is passed to the server using the server certificate public key encryption.
5. The fifth step, the same as the normal process of server-side operation, the server with the private key after the establishment of trust, and then send an encrypted handshake message to the client.
6. Sixth step, fiddler intercept the ciphertext sent by the server, with the symmetric key to solve, and then use their own forged certificate private key encryption to the client.
7. The seventh step, the client gets encrypted information, with the public key to solve, verify the hash. The handshake process is formally completed, and the client and server side establish a "trust".

How does fiddler act as a third party between the server and the client in the subsequent normal encrypted communication process?

Server-Client : Fiddler received the ciphertext sent by the server, with a symmetric key to unlock, to obtain the plaintext sent by the server. Encrypted again, sent to the client.
client-to- server: The client is encrypted with a symmetric key and is intercepted by the fiddler, and the decryption obtains plaintext. Encrypted again, sent to the server side. Since Fiddler has been communicating with a symmetric key enc_key, the information is transparent to the entire HTTPS communication process.

As can be seen from the above, the key to the success of the Fiddler Crawl HTTPS protocol is the root certificate (which is what, Google), which is the starting point of a chain of trust, which is also the key to fiddler forged CA certificates to gain client and server-side trust.
Next we'll see if the settings let fiddler crawl the HTTPS protocol.

Third, fiddler crawl HTTPS settings

Note the following operations, the premise is that the phone has been able to connect the fiddler, this part of the configuration process is simply not to repeat, can refer to: How to connect the mobile phone fiddler.
How do I continue to configure the Fiddler crawl to the HTTPS protocol?
(i) first set up the Fiddler: Open the toolbar->tools->fiddler Options->https

Setting up the Fiddler
Select Capture HTTPS connects because we want to use Fiddler to get the HTTPS request from the mobile client, so the drop-down menu in the middle selects from remote clients only. Select Ignore server certificate errors below.
(b) Then, the phone is installed fiddler certificate.
This step, which we analyzed above, is the key to crawling HTTPS requests.
The procedure is simple, open the mobile browser, enter the proxy server IP and port in the browser address, and you will see a fiddler provided page.
! [] (//upload-images.jianshu.io/upload_images/1430132-60573b1ef023e99d.png?imagemogr2/auto-orient/strip%7
cimageview2/2/w/1240)
Then click on the bottom of the Fiddlerroot certificate, click OK to install to download the Fiddler certificate.
After the download installation is complete, we use the mobile phone client or browser to make HTTPS requests, Fiddler can intercept, and intercept ordinary HTTP requests as well.

OK, the above is about the introduction of HTTPS and fiddler How to get the principle and configuration of HTTPS protocol, see fiddler Uniform intercept HTTP and complex HTTPS protocol, the heart is a little excited about it.

Sui Fat Lovefat
Links: Https://www.jianshu.com/p/54dd21c50f21
Source: Pinterest
Copyright belongs to the author. Commercial reprint please contact the author for authorization, non-commercial reprint please specify the source.

Talking about HTTPS and fiddler crawling HTTPS protocol