Three methods to decrypt HTTPS traffic

Three methods to decrypt HTTPS traffic

Web security is a system engineering. Any minor negligence may cause the collapse of the entire security barrier. For HTTPS, it provides three security guarantees: Content encryption, data integrity, and identity authentication, it may also be affected by risks such as illegal root certificates, server configuration errors, SSL Library Vulnerabilities, and private key theft. Many people think that a small green lock is absolutely safe before the website address they visit. This article introduces the three most common HTTPS traffic decryption methods and principles to discuss the security risks of HTTPS.

Man-in-the-middle

Man-in-the-middle (MITM) is used to create connections with the two ends of network communication and exchange the received data, so that both ends of the Communication think that they are directly talking to each other. In fact, the entire session is controlled by the intermediary. In short, the man-in-the-middle is the client, and the real client thinks that the man-in-the-middle is the server.

There are various ways to implement man-in-the-middle attacks. We will not discuss them here. Some common HTTP/HTTPS packet capture debugging tools are used to create a local Proxy service and modify the browser Proxy settings to intercept traffic. Their working principles are consistent with man-in-the-middle attacks. Tools I have used include Fiddler, Charles, and whistle. The HTTP common proxy I introduced in "HTTP Proxy Principle and Implementation (I)" plays the HTTP man-in-the-middle role.

This article mainly discusses the HTTPS man-in-the-middle. The following is a simple example:

Server <---> Local Proxy <---> Browser         ^                 ^       HTTPS(1)          HTTPS(2)

The above HTTPS (1) connection is a connection established between the man-in-the-middle impersonate the client and the server. Because the HTTPS server generally does not authenticate the client identity, this step is usually normal. For HTTPS (2) connections, if a man-in-the-middle wants to impersonate a server, the man-in-the-middle must have the certificate and private key for the corresponding domain name. To obtain the private key, attackers can only use these methods: 1) go to the website server to get the certificate; 2) issue the certificate from the CA; 3) issue the certificate by yourself.

To prevent the first two points, the website should be protected in all aspects, from host security to website security (to prevent private key theft ), from domain name resolution security to domain name mailbox Security (prevent attacker from re-signing ). The certificate issued by the attacker cannot pass the system's built-in root certificate verification. By default, the certificate cannot be used for man-in-the-middle attacks.

For Fiddler debugging tools, the key to decrypting HTTPS traffic is that they will import their certificates to the Trusted Root Certificate list of the system, in this way, their self-signed documents can be trusted by browsers. Go to the "HTTPS" Tab in the Fiddler settings and check related functions to decrypt and modify HTTPS traffic. In this case, you can see the certificate chain in the browser:

RSA Private Key

In the article "Debug HTTP/2 traffic with Wireshark", I wrote: Wireshark's packet capture principle is to directly read and analyze NIC data. There are two ways to decrypt HTTPS traffic: 1) if you have an HTTPS website's encrypted private key, it can be used to decrypt the encrypted traffic of the website; 2) Some browsers support storing the symmetric key used in TLS sessions in external files, wireshark encryption is supported. This article introduces the second solution. This article briefly introduces the first solution.

Enable Wireshark's SSL protocol settings. For more information, see Add the IP address, port, protocol, and certificate private key (the private key must be saved in PEM format ):

Then visit the website corresponding to the private key to see that the traffic has been decrypted:

The encrypted data in is transmitted through HTTP/1. Can this method decrypt HTTP/2 traffic? The conclusion is: No! The specific cause is analyzed slowly.

We know that key exchange and server authentication are required in the TLS handshake phase, key Exchange aims to generate a shared Key Premaster Secret that only the communication parties know in the insecure data channel, and then generate the Master Secret and the symmetric encryption Session Key and MAC Key in the future. The client performs server authentication to ensure that it is connected to a legitimate server with the private key of the website.

The most common key exchange method is RSA. The following figure clearly describes the process:

Image Source

Client Random and Server Random are transmitted in plaintext, which can be directly viewed by middlemen. After the client generates a Premaster Secret, it is encrypted with the public key of the server certificate and sent. If the server has the corresponding private key, it can be decrypted to obtain the Premaster Secret. In this case, the Client and Server have the same Client Random, Server Random, and Premaster Secret, and the same Subsequent Key can be calculated respectively.

We can see that this method combines two steps: Key Exchange and server authentication. If the server can decrypt the Premaster Secret, it means that the server has the correct private key. If the intermediary does not have a private key and cannot obtain the Premaster Secret, the subsequent traffic cannot be decrypted.

For Wireshark, After configuring the private key of a website, you can easily decrypt the encrypted traffic of the Website "using RSA for key exchange.

Obviously, there is a big problem with RSA key exchange: There is no Forward security (Forward Secrecy ). This means that the attacker can store the encrypted traffic first, and once the private key is obtained, all the traffic can be successfully decrypted.

 

In fact, most HTTPS traffic currently uses ECDHE key exchange. ECDHE uses the DH (Diffie-Hellman) algorithm of the elliptic curve (ECC. Is the DH key exchange process:

Image Source

Server DH Parameter in is signed with the certificate private key. The client can use the certificate public key to verify the validity of the Server. Compared with RSA key exchange, DH changes from passing the Premaster Scret to the Parameter required to pass the DH algorithm, and then both parties calculate the Premaster Secret respectively.

In this case, because the Premaster Secret does not need to be exchanged, even if the intermediary has a private key, it cannot obtain the Premaster Secret and Master Secret. That is to say, Wireshark cannot decrypt the encrypted traffic of "using ECDHE for Key exchange" by configuring RSA Private Key. Of course, after using ECDHE, even though the man-in-the-middle won't be able to decrypt the previous traffic, he can perform the MITM attack to decrypt the traffic, so the private key should be kept properly.

Compared with RSA, it can be used for both key exchange and digital signature. ECC has a clear distinction: ECDHE is used for key exchange, and ECDSA is used for digital signature. That is, there are three mainstream options for key exchange + signature:

RSA key exchange, RSA digital signature, ECDHE Key Exchange, RSA digital signature, ECDHE Key Exchange, and ECDSA digital signature;

The following are websites using these three key exchange methods in Chrome:

HTTP/2 can only use TLSv1.2 +, and hundreds of herhersuite types are disabled (For details, refer to TLS 1.2 Cipher Suite Black List ). In fact, the CipherSuite that HTTP/2 allows must adopt a key exchange algorithm with forward security, and RSA key exchange is not allowed. This is also why RSA Private Key cannot decrypt HTTP/2 encrypted traffic.

SSLKEYLOGFILE

Both Firefox and Chrome Save the Premaster Secret or Master Secret generated by each HTTPS connection when the system environment variable has an SSLKEYLOGFILE file path. With this file, Wireshark can easily decrypt HTTPS traffic, even if ECDHE is used for key exchange with forward security, such:

For more information about this solution, see the article "Debug HTTP/2 traffic with Wireshark.

The SSLKEYLOGFILE file records the most important encryption information in HTTPS data transmission. If it is not for debugging purposes, no one will take the initiative to configure this environment variable, therefore, this solution will not affect HTTPS security.

Summary

Tools such as Fiddler can decrypt HTTPS traffic by importing the root certificate to the system and act as the intermediary. To prevent real HTTPS man-in-the-middle attacks, the website must keep its certificate private key and Domain Name authentication information. To prevent illegal CA issuance of its own website certificate to a third party, enable Certificate Transparency, HTTP Public Key Pinning, and other policies as much as possible. Users should not trust unknown certificates, or import certificates to the root Certificate list at will, we also need to develop the habit of checking frequently used website certificate chains.

RSA key exchange has no forward security, which means that all previously encrypted traffic can be unlocked once the private key is leaked. To this end, the website must enable the CipherSuite that uses ECDHE as the key exchange, or directly use the ECC certificate. The user must discard the antique operating system and browser that does not support ECDHE.

For browsers, there is no secret to HTTPS. Wireshark can easily decrypt HTTPS traffic through the SSLKEYLOGFILE file generated by the browser. In addition, if the browser is installed with malicious extensions, even if you access a secure HTTPS website, the submitted data can be intercepted. Security issues caused by attackers can not be solved through HTTPS.