Analytic problem of Http/https mirror traffic

Scene

Due to business needs, customers provide HTTP/HTTPS traffic mirroring, allowing us to analyze for abnormal behavior. Since the input source of our system can only be a log, the quickest way is to convert the HTTP/HTTPS traffic to the HTTP log to meet the needs of the exception analysis.

Traffic mirroring or Copy mode port mirroring

Through the port mirroring function on the switch or router, the data traffic of one or more source ports is forwarded to a specified port for listening to the network, and the specified port is called "Mirror Port" or "destination port", without seriously affecting the normal throughput of the source port. Network traffic can be monitored and analyzed through a mirrored port. In order to obtain HTTP/HTTPS request and response data, two-way mirroring is done.

Advantages:
1, low cost, do not need to increase any network equipment;
2, when the Port mirroring session (session) is started, the performance of the switch is basically not affected;
3. All user access request data can be collected from the switch;
4, fault protection, when the acquisition system or front-mounted machine failure, the existing network and business has no impact.

Disadvantages:
1, occupy the switch port: The acquisition system needs and the switch direct connection, will occupy the switch a certain number of GE and FE ports;
2, need to modify the switch configuration: The configuration of the switch needs to be modified to copy the appropriate traffic to the mirror port, but when the configuration changes will not affect the performance and business of the switch;

Light Splitter

For some nodes, the broadband access server through the optical port GE link directly with the core router (typically Cisco GSR) connected to the broadband access server and GSR are not supported port mirroring, the use of splitter for the flow of data acquisition is the most appropriate method. When some nodes of the core switch, the aggregation layer switch does not have enough GE Port, not suitable for the port image for the traffic acquisition, or want to collect the network traffic at the exit, we can collect the optical splitter for the flow. Optical splitter is a passive optical device, through the physical layer of optical replication to the user access request data acquisition, its advantages and disadvantages are as follows.

Advantages:
1, excellent performance: can support GE even on the 2.5Gbps POS link through the optical splitter for the flow of data acquisition;
2, fault protection: When the acquisition system failure, the existing network and business has no impact;
3, no need to modify any configuration of existing network equipment, do not change the network structure, can collect all the network traffic, and network seamless integration;
4, High reliability: The spectrometer is a passive optical device, can be regarded as a special optical fiber, high reliability;
5, does not occupy the network equipment port, the input cost is low.

Disadvantages:
The need to change the device's on-line optical fiber splitter, which involves a simple network cut, which will lead to network instantaneous interruption (not more than 5 seconds), the business has a slight impact;

Iptables Tee Module

Use iptables to perform analysis on other machines that mirror traffic on a Web server to the same network segment

Advantages:
1, pure software, easy to use
2, no impact on existing network and business
3, retain most of the original data packet raw information

Disadvantages:
1, the source Mac and destination MAC address of the package will be changed
2, must be the same network segment to do traffic mirroring

Traffic Replication related software tools

Using the Tcpcopy,gor tool, only the traffic copy for the HTTP protocol, the response data that requires the backend of the real business, and the need to install the agent on the customer's Web server, are typically used to test the drainage of the environment.

HTTP traffic mirroring converted to log

Mirroring the traffic to the Web server through the port mirror of the switch to a free network port on the Analysis server. Using the Bro tool, the network port of the Analysis server to which traffic is mirrored is automatically crawled by default, and HTTP-related data is logged to the default HTTP log file, and the log format contains most of the data for HTTP request and response. However, this tool does not support HTTPS.

The official address is as follows:
https://www.bro.org/
Https://github.com/bro/bro

HTTPS traffic mirroring converted to log

This paper studies the Bro,snort,wireshark and other network monitoring tools, and draws the following conclusions:

1, the IDs tool like Bro,snort does not support HTTPS
The 2,wireshark (command line has the Tshark tool) can decrypt HTTPS traffic by importing the HTTPS server-side private key, which decrypts the encryption key exchange algorithm as RSA traffic.
Official Document: Https://wiki.wireshark.org/SSL

Since Wireshark can meet some of the needs, we continue to follow. Key exchange algorithm is the ciphersuite inside, so we first understand the ciphersuite.

Encryption Suite

Cryptographic Suite (Ciphersuite) is an important parameter that needs to be negotiated in the SSL handshake. The client will take the list of ciphersuite that it supports in the customer hello, and the server will select one from it and return it through the servers hello. If the client-supported Ciphersuite list does not intersect with the Ciphersuite list configured on the server side, the negotiation cannot be completed and the handshake fails.

Ciphersuite contains a variety of technologies, such as authentication Algorithms (authentication), cryptographic algorithms (encryption), message authentication code algorithms (Messages authentication code, MAC), key exchange algorithms (key Exchange) and the key derivation algorithm (key derivation Function).

The ciphersuite negotiation mechanism for SSL has good extensibility, and each ciphersuite needs to be registered in the IANA and is assigned a two-byte flag. All Ciphersuite can be viewed on the IANA TLS Cipher Suite Registry page.

All ciphersuite supported by the OpenSSL library can be viewed with the following commands:

# openssl ciphers -V0xC0,0x30 - ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 Kx=ECDH     Au=RSA  Enc=AESGCM(256) Mac=AEAD

1,0XC0,0X30 is the ciphersuite number, which is used in the SSL handshake.
2,ecdhe-rsa-aes256-gcm-sha384 is the name of the cryptographic suite
The 3,tlsv1.2 identity is used for the TLSv1.2 protocol. The predecessor of TLS (Transport layer Security, Transport layer safety) is the SSL (secure Sockets layer), the first few versions (SSL 1.0, SSL 2.0, SSL 3.0) developed by Netscape, from 3.1 began to be standardized by the IETF and renamed, the development so far has TLS 1.0, TLS 1.1, TLS 1.2 three versions. SSL 1.0 has never been exposed, and SSL 2.0 and SSL 3.0 are security issues and are not recommended for use. Nginx starts with 1.9.1 only three versions of TLS are supported by default
4,KX=ECDH identification using ECDH for key exchange
5,AU=RSA identification using RSA for authentication
6,ENC=AESGCM (256) identifies symmetric encryption using AESGCM (256)
7,mac=aead identity chacha20-poly1305 is a aead mode that does not require a MAC algorithm

Take Nginx and Firefox as an example to see the server-side and client-side encryption suites supported

1, view the instructions for configuring the Nginx server-side cryptographic suite as follows:

Syntax: ssl_ciphers ciphers;Default:    ssl_ciphers HIGH:!aNULL:!MD5;Context:    http, serverSpecifies the enabled ciphers. The ciphers are specified in the format understood by the OpenSSL library, for example:      ssl_ciphers ALL:!aNULL:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;The full list can be viewed using the “openssl ciphers” command.The previous versions of nginx used different ciphers by default.

2, corresponding to the client side of HTTPS, there will be a corresponding supporting encryption suite, such as curl, browser, etc. will have the relevant standards. The following are the encryption suites supported by Firefox:
Https://wiki.mozilla.org/Security/Server_Side_TLS

Knowing the cryptographic suite, we know which Ciphersuite are using the RSA key exchange algorithm. The next question is why the server side of the HTTPS certificate, only the RSA key exchange algorithm Ciphersuite can be cracked. Like the mainstream DH/ECDH key exchange algorithm can not be cracked? Let's take a look at their differences.

The difference between RSA and DH key exchange algorithms

The role of the Key exchange algorithm: (under the premise of identity authentication) to circumvent the "peeping" risk. Colloquially, even if there is xxx peeping your network transmission with the server, the client can still use the "key negotiation mechanism" and the server side (server) to negotiate a key to encrypt the application layer data (also known as "Session key").

1,rsa Key exchange algorithm
RSA is a "non-symmetric" encryption algorithm. In the background of this 1th series, we have talked about the characteristics of this algorithm--encryption and decryption using "different" keys. and "Asymmetric encryption algorithm" can be used to do "encryption/decryption", can also be used to do "digital signature."

Handshake process:

Approximate steps for key negotiation:

1. 客户端发送client hello，client随机数以及客户端支持的加密套件等信息到服务端2. 服务端发送server hello，发送服务端端随机数和CA证书等信息给客户端3. 客户端验证该证书的可靠性，并从 CA 证书中取出公钥。客户端生成一个随机密钥 Premaster Secret k，并用这个公钥加密得到 k‘，客户端把 k‘ 发送给服务端4. 服务端收到 k‘ 后用自己的私钥解密得到Premaster Secret k5. 这时，客户端和服务端拥有相同的 client随机数、server随机数和 Premaster Secret，可以各自算出相同的后续所需的用于对称加密的session key。这个session key用于后期传输数据的加密和解密。

How to guard against peeping (sniffing)

×××方式1:×××者虽然可以监视网络流量并拿到公钥，但是【无法】通过公钥推算出私钥（这点由 RSA 算法保证）×××方式2:×××者虽然可以监视网络流量并拿到 k‘，但是×××者没有私钥，【无法解密】 k‘，因此也就无法得到Premaster Secret k

How to prevent tampering (fake identity)

×××方式1:如果×××者在第2步篡改数据，伪造了证书，那么客户端在第3步会发现（这点由证书体系保证）×××方式2:如果×××者在第6步篡改数据，伪造了k‘，那么服务端收到假的k‘之后，解密会失败（这点由 RSA 算法保证），服务端就知道被×××了。

According to the above principle, we know that as long as we get the server private key, we can decrypt the client random number, server random number and premaster Secret, so you can calculate the session key, it can also decrypt the transmitted data.

2,DH Key exchange algorithm
The DH algorithm is also called "Diffie–hellman algorithm". This is the name of the two-bit mathematical bull, who created the algorithm. This algorithm is used to achieve "secure" "key exchange". It can be done-"The communication parties create a key through an unsecured channel without any prior information from each other". This sentence more around the mouth, in layman's words, can be attributed to two advantages:

1. Communication between the two parties beforehand "no" need to have a shared secret.
2. Using this algorithm to negotiate the password, even if the negotiation process by others throughout the whole voyeur (such as "network sniffing"), the peeping also "can't" know what the key negotiated.

However, the DH algorithm itself has drawbacks-it does not support authentication. In other words: Although it can fight "peeping", but cannot resist "tamper", naturally also cannot fight "middleman Xxx/mitm". In order to avoid encountering MITMXXX,DH need to cooperate with other signature algorithms (such as RSA, DSA, ECDSA)--rely on the signature algorithm to help authentication. When DH is used in conjunction with RSA, it is called "Dh-rsa", and the DSA mate is referred to as "DH-DSA" and so on.

Handshake process:

The algorithm model of DH:

DH utilizes a mathematically "irreversible" operation, which is a discrete logarithm. Finally two people get the secret numbers are g^ (AB) mod p, and eavesdroppers only from p,g,a,b Four public information, is unable to get this secret number!
For example, if P=23,g=5,alice selected the secret number a=6, then a=5^6 mod =8,bob The secret number selected is 15, then b=5^15 mod 23 = 19, after swapping A and B, Alice calculated the key s=19^6 mod 23=2, Bob calculates the key s=8^15 mod 23=2
Of course, the actual operation is not possible to take such a small value, such as if you need a 128bit length of the key, then the P value needs to be a prime number of 128bit length, because of the modulo operation, the obtained key will not be greater than p, so the P-value can be the largest number of 128bit in a prime, G can be arbitrarily set a small prime number.

Approximate steps for key negotiation:

1. 客户端发送client hello，client随机数以及客户端支持的加密套件等信息到服务端2. a：服务端发送server hello，发送服务端端随机数和CA证书等信息给客户端。   b: 服务端利用私钥将客户端随机数，服务端随机数，服务端DH参数签名，生成服务端的签名。3. 服务端向客户端发送服务端DH参数以及服务器签名(Server Key Exchange)4. 客户端验证签名的有效性，然后向服务端发送客户端DH参数(Client Key Exchange)。5. 客户端与服务端各自利用服务端DH参数、客户端DH参数生成预主密钥Premaster Secret。然后，客户端和服务端各自再通过预主密钥、客户端随机数、服务端随机数生成session key。这个session key用于后期传输数据的加密和解密。

Through the above understanding of RSA and DH key exchange algorithm, we know that premaster Secret is calculated by the client and the server based on certain algorithms and transmitted data can not be calculated premaster Secret, and Premaster Secret will not be transferred to the client and server side, so it is secure, even if the private key is not able to decrypt the DH key exchange algorithm traffic.

Wireshark using private key to resolve key exchange algorithm for RSA traffic

1,wireshark version is 2.4.2
2, open the prepared Pcap file, we can see that the data is encrypted

3, select the Perferences>protocols->ssl in Wireshark

4. Configure SSL-related Settings

IP Address:ssl IP address on the server side
PORT:SSL ports on the service side
Protocol: The protocol used after the encryption protocol is decrypted. If it is HTTPS, it is the HTTP protocol after decryption.
Key file: Path to the private key
Password: The private key is in PEM format and is generally set to empty.

5. View the decrypted traffic

Analytic problem of Http/https mirror traffic