HTTPS practice for large Web sites (i)---HTTPS protocol and principles

This is a creation in Article, where the information may have evolved or changed.

1 Preface

Baidu has recently launched an all-station HTTPS security search, the default will be HTTP requests to jump to HTTPS. This article focuses on the HTTPS protocol, and briefly describes the significance of deploying full-site HTTPS.
This article was first published in the Baidu Operations and maintenance Department official blog

2 HTTPS protocol overview

HTTPS can be thought of as HTTP + TLS. The HTTP protocol is familiar, and most Web applications and websites are now transmitted using the HTTP protocol.
TLS is the Transport Layer encryption protocol, its predecessor is the SSL protocol, was first released by Netscape Company in 1995, 1999 after IETF discussion and specification, renamed TLS. If not specifically stated, SSL and TLS are all the same protocol.
The location of HTTP and TLS at the protocol layer and the composition of the TLS protocol are as follows:

There are five main parts of the TLS protocol: Application Data layer protocol, handshake Protocol, alarm protocol, encrypted Message Acknowledgement protocol, heartbeat protocol.
The TLS protocol itself is transmitted by the record Protocol, with the format of the record protocol as shown at the very right.

The commonly used HTTP protocol is HTTP1.1, the common version of the TLS protocol is as follows: TLS1.2, TLS1.1, TLS1.0 and SSL3.0. Among them, SSL3.0 because the poodle attack has been proven unsafe, but the statistics found that there are still less than 1% of browsers use SSL3.0. TLS1.0 also has some security vulnerabilities, such as RC4 and beast attacks.
TLS1.2 and TLS1.1 temporarily have no known security vulnerabilities, more secure, while there is a large number of expansion speed and performance, recommended for everyone to use.
One thing to be concerned about is that TLS1.3 will be a very significant reform of the TLS protocol. Both security and user access speed will be a qualitative improvement. However, there is no definitive release time.
At the same time HTTP2 has been formally finalized, the Protocol evolved from the Spdy protocol HTTP1.1 is a very significant change, can significantly improve the efficiency of application layer data transfer.

3 HTTPS Feature Introduction

Baidu uses the HTTPS protocol primarily to protect user privacy and prevent traffic hijacking.
HTTP itself is transmitted in plaintext, without any security processing. For example, users in Baidu search for a keyword, such as "Apple phone", the middle is fully able to look at this information, and may call to harass users. There are also some users complained about the use of Baidu, found the homepage or the results of the page floated a very long and large ads, which is certainly the middle of the page to plug in the advertising content. If hijacking technology is inferior, users can't even access Baidu.
The intermediary mentioned here mainly refers to some network nodes, is the user data in the browser and Baidu Server intermediate transmission must pass through the node. such as WiFi hotspots, routers, firewalls, reverse proxies, cache servers and so on.
Under the HTTP protocol, the intermediary can sniff the user's search content, steal privacy and even tamper with the webpage. However, HTTPS is the bane of these hijacking actions and can be fully and effectively defended.
In general, the HTTPS protocol provides three powerful features to combat the above hijacking behavior:
1. Content encryption. Browser to Baidu server content is encrypted form of transmission, the intermediary can not directly view the original content.
2. Identity authentication. To ensure that users access to the Baidu service, even if the DNS hijacked to a third-party site, will also remind users not to visit Baidu Services, there may be hijacked
3. Data integrity. Prevent content from being impersonated or tampered with by a third party.

How does HTTPS do the above three points? Here's a brief introduction to the principle.

4 HTTPS Principle Introduction

4.1 Content Encryption

Encryption algorithms are generally divided into two types, symmetric and asymmetric encryption. Symmetric encryption (also known as key encryption) means that encryption and decryption use the same key. Asymmetric encryption (also known as public-key encryption) means that encryption and decryption use different keys.

Symmetric content encryption is very strong, generally can not be cracked. But there's a big problem with the inability to safely generate and keep keys. If each session between the client software and the server uses fixed, the same key encryption and decryption, there must be a great security risk. If someone obtains a symmetric key from the client side, the entire content is not secure, and managing a huge amount of client-side keys is a complex matter.
Asymmetric encryption is mainly used for key exchange (also called key negotiation), which can solve this problem well. Each new session of the browser and the server uses an asymmetric key exchange algorithm to negotiate the symmetric key, using these symmetric keys to complete the application data encryption and decryption and validation, the entire session of the key is only generated and saved in memory, and each session of the symmetric key is not the same (unless the session is reused), the intermediary cannot steal.
Asymmetric key exchange is secure, but it is also the "culprit" for HTTPS performance and speed degradation. To know why HTTPS affects speed and why it consumes resources, you must understand the entire process of asymmetric key exchange.
The following emphasis is on the mathematical principle of asymmetric key exchange and its application in the process of TLS handshake.

4.1.1 Asymmetric key exchange

Before the asymmetric key exchange algorithm appears, a big problem with symmetric encryption is that you don't know how to safely generate and store the key. The asymmetric key exchange process is mainly to solve this problem, making symmetric key generation and use more secure.
Key exchange algorithm itself is very complex, the key exchange process involves random number generation, modulo exponential operation, blank completion, encryption, signature and other operations.
The common key exchange algorithm has Rsa,ecdhe,dh,dhe and other algorithms. They are characterized by the following:

• RSA: The algorithm is simple, born in 1977, has a long history, after a lengthy break test, high security. The disadvantage is that it takes a large number of primes (currently 2048-bit) to ensure security intensity and consumes CPU computing resources. RSA is currently the only algorithm that can be used for both key exchange and certificate signing.
• Dh:diffie-hellman key exchange algorithm, the birth time is earlier (1977), but 1999 is not public. The disadvantage is that CPU performance is more consumed.
• ECDHE: The DH algorithm using Elliptic curve (ECC) has the advantage of being able to achieve the same security level as RSA with a smaller prime number (256 bits). The disadvantage is that the algorithm is complex and the history of the key exchange is not long, and it has not been tested for long time security attack.
• ECDH: PFS is not supported, security is low, and false start cannot be implemented.

It is recommended that RSA and ECDH_RSA key exchange algorithms be supported first. The reasons are:
1. Ecdhe supports ECC acceleration for faster computation. More secure with PFS support. Support False start, user access speed faster.
2. At least 20% of the clients currently do not support Ecdhe, we recommend using RSA instead of DH or dhe because the DH series algorithm consumes CPU (equivalent to two RSA calculations).

It is important to note that the Ecdhe key exchange is usually referred to as ECDHE_RSA, using Ecdhe to generate the public private key required by the DH algorithm, and then using the RSA algorithm for signing and then calculating the symmetric key.
Asymmetric encryption is more secure than symmetric encryption, but there are two obvious drawbacks:
1. CPU Compute resource consumption is very large. A full TLS handshake, where the asymmetric decryption computation of the key exchange accounts for more than 90% of the entire handshake process. Symmetric encryption is only equivalent to 0.1% of asymmetric encryption, if the application layer data also uses asymmetric encryption and decryption, the performance overhead is too large to withstand.
2. The asymmetric encryption algorithm has a limit on the length of the encrypted content and cannot exceed the public key length. For example, the current common public key length is 2048 bits, which means that the content to be encrypted cannot exceed 256 bytes.
Therefore, public key encryption can only be used for key exchange or content signature, and is not suitable for the application layer to transmit the content of the encryption and decryption.

Asymmetric key exchange algorithm is the cornerstone of the whole HTTPS security, fully understanding the asymmetric key exchange algorithm is the key to understand the HTTPS protocol and function.
The following is a brief introduction of RSA and Ecdhe in the process of key exchange application.

4.1.1.1 RSA key Negotiation

4.1.1.1.1 RSA Algorithm Introduction

The security of RSA algorithm is based on the non-reversible multiplication or the hard decomposition of large number factor. The derivation and implementation of RSA involves the concept of Euler function and Fermat theorem and modulo inverse element, and interested readers can own Baidu.
RSA algorithm is one of the most important algorithms to rule the world, and from now on, RSA is also the most important algorithm in the HTTPS system, not one.
The calculation steps for RSA are as follows:

1. Randomly select two prime numbers p, q, assuming p =, q = 19. n = p * q = 13 * 19 = 247.
2. ∅ (n) represents the number of Inma with the integer n. If n equals the product of two prime numbers, then ∅ (n) = (p-1) * (q-1) picks a number E, satisfies 1< e <∅ (n) and E with ∅ (n) coprime, assuming e = 17.
3. Calculates the modulo inverse of e about N, ed≡1 mod∅ (n), from e = 17, ∅ (n) =216 can get d = 89
   

In practice, (n,e) consists of a public key pair, (n,d) consisting of a private key pair, where N and D are a large number close to 2^2048. Even now with a high performance CPU, to calculate 〖m≡c〗^d mod (n), it also needs to consume larger computational resources and time.
Public key pairs (n, e) are generally registered in the certificate, anyone can directly view, such as the public key of the Baidu certificate, such as, where the last 6 digits (010001) converted to 10 binary is 65537, that is, the public key pair of E. There are two advantages to E for smaller values:

1. by c= m^e MoD (n), E is smaller, and client CPU computing consumes less resources.
2. Increase the server side of the crack difficulty. E is small, and d in the private key pair is bound to be very large. So the value of D space is very large, increasing the difficulty of cracking.
   Then why (n,e) be open for public key, even everyone can see directly from the certificate, so safe? The analysis is as follows:
   Since Ed≡1 mod∅ (n), knowing E and N, want to ask for the private key D, you must know ∅ (n). ∅ (n) = (p-1) * (q-1), you must calculate p and Q to determine the private key D. But when n is large enough (for example, near 2^2048), even now the fastest CPU is unable to do this factorization, it is impossible to know which number p and q are multiplied by N. So even if you know the public key, the entire encryption and decryption process is very safe.
   

RSA key negotiation during 4.1.1.1.2 handshake

After introducing the principle of RSA, how is the symmetric key required for the final session generated? What's the relationship with RSA?
Take TLS1.2 as an example, omit the handshake message unrelated to the key exchange. The process is as follows:

1. The browser sends a Client_hello that contains a random number random1.
2. Server-side reply Server_hello, containing a random number random2, while replying to certificate, carrying the certificate public key p.
3. After the browser receives the RANDOM2, it can generate Premaster_secrect and master_secrect. Where the Premaster_secret length is 48 bytes, the first 2 bytes are the protocol version number, and the remaining 46 bytes are populated with a random number. The structure is as follows:

Struct {    byte Version[2];    bute random[46];}

The generation algorithm of Master Secrect is summarized as follows:

Master_key = PRF(premaster_secret, “master secrect”, 随机数1+随机数2labellabel + seed)  XOR  P_SHA-1label + seed)

As can be seen from the above, the Premaster_key is assigned to secret, "master key" assigned to the label, the browser and server side of the two random number of seeds can be determined to find a 48-bit long random number.
The master Secrect contains six parts, which are keys for verifying content consistency, keys for symmetric content encryption and decryption, and initialization vectors (for CBC mode), client and server.
At this point, the browser-side key has been negotiated.

• The browser uses the certificate public key p to encrypt the premaster_secrect and send it to the server.

• The server uses the private key to decrypt to get premaster_secrect. Because the server received a random number of 1 before, so the service side based on the same generation algorithm, under the same input parameters, the same master Secrect is obtained.

  The RSA key negotiation handshake process is illustrated below:
  
  As you can see, the key negotiation process requires 2 RTT, which is also an important reason for https slowness. The key role RSA plays is to encrypt and decrypt premaster_secrect. It is impossible for the intermediary to crack the RSA algorithm, it is impossible to know the premaster_secrect, thus guaranteeing the security of the key negotiation process.

  4.1.1.2 Ecdhe Key Negotiation

  Principle of 4.1.1.2.1 DH and ECC algorithm

  Ecdhe algorithm implementation is much more complex, mainly divided into two parts: Diffie-hellman algorithm (DH) and ECC (elliptic curve arithmetic). Their safety is based on the difficulty of discrete logarithm computation.
  Briefly introduce the implementation of the DH algorithm, first introduced two basic concepts:

• Primitive root: If integer A is the primitive root of prime p, then A, a^2, ..., a^ (p-1) are different under mod p.
• Discrete logarithm: For the primitive root A of any integer B and prime p, there is a unique exponent I satisfies

b ≡a^i  mod p (0≤i ≤p-1)

It is called I is the discrete logarithm of B's modulo p with a as the base.

Understanding these two concepts, the DH algorithm is very simple, the example is as follows:
Assuming that the client and server need to negotiate the key, p=2579, the primitive root a = 2.
1. The client chooses the random number Kc = 123 as its own private key, calculates YC = A^KC mod p = 2^123 mod 2579 = 2400, sends the YC as the public key to the server.
2. The server chooses the random number Ks = 293 as the private key, calculates the YS = a^ks mod p = 2^293 mod 2579 = 968, sends the YS as the public key to the client.
3. Client compute shared key: Secrect = ys^ (KC) mod (p) = 968^123 mod (2579) = 434
4. Server compute shared key: Secrect = yc^ (k_s) mod (p) = 2400^293 mod (2579) = 434

The above formula, Y_s,y_c,p, a, are public information, can be viewed by the intermediary, only K_ (S,) K_c as the private key is not public, when the private key is small, through the exhaustive attack can calculate the shared secret key, but when the private key is very large, the exhaustive attack is certainly not feasible.
A big drawback of the DH algorithm is that it needs to provide a large enough private key to ensure security, so it consumes CPU compute resources. ECC Elliptic curve arithmetic can solve this problem well, the 224-bit key length can reach the security intensity of RSA2048 bit.
ECC's curve formula is actually not an ellipse, just the oval curve circumference formula is called elliptic curve encryption arithmetic. ECC involves many concepts of finite field, group and other modern algebra, and does not introduce in detail.
ECC security relies on the fact that:

P = kQ, 已知k, Q求出P相对简单，但是已知P和Q求出k却非常困难。

The upper-style looks very simple, but has the following constraints:
1. Q is a very large prime number, p, K, q are discrete points on the finite field of an elliptic curve.
2. Finite fields define their own addition and multiplication rules, even if the KQ operation is very complex.

ECC is applied to the Diffie-hellman key exchange process as follows:
1. Define a finite field that satisfies the elliptic equation, i.e. select p, a, B to satisfy the following equation: y^2 mod p=(x^3+ax+b) mod p
2. Select the base point g = (x, y), and the Order of G is n. n is the smallest positive integer that satisfies ng = 0.
3. Client Select private key K_ (c) (0 < K_ (c
4. Server selects the private key K_ (s) and generates the public key y_s = K_sxg
5. The client computes the shared key K = k_cxy_s, the server side computes the shared key K_sxy_c, and the result is the same because:

K_c ×  Y_s = K_c ×  (K_(s  )× G) = K_s × ( K_(c )× G) = K_s  ×  Y_c

As described above, as long as the determination of P, a, B can determine the elliptic curve in a finite field, because not all elliptic curves can be used for encryption, so the choice of P, a, B is very fastidious, the direct relationship between the curve security and computational speed.
The elliptic curve parameters on the FIPS-recommended 256-bit prime number domain are defined as follows for OpenSSL:

Prime number p =115792089210356248762697446949407573530086143415290314195533631308867097853951Order n =115792089210356248762697446949407573529996955224135760342422259061068512044369SEED = c49d360886e70493 6A6678e1139D26b7819f7e90 C =7efba1662985Be94Genevacb055c theD4f7e0 ce8d84a9 c5114abcaf3177680104fThe coefficient of a0d elliptic curve a =0System B of the elliptic curve =5Ac635d8 Aa3a93e7 b3ebbd55769886Bc651D06b0 cc53b0f63bce3c3e -D2604B base point g x =6B17d1f2 e12c4247 F8bce6e5 theA440f277037D812deb33a0f4a13945 d898c296 base point g y =4fE342e2 fe1a7f9b8ee7eb4a7C0f9e162bce33576b315ececbb64068Panax NotoginsengBf51f5

Ecdhe key negotiation during 4.1.1.2.2 handshake

This paper briefly introduces the mathematical principle of ECC and DH algorithm, and we look at the application of Ecdhe in the process of TLS handshake.
It takes more than Rsa,ecdhe to send a server_key_exchange handshake message to complete the key negotiation.
Also take TLS1.2 as an example to briefly describe the process:
1. The browser sends Client_hello, which contains a random number random1, and requires 2 extensions:
A) Elliptic_curves: the curve type and finite field parameters supported by the client. Now the most used is the 256-bit prime number field, which is defined as described in the previous section.
b) Ec_point_formats: Supported curve point format, default is uncompressed.
2, server reply Server_hello, including a random number random2 and ECC extension.
3, the server reply certificate, carried the certificate public key.
4, the server generates ECDH temporary public key, while replying to Server_key_exchange, which contains three important parts:
A) ECC-related parameters.
b) ECDH the temporary public key.
c) The ECC parameter and the signature value generated by the public key for client-side validation.
5, after the browser receives the Server_key_exchange, uses the certificate public key to carry on the signature decryption and the checksum, obtains the server side ECDH temporary public key, generates the shared secret key which the session needs.
At this point, the browser side has completed the key negotiation.
6, the browser generates ECDH temporary public key and Client_key_exchange message, unlike RSA key negotiation, this message does not need to be encrypted.
7, the server processes the Client_key_exchang message and gets the client ECDH temporary public key.
8, the server generates the shared key required for the session.
9, the server-side key negotiation process is complete.
This is illustrated below:

4.1.2 Symmetric content Encryption

The asymmetric key exchange process concludes with the symmetric key that is required for this session. Symmetric encryption is divided into two modes: streaming encryption and packet encryption. Streaming encryption is now commonly used RC4, but RC4 is no longer secure, Microsoft also recommends that the site try not to use RC4 streaming encryption.
A new alternative to RC4 's streaming encryption algorithm, called CHACHA20, is the faster and more secure encryption algorithm Google launches. It has been adopted by Android and Chrome, has been compiled into Google's open source OpenSSL branch-boring SSL, and Nginx 1.7.4 also supports compiling BORINGSSL.
The previously common pattern for packet encryption is AES-CBC, but CBC has been shown to be susceptible to beast and LUCKY13 attacks. The currently recommended packet encryption mode is AES-GCM, but its disadvantage is that it is computationally expensive, with high performance and power consumption, and is not suitable for mobile phones and tablets.

4.2 Identity Verification

Identity authentication mainly involves PKI and digital certificates. Generally speaking, PKI (Public Key Infrastructure) contains the following parts:

• End entity: The terminal body, which can be a terminal hardware or Web site.
• CA: Certificate issuing authority.
• RA: Certificate registration and Auditing authority. such as reviewing the application website or the authenticity of the company.
• CRL Issuer: Responsible for publishing and maintaining the certificate revocation list.

• Repository: Responsible for digital certificates and CRL content storage and distribution.
  Applying for a trusted digital certificate usually has the following process:

  1. The terminal entity generates a public private key and a certificate request.
  2. RA examines the legality of the entity. This step is not required if you are an individual or a small website.
  3. The CA issues a certificate and sends it to the requester.
  4. The certificate is updated to Repository, the terminal subsequently updates the certificate from repository, queries the certificate status, and so on.

  The current certificate used by Baidu is x509v3 format, consisting of the following three parts:

  1. Tbscertificate (to be signed certificate the content of the certificate to be signed), this part contains 10 elements, namely the version number, serial number, Signature algorithm identification, issuer name, validity period, certificate principal name, certificate principal public key information, Publisher Unique identifier, subject unique identifier, extension, etc.
  2. Signaturealgorithm, which identifies the signature algorithm, which specifies the algorithm to sign the tbscertificate.
  3. Signaturvalue (signature value), using Signaturealgorithm to calculate the signature value for Tbscertificate. The
     Digital certificate has two functions:
  4. identity Authorization. Make sure that the Web site that your browser accesses is a trusted, CA-verified site.

  5. Distributes the public key. Each digital certificate contains the public key generated by the registrant. The SSL handshake is transmitted to the client through the certificate message. For example, the RSA certificate public key encryption and Ecdhe signature mentioned earlier are all used in this public key.

     The requester gets the CA's certificate and deploys it on the Web server side, and after the browser initiates the handshake to receive the certificate, how to confirm that the certificate is issued by the CA? How to avoid a third party to forge this certificate? The
     answer is digital signature. The digital signature is the security label for the certificate, and the process of making and verifying the most widely used Sha-rsa digital signature is as follows:

  6. Signing the digital signature. The first is to use a hash function to treat the signature content as a secure hash, generate a message digest, and then encrypt the message digest using the CA's own private key.

  7. The checksum of the digital signature. Use the CA's public key to decrypt the signature and then use the same signature function to sign the content of the signing certificate and compare it with the signature content in the server-side digital signature, if the same is considered successful.
     

Here are a few things to note:
1. The key pair used for digital signature issuance and verification is the CA's own public-private key, which is not related to the public key submitted by the certificate requester.
2. The process of signing a digital signature is just the opposite of the public key encryption process, which is to encrypt the private key and decrypt the public key.
3. Now the large CA will have a certificate chain, the benefits of the certificate chain is security, keep the root CA's private key offline use. The second benefit is ease of deployment and revocation, that is, if there is a problem with the certificate, simply revoke the appropriate level of the certificate and the root certificate remains secure.
4. The root CA certificate is self-signed, which completes the production and validation of the signature with its own public and private keys. The certificate signature on the certificate chain is signed and validated using the key pair of the previous certificate.
5. How do I get the key pair for a root CA and a multi-level CA? Are they credible? Of course, because these vendors work with browsers and operating systems, their public keys are installed by default in the browser or operating system environment. Firefox, for example, maintains a list of trusted CAs, and chrome and IE use a list of the operating system's CAs.

4.3 Data integrity

This part of the content is better understood, similar to the usual MD5 signature, but the security requirements are much higher. OpenSSL now uses two kinds of integrity check algorithms: MD5 or SHA. Since MD5 is more likely to conflict in practice, try not to use MD5 to verify content consistency. SHA also cannot use SHA0 and SHA1, and Professor Xiao of Shandong University in China announced in 2005 that the SHA-1 full version of the algorithm was cracked.
Both Microsoft and Google have announced that they will no longer support SHA1 signing certificates after 16 and 17.

5 HTTPS usage costs

The only problem with HTTPS at the moment is that it has not been applied on a large scale and has received less attention and research. As for the cost of use and the extra cost, don't worry too much at all.
Generally speaking, you may be very concerned about the following issues before using https:
1. Certificate fees and maintenance updates. We feel that the application of the certificate is very troublesome, the certificate is also very expensive, but the certificate is not expensive, cheap dozens of yuan a year, up to hundreds of. And now there are free certification authorities, such as the famous Mozilla-sponsored free Certificate project: Let's Encrypt (https://letsencrypt.org/) supports free certificate installation and Automatic Updates. The project will be formally used in the middle of this year.
The cost of digital certificates is actually not high, for small and medium-sized websites can use cheap or even free digital Certificate Services (there may be security risks), like the famous VeriSign Company's certificate generally thousands of to tens of thousands of blocks a year. Of course, if the company's demand for certificates is relatively large, high customization requirements, you can establish their own CA site, such as Google, can be free to issue Google-related certificates.
2. HTTPS reduces user access speed. HTTPS has a somewhat reduced speed, but the impact of HTTPS on speed is fully acceptable as long as it is properly optimized and deployed. In many scenarios, the HTTPS speed is exactly the same as HTTP, if you use SPDY,HTTPS even faster than HTTP.
We now use Baidu HTTPS security search, have you feel slow?
3. HTTPS consumes CPU resources and requires a large number of machines to be added. The asymmetric key exchange is described earlier, which is a major drain on CPU computing resources, and in addition, symmetric plus decryption requires CPU computing.
Similarly, as long as reasonable optimization, HTTPS machine costs will not increase significantly. For small and medium-sized websites, there is no need to add machines to meet performance requirements.

6 PostScript

Many foreign large internet companies have enabled the full-site HTTPS, which is the future trend of the Internet. The large domestic internet does not deploy HTTPS all the time, but HTTPS is enabled on some sub-pages/sub-requests involving accounts or transactions. Baidu search for the first full-site deployment of HTTPS, the domestic Internet, the full-site HTTPS process will have a huge role in promoting.
At present the internet on the https on the Chinese material is relatively few, this article emphatically introduced the HTTPS protocol to involve the important knowledge point and usually not too easy to understand the blind zone, hoping to understand the HTTPS protocol to everyone to help. Baidu HTTPS performance optimization involves a lot of content, from the front page, back-end architecture, protocol features, encryption algorithms, traffic scheduling, architecture and operation, security and anti-attack and so do a lot of work. The articles in this series will be described in turn.