HTTPS theory Foundation and its best practices in Android

We know that HTTP requests are transmitted in plaintext, so-called plaintext refers to unencrypted information, which can be very dangerous if the HTTP request is intercepted by a hacker and contains sensitive data such as a bank card password. In order to solve this problem, Netscape company developed the HTTPS protocol, HTTPS can encrypt the transmission of data, that is, the transmission is ciphertext, even if the hacker in the transmission process to intercept the data can not be deciphered, which ensures the security of network communications.

Fundamentals of Cryptography

Before we formally explain the HTTPS protocol, we first need to know some knowledge of cryptography.

PlainText: plaintext refers to raw data that has not been encrypted.

Ciphertext: The plaintext is encrypted by some encryption algorithm, thus ensuring the security of the original data. Ciphertext can also be decrypted to get the original plaintext.

Key: A key is a parameter that is entered in an algorithm that converts plaintext into ciphertext or converts cipher text to plaintext. The key is divided into symmetric key and asymmetric key, which are applied to symmetric and asymmetric encryption respectively.

Symmetric encryption: Symmetric encryption is also called private key encryption, that is, the sender and receiver of the information use the same key to encrypt and decrypt data. The characteristic of symmetric encryption is that the algorithm exposes, encrypts and decrypts fast, and is suitable for encrypting large data volume, and the common symmetric encryption algorithms are DES, 3DES, Tdea, Blowfish, RC5 and idea.
The encryption process is as follows: PlainText + encryption algorithm + private key = Cipher
The decryption process is as follows: Ciphertext + decryption algorithm + private key = Clear text
The key used in symmetric encryption is called the private key, which represents the private key of the individual, i.e. the key cannot be compromised.
The private key in its encryption process is the same key as the private key used in the decryption process, which is why encryption is called "symmetry". Because the symmetric encryption algorithm is public, so once the private key is leaked, then ciphertext is easily cracked, so the disadvantage of symmetric encryption is the key security management difficulties.

Asymmetric encryption: Asymmetric encryption is also known as public-key cryptography. Asymmetric encryption is more secure than symmetric encryption. Symmetric encrypted communications both use the same key, and if one party's key is compromised, the entire communication will be cracked. Asymmetric encryption uses a pair of keys, the public and private keys, and they appear in pairs. The private key is saved by itself and cannot be leaked. A public key refers to a common key that can be obtained by anyone. Encrypted with one of the public or private keys and decrypted with the other.
Ciphertext encrypted by the public key can only be decrypted by the private key, the process is as follows:
PlainText + encryption algorithm + Public key = cipher, ciphertext + decryption algorithm + private key = Clear text
Ciphertext encrypted by the private key can only be decrypted by the public key, the process is as follows:
PlainText + encryption algorithm + private key = cipher, ciphertext + decryption algorithm + Public key = Clear text
Because encryption and decryption use two different keys, this is why asymmetric encryption is "asymmetric".
The disadvantage of asymmetric encryption is that encryption and decryption take a long time and are slow, and are only suitable for encrypting small amounts of data.
The main algorithms used in asymmetric encryption are RSA, Elgamal, Rabin, D-h, ECC (elliptic curve encryption algorithm) and so on.

HTTPS Communication process

HTTPS protocol = HTTP protocol + SSL/TLS protocol, in the process of HTTPS data transmission, it is necessary to use SSL/TLS to encrypt and decrypt data, it is necessary to use HTTP to transmit the encrypted data, thus we can see that HTTPS is by HTTP and ssl/ TLS to work together.

The full name of SSL is secure Sockets layer, an all-in-one layer protocol, which is a security protocol that provides security and data integrity for network communications. The SSL protocol was invented by Netscape in 1994, and later all browsers support SSL, the latest version is 3.0

The full name of TLS is Transport layer Security, the secure Transport layer protocol, and the latest version of TLS (Transport layer Security, Transport Layer Secure Protocol) is the IETF (Internet Engineering Task Force,internet Engineering Task Force) developed a new protocol that is built on the SSL 3.0 protocol specification and is a subsequent version of SSL 3.0. There is a significant difference between TLS and SSL3.0, mainly because they support different cryptographic algorithms, so TLS and SSL3.0 cannot interoperate. Although TLS differs from SSL3.0 in cryptographic algorithms, we can see SSL and TLS as the same protocol in our understanding of HTTPS.

HTTPS is designed to take security and efficiency into account, while using symmetric and asymmetric encryption. The data is transmitted by symmetric encryption, the symmetric encryption process needs a client's key, in order to ensure that the key can be safely transmitted to the server side, using asymmetric encryption to encrypt the transmission of the key, in general, symmetric encryption of the data, symmetric encryption to use the key through asymmetric encryption transmission.

Blog from Limboy

HTTPS involves three keys in the process of transmission:

• Server-side public and private keys for asymmetric encryption

• A random key generated by the client for symmetric encryption

An HTTPS request actually contains two HTTP transmissions, which can be broken down into 8 steps.

1. The client initiates an HTTPS request to the server, connecting to port 443 of the server.

2. The server side has a key pair, that is, the public key and private key, is used for asymmetric encryption, the server side holds the private key, cannot be disclosed, the public key can be sent to anyone.

3. The server sends its own public key to the client.

4. After the client receives the server-side public key, it checks the public key, verifies its legitimacy, and if it finds a problem with the public key, the HTTPS transport cannot continue. Strictly speaking, this should be to verify the legitimacy of the digital certificate sent by the server, and explain how the client verifies the legitimacy of the digital certificate. If the public key is qualified, then the client generates a random value, which is the key used for symmetric encryption, which we refer to as client key, which is an easy way to differentiate between conceptually and server-side keys. The client key is then asymmetric encrypted with the server's public key, so that the client key becomes ciphertext, and at this point, the first HTTP request in HTTPS ends.

5. The client initiates a second HTTP request in HTTPS and sends the encrypted client key to the server.

6. After the server receives the ciphertext from the client, it uses its own private key to decrypt it, and the decrypted plaintext is the client key, then the data is symmetric encrypted with the client key, so the data becomes ciphertext.

7. The server then sends the encrypted ciphertext to the client.

8. The client receives the ciphertext sent by the server, and uses the client key to decrypt it, and obtains the data sent by the server. So the second HTTP request in HTTPS ends and the entire HTTPS transfer is complete.

Digital certificates

By observing the transmission process of HTTPS, we know that when the server receives the request from the client, it sends the server's own public key to the client, but the hacker may tamper with the public key and change it into a hacker's own, so there is a problem. How does the client trust that the public key is the public key of the server that it wants to access rather than a hacker? A digital certificate is required.

Before you speak a digital certificate, let's say a small example. Suppose there are two people in a town. A and B,a are rich, B want to borrow money from a, but A and B are not familiar, afraid B after borrowing money not yet. At this time B found the mayor, the mayor to B as a guarantee, told a said: "B good character, will not owe money not to pay back, you can rest assured to lend him." "A after listening to this, my heart thought:" The mayor is the most respected in the town, he said B No problem then it is OK, I am relieved. " So a believed in B and lent the money to B.

Similarly, in order for the client to rely on the public key, the public key to find a guarantor, and the identity of the guarantor must be respected, otherwise unconvincing. This guarantor is the Certificate Certification center (Certificate Authority), referred to as CA. In other words, the CA is a special public key authentication, guarantee, that is, a special guarantee for the public key guarantee company. Worldwide well-known CAs are more than 100, these CAs are recognized globally, such as VeriSign, GlobalSign, etc., the domestic well-known CA has wosign.

How does the CA guarantee the public key certification? The CA itself also has a pair of public and private keys, the CA will use the CA's own private key to authenticate the public key for asymmetric encryption, where the public key to be authenticated is the equivalent of clear text, after encryption, the resulting ciphertext plus the certificate expiration time, issued to, the issuer and other information, the formation of a digital certificate.

Regardless of the platform, there are more than 100 globally recognized CAS built into the operating system of the device, saying that the public key of these well-known CAS is stored in the device. When the client receives a digital certificate from the server, it verifies the following:

1. First the client attempts to decrypt the digital certificate with the public key of the CA built into the device, and if the public key of all the built-in CAS cannot decrypt the digital certificate, the digital certificate is not issued by a globally known CA, so that the client cannot trust the server's digital certificate.

2. If a CA's public key can successfully decrypt the digital certificate, the digital certificate is issued by the CA's private key, because ciphertext encrypted by the private key can only be decrypted by its paired public key.

3. In addition to this, you need to check that the domain of the server that the client is currently accessing is the same as the "issued to" entry in the digital certificate, and check that the digital certificate is out of date.

It is easy to get the public key of the server directly from the browser, and the browser operations are similar. Baidu has now implemented the full site of HTTPS, we take Baidu as an example how to get its public key from Chrome.

1. Open Baidu homepage with Chrome, on the left of HTTPS we will find a green lock.
   

2. Click on the lock, a popup panel appears, click on the "Details" in the panel.
   

3. This will open Chrome's developer Tool and automatically switch to the Security page.
   

4. Click on the "View Ceertificate" button to view the certificate of the website as follows:
   

   In the "General" panel, we can see that the certificate is and Symantec issued to Baidu.com, which is valid from September 2015 to 17-2016 September 1.

5. Switch to the details panel to view some details about the certificate, such as the algorithm for the digital signature used by the certificate, as shown in:
   

   There is a "Copy to File" button, click on the button to export Baidu Digital certificate into a file, so we can save to our own machine, the interface is as follows:

   We export it to a certificate in the form of a. cer as the file extension and finally to the local machine as shown below:
   

6. Switch to the certificate path panel to view the certificate chain for the certificates.
   

   Let's explain what a certificate chain is. As we mentioned before, VeriSign is a global well-known CA, but generally, the CA does not use its own private key to directly sign a website's digital certificate, the general CA will first issue a certificate, and then use this certificate to issue Baidu and other digital certificates. In this example, VeriSign has issued a Symantec certificate, Then Symantec issued the Baiduc.om,verisign at the top, similar to the root node, so called Root Ca,symatec is located in the middle, called the intermediate CA, of course, there may be multiple intermediate CAs, from the root CA to the intermediate CA, and then to the final website certificate, so top-down form a card Book chain. If you want to view a certificate in the certificate chain, you only need to select it, such as by selecting Symantec and clicking the "View Certificate" button below, a different dialog will pop up, where you can view your Symantec digital certificate, or you can export it as a certificate file on your hard disk.

Access HTTPS in Android

In Android we also often send HTTPS requests, this time need to use httpsurlconnection this class, Httpsurlconnection is inherited from HttpURLConnection, Its usage is the same as httpurlconnection, for example, we want to use HTTPS to visit the homepage of Baidu, the code is as follows:

URL url = new URL("https://www.baidu.com");HttpsURLConnection conn = (HttpsURLConnection)url.openConnection();InputStream is = conn.getInputStream();...

We can parse the return result of the server by the InputStream we get.

If there is a problem with the digital certificate of the corresponding server, then the client cannot trust the certificate and cannot establish the HTTPS link, we use the domestic 12306.cn website as an example to illustrate.

12306.cn User login is required to access HTTPS, if the browser first opened the page https://kyfw.12306.cn/otn/regist/init, then the browser will either display a certificate warning message, or simply do not display the page, There is a problem with the digital certificate for 12306.cn.

The certificate chain is as follows:

As you can see, the 12306.cn certificate is issued by Srca, which means that Srca is the root CA on the certificate chain.

But what is Srca? Never heard of it!

After we select Srca, click the "View Certificate" button and the Srca certificate is as follows:

That is to say the full name of the SRCA is sinorail certification authority, in the search for the title in Baidu, you can find a website called the Chinese Railway Information Engineering group, http://www.sinorail.com/ productinfos.aspx?id=185, there is such a description:

That is to say, Srca is a so-called CA that the Ministry of Railways has signed for its website, etc., but it does not have credibility because it is not a globally known CA, so the client simply does not recognize it.

When we access Https://kyfw.12306.cn/otn/regist/init directly in Android, we get the following exception:
Java.security.cert.CertPathValidatorException:Trust anchor for certification path is not found.

This is because the list of trusted CAs built into the Android client does not contain the so-called Srca, so there is an exception that the 12306.cn certificate is not trusted by the client.

In order to resolve the problem that the client does not trust the server digital certificate, the majority of the solution on the network is to let the client do not check the certificate, This is a very large security vulnerability approach, as follows:

First define your own Ssltrustallmanager, which inherits from X509trustmanager, specifically, creates a sslcontext with Ssltrustallmanager, The Sslcontext is then used to generate a sslsocketfactory, which is then used by calling Httpurlconnectoin's Setsslsocketfactory method to a specific connection object. Since Ssltrustallmanager does not implement any of the three core methods, it does not make any review of the certificate. This allows an HTTPS connection to be established regardless of the server's certificate, because the client directly ignores the step of validating the server certificate.

This makes it possible to establish an HTTPS connection, but there is a large security vulnerability. Because hackers are likely to intercept our HTTPS requests, Then the hacker used his fake certificate to impersonate 12306.cn digital certificate, because the client does not verify the certificate, so that the client and the hacker's server to establish a connection, which will cause the client to send their 12306 account and password to the hacker, so the client does not do any verification of the certificate of the practice has a great security vulnerability.

The solution to this problem is to have the Android client trust the 12306 certificate instead of making any checks on the certificate.

We get 12306 certificate 12306.cer from the above mentioned method, put it into the assets directory, that is, the 12306.cer package into our app, and then use the following code to access the 12306 HTTPS site.

 class downloadthread extends Thread{@Override Public voidRun () {Httpsurlconnection conn =NULL; InputStream is =NULL;Try{URL url =NewURL ("Https://kyfw.12306.cn/otn/regist/init"); conn = (httpsurlconnection) url.openconnection ();//Create certificatefactory in the format of theCertificatefactory CF = Certificatefactory.getinstance (" the");//Get the stream of certificates from assertsInputStream Cerinputstream = Getassets (). Open ("12306.cer");//srca.cer                //ca is java.security.cert.Certificate, not java.security.Certificate,                //Not javax.security.cert.CertificateCertificate CA;Try{//Certificate factory generates certificates based on the flow of certificate files certificateCA = Cf.generatecertificate (cerinputstream); System.out.println ("Ca="+ ((x509certificate) CA). Getsubjectdn ()); }finally{Cerinputstream.close (); }//Create a default type of KeyStore that stores the certificates we trustString Keystoretype = Keystore.getdefaulttype ();                KeyStore KeyStore = keystore.getinstance (Keystoretype); Keystore.load (NULL,NULL);//Put the certificate CA as a trusted certificate into KeyStoreKeystore.setcertificateentry ("Ca", CA);//trustmanagerfactory is used to generate TrustManager, we create a default type of TrustmanagerfactoryString tmfalgorithm = Trustmanagerfactory.getdefaultalgorithm (); Trustmanagerfactory TMF = trustmanagerfactory.getinstance (tmfalgorithm);//Use our previous KeyStore instance to initialize the trustmanagerfactory so that TMF trusts the certificate in KeyStoreTmf.init (KeyStore);//through TMF get TrustManager array, TrustManager will also trust the certificate in KeyStoretrustmanager[] trustmanagers = Tmf.gettrustmanagers ();//Create a Sslcontext object of type TLS, that's uses our TrustManagerSslcontext Sslcontext = sslcontext.getinstance ("TLS");//Use the trustmanagers above to initialize the Sslcontext so that Sslcontext will trust the certificate in KeyStoreSslcontext.init (NULL, Trustmanagers,NULL);//Get Sslsocketfactory object through SslcontextSslsocketfactory sslsocketfactory = Sslcontext.getsocketfactory ();//Apply Sslsocketfactory to Httpsurlconnection object by Setsslsocketfactory method                //So the Conn object will trust the certificate object we obtained beforeConn.setsslsocketfactory (sslsocketfactory); is = Conn.getinputstream ();//Convert the resulting inputstream to a string                FinalStringStr= Getstringbyinputstream (IS); Handler.post (NewRunnable () {@Override Public voidRun () {Textview.settext (Str); Btn.setenabled (true);            }                }); }Catch(Exception e) {E.printstacktrace ();FinalString errmessage = E.getmessage (); Handler.post (NewRunnable () {@Override Public voidRun () {btn.setenabled (true); Toast.maketext (mainactivity. This, Errmessage, Toast.length_long). Show ();            }                }); }finally{if(Conn! =NULL) {conn.disconnect (); }            }        }

The above comments are written in more detail, here we are still on the above code to explain.

1. First get the file stream of the 12306.cer certificate from the asserts directory, and then use the Certificatefactory Generatecertificate method to flow the file into a Certificate object certificate, which is the digital certificate of the 12306 Web site.

2. Create a default type of KeyStore instance, KeyStore to store the digital certificate we trust, and put 12306 of the digital certificate into KeyStore.

3. We get an instance of the default Trustmanagerfactory and initialize it with the previous keystore so that trustmanagerfactory instances will also trust the KeyStore 12306.cer certificate. The TrustManager array is obtained through the Gettrustmanagers method of Trustmanagerfactory, and the TrustManager in the array trusts the 12306.cer certificate.

4. Create a Sslcontext instance of the TLS type and initialize the Sslcontext instance with the previous TrustManager array so that the Sslcontext instance trusts the 12306.cer certificate.

5. The Sslsocketfactory object is obtained through Sslcontext, and sslsocketfactory is used for httpsurlconnection objects through the Setsslsocketfactory method. This allows the Conn object to trust the 12306.cer certificate in the KeyStore. As a result, the client trusts 12306 of the certificate to properly establish an HTTPS connection.

The above process is the official Google recommended process, the process is summarized as follows:

Certificate file stream InputStream, Certificate, KeyStore, Trustmanagerfactory, trustmanager[, Sslcontext, Slsocketfactory-Httpsurlconnection

The starting point of the above steps is to obtain the file stream of the certificate file, not necessarily from the assets directory, or by other means, as long as you can get the file stream of the certificate.

The process above is right, but there is a little problem. We put the 12306 Web site itself 12306.cer into the assets directory, and then let us create a httpsurlconnection instance trust 12306.cer. However, the digital certificate has an expiration time, if the 12306 Web site's digital certificate expires, then 12306 will go to srca there to regenerate a digital certificate, the 12306 Web site's public and private keys will be updated, then there is a problem. Our app's assets directory is stored in the old 12306.cer certificate, so that 12306 sites to regenerate the new digital certificate, then the old digital certificate is automatically obsolete, because the old public key in the 12306.cer of our app cannot decrypt the latest private key of the 12306 site (public and private keys can only be paired out Now, the old public key can only decrypt the old private key).

Unfortunately, most of the online solution is to directly trust the 12306.cer of this website's own digital certificate, although this method can temporarily solve the problem of HTTPS, but not a long-term, will be the future of digital certificate updates buried hidden trouble.

So how do you solve this problem?

The best way is not to let our app directly trust 12306.cer, but to let our app directly trust the digital certificate of the issuer of the 12306 digital certificate Srca.

We have previously mentioned the method of 12306 of the Issuer Srca digital certificate export, named Srca.cer, put it into the assets directory, we only need to change the code inside the parameters can be, we will code:

InputStream cerInputStream = getAssets().open("12306.cer");

Fix the

InputStream cerInputStream = getAssets().open("SRCA.cer");

That is, we are reading the file stream of the Srca.cer certificate, which is no longer 12306.cer.

Why is it that the HTTPS connection can be established properly by having the Httpsurlconnection instance trust Srca.cer?

We mentioned the concept of the certificate chain before, assuming that the following certificate chain exists:
C-B, CA-A

A CA is a root CA certificate, such as SRCA.CER,C is the digital certificate for the final Web site, such as 12306.cer,a and B are intermediate certificates, and in theory, c certificates are trusted as long as the client trusts any one of the certificates in the digital certificate chain. such as the client trusts the root certificate CA, because the CA trusts a, so the client also trusts a, because a trusts B, then the client also trusts B, because B trusts C, then the client also trusts C. So in the 12306 example, if you trust Srca.cer, the client trusts the 12306.cer digital certificate of the 12306 website itself.

The reason that the Android client does not trust the server certificate is primarily because the client does not trust the root certificate CA in the certificate chain. 12306 the website's own digital certificate may be regenerated and changed in a few years, but Srca as its issuer, the number of changes will be much less, or for a long time will not change, So our app to trust Srca.cer more stable than directly to trust 12306.cer.

Summarize:

1. The transmission process of HTTPS involves symmetric and asymmetric encryption, and symmetric encryption is the actual data, and asymmetric encryption encrypts the client's key required for symmetric encryption.

2. To ensure that the client is able to confirm that the public key is the public key of the Web site that it wants to access, the concept of a digital certificate is introduced, because the certificate has a first-level signing process, so a certificate chain appears, and the root CA is the top of the certificate chain.

3. The reason that the Android client does not trust the server certificate is primarily because the client does not trust the root certificate CA in the certificate chain, and we should let our app trust the root certificate CA instead of trusting the website's own digital certificate directly, because the digital certificate of the site may change.

Hope that they are helpful to everyone!

Reference:
Limboy's "Illustrated HTTPS"
What is the digital signature of Nanyi? 》
Google's official "Security with HTTPS and SSL"

Related reading:
My Android Blog Finishing summary
Android in HttpURLConnection use detailed

HTTPS theory Foundation and its best practices in Android