Android signature mechanism-Detailed description of the signature process

Android signature mechanism-Detailed description of the signature process
I. Preface

After a long time, the hands without writing articles were a little uncomfortable. Today is Christmas, or you have to go to work. A few days ago, a colleague asked me about the signature when I applied for the SDK. The result made me difficult .. I said that signatures in Android will be familiar to everyone, that is, to ensure security and prevent others from modifying your apk. But how much do we actually know? So prepare two articles to introduce the signature mechanism in Android.

Before talking about Android signatures, we need to know a few knowledge points.

1. Data Digest (Data fingerprint), signature file, and certificate file

2. jarsign tool signature and signapk tool Signature

3. Relationship between keystore files and pk8 files and x509.pem files

4. How to manually sign the apk

The four knowledge points described above are the core of today's introduction. Let's take a look at these questions one by one.

 

2. Preparation knowledge

First, let's take a look at the knowledge points of the data digest, signature file, and certificate file.

1. Data Summary

This knowledge point is easy to understand. Baidu Baike is actually an algorithm, that is, an algorithm for a data source to get a summary, also known as data fingerprints and different data sources, data fingerprints must be different, just like people.

The Message Digest Algorithm (Message Digest Algorithm) is an Algorithm that can generate special output formats. Its principle is to extract the original data in some form based on certain calculation rules, the extracted information is called the message digest of the original data.
The famous digest algorithms include RSA's MD5 Algorithm and SHA-1 algorithm and a large number of variants.
The main features of a message digest are:
1) no matter how long the input message is, the length of the calculated message digest is always fixed. For example, the message digest using the MD5 algorithm has 128 bits, and the message digest using the SHA-1 algorithm eventually has 160 bits of output.
2) in general (without considering collision), as long as the input raw data is different, the Digest generated after the Digest will be different, even if the raw data changes slightly, the output message digest is completely different. However, the same input must produce the same output.
3) irreversible, that is, only positive information summarization can be performed, and no original message can be restored from the summary.

2. signature files and certificates

The signature file and certificate appear in pairs, and the two files cannot be separated. We can see from the source code that the names of the two files are the same, but the suffix names are different.

In fact, the concept of digital signature is very simple. We all know that to ensure reliable communication, we must solve two problems: first, we must determine that the source of the message is indeed the one stated by it; second, it is necessary to ensure that the information is not tampered with by a third party during the transfer process. Even if the information is tampered with, it can be discovered.
The so-called digital signature is generated to solve these two problems. It is a specific application of the asymmetric encryption and digital digest technologies mentioned above.
For message senders, a pair of public/private key pairs must be generated to send the public key to the message receiver.
If a message sender wants to send a message to the Message Receiver one day, in addition to the original message, the message must be added with another message. The message is generated in the following two steps:
1) extract the abstract of the original message to be sent;
2) encrypt the extracted information Abstract With your own private key.
The message obtained through these two steps is the digital signature of the original information.
For the receiver of the information, the information it receives will contain two parts: the original message content and the additional digital signature. He will verify the authenticity of the message in the following three steps:
1) extract the message digest from the original message. Note that the message digest algorithm used here must be consistent with that used by the sender;
2) use the pre-obtained public key to decrypt the attached digital signature;
3) check whether the two messages obtained in the previous two steps are consistent. If they are consistent, the message is indeed sent by the expected sender, And the content has not been tampered with. On the contrary, if they are inconsistent, it indicates that a problem has been fixed during the transfer, the message is untrusted.
This so-called digital signature technology can effectively solve the problem of reliable communication. If the original message is tampered with during transmission, the abstract of the tampered message is certainly different from the original one in the message receiver. Moreover, since the tampered user does not have the message sender's private key, he cannot forge a digital signature even if he can recalculate the digest of the tampered message.
Therefore, to sum up, a digital signature is a digital string that can only be forged by the sender of information, this digital string is also a valid proof of the authenticity of the information sent by the message sender.
I don't know if you have noticed that the preceding digital signature method requires the Message Receiver to obtain the Correct Public Key in advance. If the public key is tampered with at first, the bad guys will be treated as good people by you, and the messages sent to you by the real message sender will be regarded as invalid by you. In addition, in many cases, there is no information channel to communicate with the public key in advance. So how can we ensure the security and credibility of the public key? This depends on digital certificates.
The so-called digital certificate generally contains the following content:
Issuer)
Certificate Validity period (Validity)
Public Key of the message sender
Certificate owner (Subject)
Algorithm used for Digital Signature
Digital Signature
It can be seen that digital certificates also use digital signature technology. The content to be signed is the public key of the message sender and other information. However, unlike ordinary digital signatures, digital certificates require a certain credibility rather than an ordinary organization. This is like the fact that the signatures on the university diploma are generally high-looking principals. In general, these trusted organizations have installed their root certificates on your device before the device leaves the factory. Therefore, the digital certificate can ensure that the public key in the digital certificate is indeed the owner of the certificate, or the certificate can be used to confirm the identity of the other party. Digital Certificates are mainly used to solve the security issue of public keys.
To sum up, the general process of Digital Signature and signature verification is shown in:

 

3. jarsign and signapk tools

After learning about the three files in the signature, let's continue to look at the two signing tools in Android: jarsign and signapk.

These two tools are easy to confuse at the beginning. Do you think there is any difference between them?

In fact, these two tools are very easy to understand. jarsign is a Java tool that can sign the jar. Signapk is a later signature tool dedicated to the Android Application apk. There is no difference between the two signing algorithms, mainly because the files used for signing are different, this raises the third question.

4. Differences between keystore files and pk8 and x509.pem files

We have learned that both jarsign and signapk can be used for signatures in Android. The difference between them is that the files used for signatures are different.

The jarsign tool uses the keystore file for signing.

When signing the signapk tool, pk8 and x509.pem files are used.

When we use the Eclipse tool to write a program and release the Debug package, the jarsign tool is used for signing by default, and Eclipse has a default signature file:

We can see the default signature of the keystore file, of course we can choose our own designated keystore file.

Here is another knowledge point:

We can see that there is a summary of MD5 and SHA1, which is the data digest of the private key in the keystore file. This information is also the information we need to fill in when applying for many development platform accounts,For example, if you apply for a Baidu map or SDK, you must enter the MD5 or SHA1 information of the application..

5. Manually sign the Apk package

1. Use keytool and jarsigner for signature

Of course, when we release the ASE package in the formal signature, we need to create our own keystore file:

Here we can name the keystore file, and the suffix is irrelevant. After the file is created, the MD5 and SHA1 values are generated. You do not need to record the values. You can run the command to view the MD5 and SHA1 values of the keystore file.

Keytool-list-keystore debug. keystore

Of course, we all know the importance of this keytstore file. To put it bluntly, it is equivalent to your bank card password. You know.

Here we can see that the Eclipse automatic signature is used to generate a keystore file. We can also use the keytool to generate a keystore file. This method is available online, so I will not introduce it too much here. Then we can use jarsign to sign the apk package.

We can manually generate a keystore file:

Keytool-genkeypair-v-keyalg DSA-keysize 1024-sigalg SHA1withDSA-validity 20000-keystore D: \ jiangwei. keystore-alias jiangwei-keypass jiangwei-storepass jiangwei

This command is a bit long and has several important parameters to describe:

-Alias is the alias defined. debug is used here.

-Keyalg indicates the signature algorithm, which is DSA. The algorithm is directly related to the suffix of the signature file in the apk, which will be detailed later.

 

Signing with jarsigner

Jarsigner-verbose-sigalg SHA1withDSA-digestalg SHA1-keystore D: \ jiangwei. keystore-storepass jiangwei D: \ 123.apk jiangwei

In this way, we have successfully signed the apk.

Problems encountered during the signature process:

1. The certificate chain cannot be found

This is because the last parameter alias is an incorrect keystore alias.

2. A Password error is prompted when the keystore file is generated.
This is because debug. ketystore already exists in the current directory. If a debug. keystore is generated, an error is reported.
3. Alias not found
The reason for this problem is that we started the debug alias when using keytool to generate the keystore. This problem has plagued me for a long time and was discovered only after many examples, that is, if the alias of our keystore file is debug, this error will be reported. This should be related to the alias debug in the default debug. keystore signature? The source code of jarsigner is not found, so we can only guess, but note the three questions here to avoid further problems.

 

Note: In Android, multiple keystores are allowed to sign the apk.Here I will not paste the command, and I have created several keystores to sign the apk:

Here I decompress the signed apk and find three signature files and certificates (. SF/. DSA)

Here I can also note that we use the DSA Algorithm for signing. The extension name of the file here is DSA.

The file name is the alias of the keystore.

Ah, here we have understood how we use keytool to generate the keystore and use jarsigner to sign it.

 

2. Use signapk for signature

Next let's take a look at the signapk tool signature:

Java-jar signapk. jar. testkey. x509.pem testkey. pk8 debug.apk debug.sig.apk

Two files are required:. pk8 and. x509.pem.

Pk8 is a private key file.

X509.pem is a file with a public key.

If the signature is used here, it will not be demonstrated. There is no problem here.

However, note the following:The names of the three files in the META-INF folder in the apk after signapk is signed are like this, because signapk does not automatically use aliases to name the file like jarsigner in front of it, here, the CERT name is written to death, but the file name is not affected. Later, we will analyze the Apk verification process in Android and will only search for the file by the suffix.

 

3. What are the differences between the two signature methods?

The problem is that the jarsigner uses the keystore file for signing, while the signapk uses the pk8 and x509.pem files for signing and the apk files are all signed, so the keystore file and pk8, x509.pem are there any relationships between them? The answer is yes. I searched the internet for a while, and they can be converted between them. Here I will not analyze how to convert them. There are many examples on the Internet and there are dedicated tools for conversion:

So here we will figure out the differences and links between the two signature tools.

 

Iii. Analyze the signature process mechanism in Android

Next we will start to look at the signature mechanism and principles in Android from the source code perspective.

Because the source code of jarsigner is not found on the Internet, but the source code of signapk is found, let's take a look at the source code of signapk:

Source code location:Com/android/signapk/sign. java

Through the above signature we can see that Android signature apk, there will be a META-INF folder, there are three files:

MANIFEST. MF

CERT. RSA

CERT. SF

Let's see what these three files are?

1. MANIFEST. MF

Let's take a look at the source code:

 

public static void main(String[] args) {    if (args.length != 4) {        System.err.println("Usage: signapk " +                "publickey.x509[.pem] privatekey.pk8 " +                "input.jar output.jar");        System.exit(2);    }    JarFile inputJar = null;    JarOutputStream outputJar = null;    try {        X509Certificate publicKey = readPublicKey(new File(args[0]));        // Assume the certificate is valid for at least an hour.        long timestamp = publicKey.getNotBefore().getTime() + 3600L * 1000;        PrivateKey privateKey = readPrivateKey(new File(args[1]));        inputJar = new JarFile(new File(args[2]), false);  // Don't verify.        outputJar = new JarOutputStream(new FileOutputStream(args[3]));        outputJar.setLevel(9);        JarEntry je;        // MANIFEST.MF        Manifest manifest = addDigestsToManifest(inputJar);        je = new JarEntry(JarFile.MANIFEST_NAME);        je.setTime(timestamp);        outputJar.putNextEntry(je);        manifest.write(outputJar);        // CERT.SF        Signature signature = Signature.getInstance("SHA1withRSA");        signature.initSign(privateKey);        je = new JarEntry(CERT_SF_NAME);        je.setTime(timestamp);        outputJar.putNextEntry(je);        writeSignatureFile(manifest,                new SignatureOutputStream(outputJar, signature));        // CERT.RSA        je = new JarEntry(CERT_RSA_NAME);        je.setTime(timestamp);        outputJar.putNextEntry(je);        writeSignatureBlock(signature, publicKey, outputJar);        // Everything else        copyFiles(manifest, inputJar, outputJar, timestamp);    } catch (Exception e) {        e.printStackTrace();        System.exit(1);    } finally {        try {            if (inputJar != null) inputJar.close();            if (outputJar != null) outputJar.close();        } catch (IOException e) {            e.printStackTrace();            System.exit(1);        }    }}

In the main function, we can see that four parameters need to be input, and then three things are done:

 

Write MANIFEST. MF

//MANIFEST.MFManifest manifest = addDigestsToManifest(inputJar);je = new JarEntry(JarFile.MANIFEST_NAME);je.setTime(timestamp);outputJar.putNextEntry(je);manifest.write(outputJar);

Go to the method to see:

 

 

/** Add the SHA1 of every file to the manifest, creating it if necessary. */private static Manifest addDigestsToManifest(JarFile jar)        throws IOException, GeneralSecurityException {    Manifest input = jar.getManifest();    Manifest output = new Manifest();    Attributes main = output.getMainAttributes();    if (input != null) {        main.putAll(input.getMainAttributes());    } else {        main.putValue("Manifest-Version", "1.0");        main.putValue("Created-By", "1.0 (Android SignApk)");    }    BASE64Encoder base64 = new BASE64Encoder();    MessageDigest md = MessageDigest.getInstance("SHA1");    byte[] buffer = new byte[4096];    int num;    // We sort the input entries by name, and add them to the    // output manifest in sorted order.  We expect that the output    // map will be deterministic.    TreeMap
 
   byName = new TreeMap
  
   ();    for (Enumeration
   
     e = jar.entries(); e.hasMoreElements(); ) {        JarEntry entry = e.nextElement();        byName.put(entry.getName(), entry);    }    for (JarEntry entry: byName.values()) {        String name = entry.getName();        if (!entry.isDirectory() && !name.equals(JarFile.MANIFEST_NAME) &&            !name.equals(CERT_SF_NAME) && !name.equals(CERT_RSA_NAME) &&            (stripPattern == null ||             !stripPattern.matcher(name).matches())) {            InputStream data = jar.getInputStream(entry);            while ((num = data.read(buffer)) > 0) {                md.update(buffer, 0, num);            }            Attributes attr = null;            if (input != null) attr = input.getAttributes(name);            attr = attr != null ? new Attributes(attr) : new Attributes();            attr.putValue("SHA1-Digest", base64.encode(md.digest()));            output.getEntries().put(name, attr);        }    }    return output;}
   
  
 

The Code logic is still very simple. It mainly refers to the meaning of the Loop:

 

Except for three files (MANIFEST. MF, CERT. RSA, CERT. SF), other files will perform a SHA1 algorithm on the file content, that is, calculate the summary information of the file, and then encode it with Base64, next we will use a tool to make a case to see if it is like this:

First install the tool HashTab

: Http://www.baidu.com/s? Wd = hashtab & rsv_spt = 1 & issp = 1 & f = 8 & rsv_bp = 0 & ie = UTF-8 & tn = baiduhome_pg & bs = hashtable

Then there is another website that is online computing Base64: http://tomeko.net/online_tools/hex_to_base64.php? Lang = en

Let's start our verification work:

Verify the AndroidManifest. xml file. First, find this entry in the MANIFEST. MF file and record the SHA1 value.

After installing HashTab, find the AndroidManifest. xml file, right-click and select Hashtab:

Copy the value of SHA-1: 9c64812de7373b201c292131473636a3167fd73c. Go to the Base64 conversion website above and convert it:

NGSBLec3OyAcKUEBRzY2o2l/1zw =

Exactly the same as the entries in MANIFEST. MF.

From the above analysis, we will know that what is stored in MANIFEST. MF is:

Traverse all entries one by one. If it is a directory, it will be skipped. If it is a file, the Digest of the file will be extracted using the SHA1 (or SHA256) Message Digest algorithm and base64-encoded, written to MANIFEST as the value of the SHA1-Digest property. a block in the MF file. This block has a "Name" attribute whose value is the path of the file in the apk package.

 

2. Next let's take a look at the CERT. SF file content.

The content here is similar to the content of MANIFEST. MF. Let's take a look at the Code:

 

//CERT.SFSignature signature = Signature.getInstance("SHA1withRSA");signature.initSign(privateKey);je = new JarEntry(CERT_SF_NAME);je.setTime(timestamp);outputJar.putNextEntry(je);writeSignatureFile(manifest,new SignatureOutputStream(outputJar, signature));

Go to the writeSignatureFile method:

 

 

/** Write a .SF file with a digest the specified manifest. */private static void writeSignatureFile(Manifest manifest, OutputStream out)        throws IOException, GeneralSecurityException {    Manifest sf = new Manifest();    Attributes main = sf.getMainAttributes();    main.putValue("Signature-Version", "1.0");    main.putValue("Created-By", "1.0 (Android SignApk)");    BASE64Encoder base64 = new BASE64Encoder();    MessageDigest md = MessageDigest.getInstance("SHA1");    PrintStream print = new PrintStream(            new DigestOutputStream(new ByteArrayOutputStream(), md),            true, "UTF-8");    // Digest of the entire manifest    manifest.write(print);    print.flush();    main.putValue("SHA1-Digest-Manifest", base64.encode(md.digest()));    Map
 
   entries = manifest.getEntries();    for (Map.Entry
  
    entry : entries.entrySet()) {        // Digest of the manifest stanza for this entry.        print.print("Name: " + entry.getKey() + "\r\n");        for (Map.Entry
   
     att : entry.getValue().entrySet()) {            print.print(att.getKey() + ": " + att.getValue() + "\r\n");        }        print.print("\r\n");        print.flush();        Attributes sfAttr = new Attributes();        sfAttr.putValue("SHA1-Digest", base64.encode(md.digest()));        sf.getEntries().put(entry.getKey(), sfAttr);    }    sf.write(out);}
   
  
 

 

First, we can see that we need to make a SHA1 for the entire content of the previous MANIFEST. MF file into the SHA1-Digest-Manifest field:

Let's take a look at the manifest variable that has just been written into the MANIFEST. MF file.

Here we can verify:

Then convert

See it. It's the same as the value in the file.

Let's continue to look at the Code. There is a loop:

 

Map
 
   entries = manifest.getEntries();for (Map.Entry
  
    entry : entries.entrySet()) {    // Digest of the manifest stanza for this entry.    print.print("Name: " + entry.getKey() + "\r\n");    for (Map.Entry
   
     att : entry.getValue().entrySet()) {        print.print(att.getKey() + ": " + att.getValue() + "\r\n");    }    print.print("\r\n");    print.flush();    Attributes sfAttr = new Attributes();    sfAttr.putValue("SHA1-Digest", base64.encode(md.digest()));    sf.getEntries().put(entry.getKey(), sfAttr);}sf.write(out);
   
  
 

Here we still use the mainfest variable just passed in to traverse the content of his entries, and then perform the SHA algorithm calculation in Base64:

 

In fact, the SHA is performed for each entry in the MANIFEST. MF file. Save the SHA and use an example to verify the SHA:

Taking AndroidManifest. xml as an example, we copy the entries in the MANIFEST. MF file and save them to the txt file:

Note that we need to add two line breaks after saving them. We can see the logic in the Code:

Then we calculate the SHA value of the txt document:

See it. The calculated value here is the same.

Here we will know what the CERT. SF file has done:

1. Calculate the overall SHA1 value of the MANIFEST. MF file, and then BASE64 encoded, record it under the "SHA1-Digest-Manifest" attribute value of the CERT. SF Primary attribute block (on file head)

2. Calculate the SHA1 of each block in the MANIFEST. MF file one by one, and after BASE64 encoding, record it in the same name block in CERT. SF, the attribute name is "SHA1-Digest

 

3. Finally, let's take a look at the CERT. RSA file.

Here we can see binary files. Because RSA files are encrypted, we need to use the openssl command to view their content.

Openssl pkcs7-inform DER-in CERT. RSA-noout-print_certs-text

For more information, see the following figure:

Let's take a look at the Code:

 

/** Write a .RSA file with a digital signature. */private static void writeSignatureBlock(        Signature signature, X509Certificate publicKey, OutputStream out)        throws IOException, GeneralSecurityException {    SignerInfo signerInfo = new SignerInfo(            new X500Name(publicKey.getIssuerX500Principal().getName()),            publicKey.getSerialNumber(),            AlgorithmId.get("SHA1"),            AlgorithmId.get("RSA"),            signature.sign());    PKCS7 pkcs7 = new PKCS7(            new AlgorithmId[] { AlgorithmId.get("SHA1") },            new ContentInfo(ContentInfo.DATA_OID, null),            new X509Certificate[] { publicKey },            new SignerInfo[] { signerInfo });    pkcs7.encodeSignedData(out);}

We can see that the previously generated CERT. SF file is calculated with the private key, and then the signature and the digital certificate containing the public key information are written into CERT. RSA for saving. CERT. RSA is a file in PKCS7 format. 4. Why do we need to sign the signature?

The details of the three files after the apk signature are described above. The following is a summary of why Android uses this method to encrypt the signature, is this encryption the safest? Next we will analyze what will happen if the apk file is tampered.

First, if you change any file in the apk package, the changed file digest information and MANIFEST are used during apk installation verification. the verification information of MF is different, so verification fails and the program cannot be successfully installed.
Second, if you calculate a new digest value for the modified file, and then change the MANIFEST. the attribute value corresponding to the MF file must be CERT. the Digest value calculated in the SF file is different, so verification fails.
Finally, if you are not dead, continue to calculate MANIFEST. the Digest value of MF, And the CERT is changed accordingly. the value in SF, so the digital signature value must be CERT. the record in the RSA file is different, but it still fails.
Can we continue to forge digital signatures? No, because there is no private key corresponding to the digital certificate.
Therefore, if you want to re-package the application to be installed on the Android device, you must re-sign it.

From the above analysis, we can conclude that if you modify any content in the Apk, you must re-sign it. Otherwise, the installation will fail. Of course, this will not be analyzed here, the next article will focus on analyzing why the installation fails.

V. Knowledge Point sorting

1. Data fingerprint, signature file, certificate file meaning

1. Data fingerprint is a SHA/MD5 Algorithm for a data source. This value is unique.

2. Signature File technology: Data fingerprint + RSA Algorithm

3. The certificate file contains public key information and other information.

4. After the Android signature, SF is the signature file, and RSA is the certificate file. We can use openssl to view the Certificate Information and public key information in the RSA file.

2. We have learned that there are two signature methods in Android: jarsigner and signapk:

1. When jarsigner signs, the keystore file is required, while the signapk signs pk8 and x509.pem files.

2. the SF and RSA file names signed by jarsigner are by default the keystore alias, while the file names after signapk signature are fixed: CERT

3. In Eclipse, when we run the Debug program, the jarsigner signature is used by default, and the default debug. keystore signature file is used.

4. keystore files and pk8, x509.pem files can be converted to each other

Vi. Thinking

After analyzing the signature technology, we accidentally found a problem: CERT. SF, MANIFEST. MF, the name fields in the two files are both the resource names in the apk, so there is a problem. If the Resource name is long and there are many resources in the apk, these two files will be very large. Can we optimize them here? Later I will continue to explain how to reduce the size of the apk. Here I will first raise this question.