AWS S3 V4 Signature Algorithm

This is a creation in Article, where the information may have evolved or changed.

Original, reproduced please specify: Http://www.jianshu.com/p/a6a8c3c2cead

First, the opening statement:

The following thinking direction, is based on the Android side (the same as iOS)

Aws:amazon Web Services (Amazon cloud service)

AWS S3 API Documentation: HTTPS://AWS.AMAZON.COM/CN/DOCUMENTATION/S3/

Minio: (Specific explanation self-Baidu bar) an open-source implementation of the AWS S3 Storage protocol developed in Golang language, with a Web UI interface, can be minio to build a private storage server compatible with the AWS S3 protocol.

Second, demand analysis

Project requirements: Recently the company needs to set up a file server for mobile (Android, iOS) to store images and other files. This file server is built using Minio in the background. Since Minio is based on the AWS S3 Storage protocol, we also need to implement the same protocol to upload on our mobile side. On the mobile side, we identified three possible scenarios (the pros and cons are based only on the extent to which the APK package size is affected):

Scenario One: Implement your own file upload based on AWS S3 Storage Protocol (best solution, last project introduction file minimum----Approximately 100K, does not affect apk size)

Scenario Two: Implementing file uploads using the AWS SDK (intermediate scenario, a slightly larger jar package 1.5M file that needs to be introduced into the project)

Scenario three: Using the Minio SDK for file uploads (worst-in-program, need to introduce 6M jar packages)

Third, the code implementation

Because of the different needs of the project, for everyone to choose, I will implement these three scenarios one by one description.

Programme III:

Demo Download: Https://github.com/Nergal1/minio-demo

1.Android the introduction of the Minio SDK jar package will conflict with the native, and will report some security errors.

When introduced, remove conflicting packages, Com.fasterxml.jackson.core and com.google.code.findbugs:

dependencies {

Compile (' io.minio:minio:3.0.4 ') {

Excludegroup: ' Com.fasterxml.jackson.core '

Excludegroup: ' Com.google.code.findbugs '

}

}

2. Upload the code:

/* Upload pictures

Bucketname: Server-side storage folder, you must first create

ObjectName: Server-side Store file name

InputStream: uploading a file stream

*/

Minioclient.putobject (Bucketname,objectname,inputstream,inputstream.available (), "Application/octet-stream");

Reminder: Don't forget to turn on network permissions

3. Simple source flow chart, the number represents the line number:

Minio Upload Source Simple flowchart

Scenario Two:

Demo Download: Https://github.com/Nergal1/AWS-S3-demo

1. Because the AWS SDK source is open service, then create thread pool to implement asynchronous upload, download function.

To register the service with the manifest file:

<service android:name= "Com.amazonaws.mobileconnectors.s3.transferutility.TransferService"

Android:enabled= "true"/>

Permissions:

Android.permission.INTERNET

Android.permission.ACCESS_NETWORK_STATE

Android.permission.READ_EXTERNAL_STORAGE

Android.permission.WRITE_EXTERNAL_STORAGE

The Android 6.0+ file read and Write permission requires real-time access in the code and is not compatible in the demo. If you have a file problem, please add it yourself.

2. Introduce the JAR package:

Aws-android-sdk-core-2.4.2.jar

Aws-android-sdk-s3-2.4.2.jar

3. Configure endpoint, ACCESSKEY, Secretkey, bucket_name in Constants.java first

Upload code See Uploadactivity.java (download Please check downloadactivity):

Transferutility transferutility= util.gettransferutility (this);

Upload

Transferobserver Observer =transferutility.upload (Constants.bucket_name,file.getname (),

file);

Set up upload monitoring

Observer.settransferlistener (New Uploadlistener ());

Listening class

Private class Uploadlistener implements Transferlistener {

Simply updates the UI list when notified. Upload failed monitoring

@Override

Public Voidonerror (Intid,exception e) {

LOG.E (TAG, "Error during upload:" + id,e);

Updatelist ();

}

Upload Progress Monitoring

@Override

Public voidonprogresschanged (IntID, longbytescurrent, longbytestotal) {

LOG.D (Tag,string.format ("onprogresschanged:%d, total:%d, current:%d",

Id,bytestotal,bytescurrent));

Updatelist ();

}

Upload Status Monitoring

@Override

Public voidonstatechanged (Intid,transferstate newstate) {

LOG.D (TAG, "onstatechanged:" + ID + "," + newstate);//For example: Uploadactivity:onstatechanged:9, Faild failed status

Querying file paths by ID

Transferobserver Transferobserver = Transferutility.gettransferbyid (ID);

LOG.D (TAG, "File Address---" +transferobserver.getabsolutefilepath ());

Updatelist ();

}

}

4. The source simple flowchart, the number represents the line number:

AWS SDK Upload Source Simple flowchart

Programme III:

Demo Download: Https://github.com/Nergal1/nergal-AWS-demo

1. Add Permissions:

Android.permission.INTERNET

Android.permission.ACCESS_NETWORK_STATE

Android.permission.READ_EXTERNAL_STORAGE

Android.permission.WRITE_EXTERNAL_STORAGE

Android 6.0+ file read and Write permissions need to be obtained in real-time code, the demo is compatible.

2. Configure endpoint, ACCESSKEY, Secretkey, Bucket_name, bucket_region in Constants.java

Upload code:

awstransferutility utility = Awstransferutility.getinstance ();

File upload, first set up listening, after uploading files ************* if you need to use other network framework to implement the upload, please implement Basehttpclient, and then utility.sethttpclient

Utility.setuploadlistener (New Mawsuploadlistener ()). Upload (File,constants.bucket_name,file.getname ());

The Listener class is implemented as follows

/**

* @author Zhangchen

* @date 2017/5/24 8:32

* @Description File Upload monitoring method

*/

Class Mawsuploadlistener implements Awsuploadlistener {

@Override

Public voidoncomplite (file file) {

System.out.println ("Oncomplite---" + file.getabsolutepath ());

}

@Override

Public Voidonerror (File file,throwable e) {

System.out.println ("OnError---file--" + file.getabsolutepath ());

System.out.println ("OnError---exception--" + e.tostring ());

}

@Override

Public voidonprogresschanged (file file, Longbytescurrent, longbytestotal) {

System.out.println ("onprogresschanged----" + bytescurrent);

}

}

3. Haha, wordy a lot of, finally to the point, the realization Principle analysis:

In fact, the AWS S3 Storage protocol simply adds some request headers that are negotiated with the server, and the value of the request header is calculated using the AWS S3 V4 Signature Algorithm. Therefore, the upload with the specified request header, as well as the legal signature algorithm, other places with ordinary upload no difference. After the server gets the request header, the signature value is computed according to the valid signature algorithm, then the signature of the client's request is verified, and the server is allowed to upload after the success.

First we need to solve two problems:

1. What are the specified request headers?

First we look at a file upload request:

--------------------------Request Header--------------------------

Put/test/img_20170519_165644_1.jpg http/1.1

content-md5:6pn5zj06z+d5uksg1zxhna==

x-amz-decoded-content-length:2566214

X-amz-content-sha256:streaming-aws4-hmac-sha256-payload

Content-type:image/jpeg

x-amz-date:20170522t024116z

user-agent:aws-sdk-android/2.4.2 linux/3.10.86-g6be8ceb dalvik/2.1.0/0 zh_cn transferservice/2.4.2

aws-sdk-retry:0/0

Accept-encoding:identity

aws-sdk-invocation-id:148858f7-e319-4578-b9f8-1b220f6af380

authorization:aws4-hmac-sha256 credential=1na5k80uu85nmpk4bpew/20170522/us-east-1/s3/aws4_request,signedheaders =content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length,signature= Db4e0abd4290730ed6fd27867d2fa342942372b88f1b5ad49b113ab9c77d6cc9

content-length:2568100

Host:fsst.anbanggroup.com

Connection:keep-alive

--------------------------------------Request Body--------------

20000;chunk-signature=0cfa38451a889b866dbf43907ce3a72ee85f6b176168fb875903e8353366628e

。。。 Binary file Stream ...

-------------------------------------------

A lot of request headers, in fact, according to the S3 API documentation, the authorization request header is necessary

The value of authorization, signedheaders, specifies which request headers are used to calculate the signature values for this request. That is, content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length these request headers calculate the value of signature. Host;x-amz-content-sha256;x-amz-date in signedheaders must exist according to the S3 API documentation.

So why do you want to add content-md5,x-amz-decoded-content-length these two values to this request?

Let's think about the role of signatures:

A. Verifying the request identity

ACCESSKEY, Secretkey is used to verify identity, and these two parameters are necessary to calculate the value of signature in authorization's value.

B. Prevention of tampering

Signedheaders is used to set the request header to prevent tampering, content-md5 is to prevent tampering with the request content (body), x-amz-decoded-content-length to prevent tampering with the request content length.

In the official documentation, Signedheaders must be host;x-amz-content-sha256;x-amz-date to understand, prevent tampering with the request address, request content sha256 value, request timestamp

C. Preventing the request signature from being compromised

Depending on the AWS S3 Storage Protocol, the timestamp (x-amz-date or date request header) is valid for 15 minutes, and a user without permission can tamper with the signed request by intercepting the part that is not included in the signedheaders, so the official recommendation Sign all the request headers and the request body (that is, signedheaders to include as much as possible), and the best use of HTTPS.

Summary: A lot of verbose, the necessary request header has: Authorization, Signedheaders contains (must contain the host;x-amz-content-sha256;x-amz-date), host, X-amz-content-sha256,x-amz-date These request headers are required for the service-side checksum signature.

2. How is the signature algorithm calculated? (i.e. the signature in authorization)

There are two types of signature algorithms:

The first, single-block transmission (the way I use it in my demo)

File memory read two times (when file sha256 is calculated at one time, when file is uploaded)

Single-block Upload request header:

X-amz-content-sha256:32e820d03db121caf97206f8cbcc6202cf25bf59246e8d6b0e8f6e3502d68f66

Or: X-amz-content-sha256:unsigned-payload (this is a single piece of text that does not have to be signed)

A.

Single-block upload signature calculation process

Description of function function:

Lowercase (): string converted to lowercase

Hex (): Base 16 encoding (all lowercase)

Sha256hash (): Calculates the request body body (when uploading a file, only the file in the body, that is, the SHA256 of the computed file) SHA256, and then base64

HMAC-SHA256 (): Computes an HMAC by using the key with SHA256

Java method:

public static Byte[]sumhmac (byte[] key, byte[] data)

throwsnosuchalgorithmexception,invalidkeyexception {

Mac Mac = mac.getinstance ("HmacSHA256");

Mac.init (Newsecretkeyspec (Key, "HmacSHA256"));

Mac.update (data);

Returnmac.dofinal ();

}

Trim (): Go to Space

Uriencode ():

A. Keep ' a '-' Z ', ' a '-' z ', ' 0 '-' 9 ', '-', '. ', ' _ ', ' ~ ' unchanged

B. The space character must be converted to "%20" (instead of "+")

C.byte compiled to "%" followed by two-bit hexadecimal (uppercase letters)

D. If the file name (object name) is similar to "photos/jan/sample.jpg", which contains "/", it is not escaped.

E. Method implementation:

public static String Uriencode (Charsequence input, Boolean

Encodeslash) {

StringBuilder result = new StringBuilder ();

for (int i = 0; i < input.length (); i++) {

char ch = input.charat (i);

if (ch >= ' A ' && ch <= ' Z ') | | (Ch >= ' a '

&& ch <= ' z ') | | (Ch >= ' 0 ' && ch <= ' 9 ') | | ch = = ' _ ' | |

ch = = '-' | | ch = = ' ~ ' | | ch = = '. ') {

Result.append (CH);

} else if (ch = = '/') {

Result.append (Encodeslash? "%2f": CH);

} else {

Result.append (toHexUTF8 (ch));

}

}

return result.tostring ();

}

Second, multi-block transmission

Multi-block Upload request header:

X-amz-content-sha256:streaming-aws4-hmac-sha256-payload

The effective load (request body) is divided into blocks of fixed or variable size. The signature is computed by uploading blocks to avoid reading the payload of the entire file.

A. First block, calculate the signature of all request headers, empty body request

B. Second block, the signature of the first block is computed together with the payload

C. Nth block, the signature of block n-1 is computed together with the payload

D. Finally, the block that sends 0 bytes contains the signature of the nth block.

The signature in the request header authorization is calculated with the same single block upload:

Multi-block upload signature calculation flowchart

Block signature calculation in the request body:

This is more than a single-block upload in the request body.

Chunk-signature's signature calculation flowchart

Finally, said finished, the first time to write so long, there is not clear about the many forgive, there is any problem can give me a message, can also send my mailbox 18514689920@163.com.